2022-09-13T07:14:01.469 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-13T07:14:01.471 DEBUG:teuthology.run:Teuthology command: teuthology --name yuvalif-2022-09-13_07:01:40-rgw:multisite-wip-yuval-fix-56572-distro-default-smithi --owner scheduled_yuvalif@teuthology --description rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi} --verbose --archive /home/teuthworker/archive/yuvalif-2022-09-13_07:01:40-rgw:multisite-wip-yuval-fix-56572-distro-default-smithi/7030425 -- /home/teuthworker/archive/yuvalif-2022-09-13_07:01:40-rgw:multisite-wip-yuval-fix-56572-distro-default-smithi/7030425/orig.config.yaml 2022-09-13T07:14:01.511 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-09-13T07:14:01.584 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuvalif-2022-09-13_07:01:40-rgw:multisite-wip-yuval-fix-56572-distro-default-smithi/7030425 branch: wip-yuval-fix-56572 description: rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi} email: null first_in_suite: false job_id: '7030425' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuvalif-2022-09-13_07:01:40-rgw:multisite-wip-yuval-fix-56572-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuval-fix-56572 ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 840d05418c459144c92fb2a90d74a2ded29610da wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 840d05418c459144c92fb2a90d74a2ded29610da rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 rgw-multisite-tests: args: - tests.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuval-fix-56572 sha1: 840d05418c459144c92fb2a90d74a2ded29610da owner: scheduled_yuvalif@teuthology priority: 50 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 840d05418c459144c92fb2a90d74a2ded29610da sleep_before_teardown: 0 suite: rgw:multisite suite_branch: wip-yuval-fix-56572 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 840d05418c459144c92fb2a90d74a2ded29610da targets: smithi036.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwZ10DhqfIPFcUwNSkeR9qLDxnG1HoxeKhmdLj9JcAN5J/ZJ7gkiU4iMSbViuINM2NTvcD2/EL0PGkYz+BhowcJPciculEA/wU67kHrJUdjcxBug8sEgbDHF6t7iR47g0ksrysKN+YpSUseVtgCSrHRlawbpy2GHby45dGTVM8J6XB9KpPP+733LuNmL5/ZsSCi1zu3D/WkIqHeTHWXBBkUlJB+hjOiWhXS20e9hlBWTxv13p/IQ0fK+byN48zP5uxcHlFqapgOfXbI3SBhXlBJgRke/pcipC7+UyukyvFgi0sIsHyoFTgl31Ihw1qZ7Vfiok+f04z5Z01pbbrdvuh56cjrIUxZKunLxoSTnZhDIWNeTwb3Wx5VxyOkmsucjcXgUOXRI8DpIEYwBPpiVI6QfJpQ8pLMYerFKkvxqYibT7ypK66xRruGMRK8IDy5RAM7z3TRrF8ICNPVUiTxPbm5lVhI3zD35JoLewYboMo6mLRe9MBYo2ploAK0cGynf0= smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKhIgPHWHpUJcIGujf2yUi90hlD6vf5UNtX5WGaN0K+n6B8OUcfmnejuPWM21jMK7T/xfUT+nYNdWaHs/me21DI= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: main teuthology_sha1: b58f2c18636eb10faa77ed3614abd00cb85dfc2c timestamp: 2022-09-13_07:01:40 tube: smithi user: yuvalif verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.20382 2022-09-13T07:14:01.585 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa; will attempt to use it 2022-09-13T07:14:01.587 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks 2022-09-13T07:14:01.588 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-09-13T07:14:01.590 INFO:teuthology.task.internal:Checking packages... 2022-09-13T07:14:01.630 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '840d05418c459144c92fb2a90d74a2ded29610da' 2022-09-13T07:14:01.631 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-09-13T07:14:01.632 INFO:teuthology.packaging:ref: None 2022-09-13T07:14:01.632 INFO:teuthology.packaging:tag: None 2022-09-13T07:14:01.633 INFO:teuthology.packaging:branch: wip-yuval-fix-56572 2022-09-13T07:14:01.633 INFO:teuthology.packaging:sha1: 840d05418c459144c92fb2a90d74a2ded29610da 2022-09-13T07:14:01.634 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuval-fix-56572 2022-09-13T07:14:01.790 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-14918.g840d0541 2022-09-13T07:14:01.790 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-09-13T07:14:01.797 INFO:teuthology.task.internal:no buildpackages task found 2022-09-13T07:14:01.797 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-09-13T07:14:01.804 INFO:teuthology.task.internal:Saving configuration 2022-09-13T07:14:01.819 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-09-13T07:14:01.825 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-13T07:14:01.850 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi036.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuvalif-2022-09-13_07:01:40-rgw:multisite-wip-yuval-fix-56572-distro-default-smithi/7030425', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-09-13 07:09:03.327759', 'locked_by': 'scheduled_yuvalif@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNF9H/hXtKSDgP0MbhqA1YbQNu3V592FSvUCaxDMxvcIpUZROmZMACwbyVegK0piH9IoATXL/I4pLXBAtp1FDXg='} 2022-09-13T07:14:01.875 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuvalif-2022-09-13_07:01:40-rgw:multisite-wip-yuval-fix-56572-distro-default-smithi/7030425', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-09-13 07:09:03.326114', 'locked_by': 'scheduled_yuvalif@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKhIgPHWHpUJcIGujf2yUi90hlD6vf5UNtX5WGaN0K+n6B8OUcfmnejuPWM21jMK7T/xfUT+nYNdWaHs/me21DI='} 2022-09-13T07:14:01.876 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-09-13T07:14:01.883 INFO:teuthology.task.internal:roles: ubuntu@smithi036.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2022-09-13T07:14:01.883 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2022-09-13T07:14:01.883 INFO:teuthology.run_tasks:Running task console_log... 2022-09-13T07:14:01.959 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f29a104b488>, signals=[15]) 2022-09-13T07:14:01.960 INFO:teuthology.run_tasks:Running task internal.connect... 2022-09-13T07:14:01.968 INFO:teuthology.task.internal:Opening connections... 2022-09-13T07:14:01.968 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi036.front.sepia.ceph.com 2022-09-13T07:14:01.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi036.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-13T07:14:02.075 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2022-09-13T07:14:02.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-13T07:14:02.151 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-09-13T07:14:02.158 DEBUG:teuthology.orchestra.run.smithi036:> uname -m 2022-09-13T07:14:02.180 INFO:teuthology.orchestra.run.smithi036.stdout:x86_64 2022-09-13T07:14:02.181 DEBUG:teuthology.orchestra.run.smithi036:> cat /etc/os-release 2022-09-13T07:14:02.240 INFO:teuthology.orchestra.run.smithi036.stdout:NAME="CentOS Stream" 2022-09-13T07:14:02.241 INFO:teuthology.orchestra.run.smithi036.stdout:VERSION="8" 2022-09-13T07:14:02.241 INFO:teuthology.orchestra.run.smithi036.stdout:ID="centos" 2022-09-13T07:14:02.241 INFO:teuthology.orchestra.run.smithi036.stdout:ID_LIKE="rhel fedora" 2022-09-13T07:14:02.242 INFO:teuthology.orchestra.run.smithi036.stdout:VERSION_ID="8" 2022-09-13T07:14:02.242 INFO:teuthology.orchestra.run.smithi036.stdout:PLATFORM_ID="platform:el8" 2022-09-13T07:14:02.242 INFO:teuthology.orchestra.run.smithi036.stdout:PRETTY_NAME="CentOS Stream 8" 2022-09-13T07:14:02.242 INFO:teuthology.orchestra.run.smithi036.stdout:ANSI_COLOR="0;31" 2022-09-13T07:14:02.243 INFO:teuthology.orchestra.run.smithi036.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-09-13T07:14:02.243 INFO:teuthology.orchestra.run.smithi036.stdout:HOME_URL="https://centos.org/" 2022-09-13T07:14:02.243 INFO:teuthology.orchestra.run.smithi036.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-13T07:14:02.243 INFO:teuthology.orchestra.run.smithi036.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-13T07:14:02.244 INFO:teuthology.orchestra.run.smithi036.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-09-13T07:14:02.245 INFO:teuthology.lock.ops:Updating smithi036.front.sepia.ceph.com on lock server 2022-09-13T07:14:02.265 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2022-09-13T07:14:02.286 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2022-09-13T07:14:02.287 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2022-09-13T07:14:02.347 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2022-09-13T07:14:02.347 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="8" 2022-09-13T07:14:02.348 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2022-09-13T07:14:02.348 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2022-09-13T07:14:02.349 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="8" 2022-09-13T07:14:02.349 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el8" 2022-09-13T07:14:02.350 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 8" 2022-09-13T07:14:02.350 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2022-09-13T07:14:02.351 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-09-13T07:14:02.351 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2022-09-13T07:14:02.352 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-13T07:14:02.352 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-13T07:14:02.353 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-09-13T07:14:02.353 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2022-09-13T07:14:02.372 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-09-13T07:14:02.385 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-09-13T07:14:02.392 INFO:teuthology.task.internal:Checking for old test directory... 2022-09-13T07:14:02.393 DEBUG:teuthology.orchestra.run.smithi036:> test '!' -e /home/ubuntu/cephtest 2022-09-13T07:14:02.395 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2022-09-13T07:14:02.414 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-09-13T07:14:02.420 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-09-13T07:14:02.421 DEBUG:teuthology.orchestra.run.smithi036:> test -z $(ls -A /var/lib/ceph) 2022-09-13T07:14:02.456 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2022-09-13T07:14:02.483 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-09-13T07:14:02.547 INFO:teuthology.run_tasks:Running task kernel... 2022-09-13T07:14:02.568 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-09-13T07:14:02.570 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-09-13T07:14:02.571 DEBUG:teuthology.orchestra.run.smithi036:> test -f /run/.containerenv -o -f /.dockerenv 2022-09-13T07:14:02.572 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2022-09-13T07:14:02.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T07:14:02.592 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2022-09-13T07:14:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T07:14:02.594 DEBUG:teuthology.orchestra.run.smithi036:> uname -r 2022-09-13T07:14:02.651 INFO:teuthology.orchestra.run.smithi083.stdout:4.18.0-408.el8.x86_64 2022-09-13T07:14:02.652 INFO:teuthology.task.kernel:Running kernel on smithi083: 4.18.0-408.el8.x86_64 2022-09-13T07:14:02.652 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum install -y kernel 2022-09-13T07:14:02.653 INFO:teuthology.orchestra.run.smithi036.stdout:4.18.0-408.el8.x86_64 2022-09-13T07:14:02.654 INFO:teuthology.task.kernel:Running kernel on smithi036: 4.18.0-408.el8.x86_64 2022-09-13T07:14:02.654 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum install -y kernel 2022-09-13T07:14:05.844 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-8 - AppStream 30 MB/s | 25 MB 00:00 2022-09-13T07:14:05.974 INFO:teuthology.orchestra.run.smithi036.stdout:CentOS-8 - AppStream 27 MB/s | 25 MB 00:00 2022-09-13T07:14:11.019 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-8 - BaseOS 36 MB/s | 25 MB 00:00 2022-09-13T07:14:11.408 INFO:teuthology.orchestra.run.smithi036.stdout:CentOS-8 - BaseOS 33 MB/s | 25 MB 00:00 2022-09-13T07:14:14.851 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-8 - Extras 142 kB/s | 18 kB 00:00 2022-09-13T07:14:15.204 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-8 - PowerTools 21 MB/s | 5.0 MB 00:00 2022-09-13T07:14:15.474 INFO:teuthology.orchestra.run.smithi036.stdout:CentOS-8 - Extras 139 kB/s | 18 kB 00:00 2022-09-13T07:14:15.876 INFO:teuthology.orchestra.run.smithi036.stdout:CentOS-8 - PowerTools 18 MB/s | 5.0 MB 00:00 2022-09-13T07:14:18.512 INFO:teuthology.orchestra.run.smithi083.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2022-09-13T07:14:18.884 INFO:teuthology.orchestra.run.smithi036.stdout:Copr repo for python3-asyncssh owned by ceph 1.7 kB/s | 3.5 kB 00:01 2022-09-13T07:14:19.837 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 11 MB/s | 13 MB 00:01 2022-09-13T07:14:20.650 INFO:teuthology.orchestra.run.smithi036.stdout:Extra Packages for Enterprise Linux 7.8 MB/s | 13 MB 00:01 2022-09-13T07:14:22.920 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 675 kB/s | 24 kB 00:00 2022-09-13T07:14:23.901 INFO:teuthology.orchestra.run.smithi036.stdout:lab-extras 279 kB/s | 24 kB 00:00 2022-09-13T07:14:25.751 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2022-09-13T07:14:25.752 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2022-09-13T07:14:25.862 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:14:25.863 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2022-09-13T07:14:25.864 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:14:25.963 DEBUG:teuthology.orchestra.run.smithi083:> echo no | sudo yum reinstall kernel || true 2022-09-13T07:14:26.530 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:04 ago on Tue 13 Sep 2022 07:14:22 AM UTC. 2022-09-13T07:14:26.915 INFO:teuthology.orchestra.run.smithi036.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2022-09-13T07:14:26.915 INFO:teuthology.orchestra.run.smithi036.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2022-09-13T07:14:27.031 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:14:27.032 INFO:teuthology.orchestra.run.smithi036.stdout:Nothing to do. 2022-09-13T07:14:27.033 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:14:27.224 DEBUG:teuthology.orchestra.run.smithi036:> echo no | sudo yum reinstall kernel || true 2022-09-13T07:14:27.527 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:14:27.528 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:14:27.528 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2022-09-13T07:14:27.528 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:14:27.529 INFO:teuthology.orchestra.run.smithi083.stdout:Reinstalling: 2022-09-13T07:14:27.529 INFO:teuthology.orchestra.run.smithi083.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-09-13T07:14:27.530 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:14:27.530 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:14:27.530 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:14:27.531 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:14:27.531 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 8.5 M 2022-09-13T07:14:27.532 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 0 2022-09-13T07:14:27.532 INFO:teuthology.orchestra.run.smithi083.stderr:Operation aborted. 2022-09-13T07:14:27.590 INFO:teuthology.orchestra.run.smithi083.stdout:Is this ok [y/N]: 2022-09-13T07:14:27.591 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum reinstall -y kernel || true 2022-09-13T07:14:27.819 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:00:04 ago on Tue 13 Sep 2022 07:14:23 AM UTC. 2022-09-13T07:14:28.152 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:06 ago on Tue 13 Sep 2022 07:14:22 AM UTC. 2022-09-13T07:14:28.892 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:14:28.893 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:14:28.893 INFO:teuthology.orchestra.run.smithi036.stdout: Package Architecture Version Repository Size 2022-09-13T07:14:28.893 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:14:28.893 INFO:teuthology.orchestra.run.smithi036.stdout:Reinstalling: 2022-09-13T07:14:28.894 INFO:teuthology.orchestra.run.smithi036.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-09-13T07:14:28.894 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:14:28.894 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:14:28.894 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:14:28.895 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:14:28.895 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 8.5 M 2022-09-13T07:14:28.895 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 0 2022-09-13T07:14:28.896 INFO:teuthology.orchestra.run.smithi036.stderr:Operation aborted. 2022-09-13T07:14:28.962 INFO:teuthology.orchestra.run.smithi036.stdout:Is this ok [y/N]: 2022-09-13T07:14:28.963 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum reinstall -y kernel || true 2022-09-13T07:14:29.156 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:14:29.157 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:14:29.157 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2022-09-13T07:14:29.158 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:14:29.158 INFO:teuthology.orchestra.run.smithi083.stdout:Reinstalling: 2022-09-13T07:14:29.158 INFO:teuthology.orchestra.run.smithi083.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-09-13T07:14:29.158 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:14:29.159 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:14:29.159 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:14:29.159 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:14:29.159 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 8.5 M 2022-09-13T07:14:29.160 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 0 2022-09-13T07:14:29.160 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:14:29.414 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-4.18.0-408.el8.x86_64.rpm 33 MB/s | 8.5 MB 00:00 2022-09-13T07:14:29.415 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:14:29.415 INFO:teuthology.orchestra.run.smithi083.stdout:Total 33 MB/s | 8.5 MB 00:00 2022-09-13T07:14:29.512 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:14:29.555 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:00:06 ago on Tue 13 Sep 2022 07:14:23 AM UTC. 2022-09-13T07:14:29.888 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:14:29.889 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:14:30.074 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:14:30.087 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:14:30.391 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:14:30.582 INFO:teuthology.orchestra.run.smithi083.stdout: Reinstalling : kernel-4.18.0-408.el8.x86_64 1/2 2022-09-13T07:14:30.582 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : kernel-4.18.0-408.el8.x86_64 2/2 2022-09-13T07:14:30.639 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:14:30.640 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:14:30.641 INFO:teuthology.orchestra.run.smithi036.stdout: Package Architecture Version Repository Size 2022-09-13T07:14:30.641 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:14:30.641 INFO:teuthology.orchestra.run.smithi036.stdout:Reinstalling: 2022-09-13T07:14:30.642 INFO:teuthology.orchestra.run.smithi036.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-09-13T07:14:30.642 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:14:30.642 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:14:30.642 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:14:30.643 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:14:30.643 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 8.5 M 2022-09-13T07:14:30.643 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 0 2022-09-13T07:14:30.643 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:14:30.861 INFO:teuthology.orchestra.run.smithi036.stdout:kernel-4.18.0-408.el8.x86_64.rpm 39 MB/s | 8.5 MB 00:00 2022-09-13T07:14:30.862 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:14:30.863 INFO:teuthology.orchestra.run.smithi036.stdout:Total 39 MB/s | 8.5 MB 00:00 2022-09-13T07:14:30.968 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:14:30.973 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-4.18.0-408.el8.x86_64 2/2 2022-09-13T07:14:30.974 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 1/2 2022-09-13T07:14:31.377 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:14:31.377 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:14:31.443 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 2/2 2022-09-13T07:14:31.444 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:14:31.444 INFO:teuthology.orchestra.run.smithi083.stdout:Reinstalled: 2022-09-13T07:14:31.444 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-4.18.0-408.el8.x86_64 2022-09-13T07:14:31.444 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:14:31.445 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:14:31.528 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel --last | head -n 1 2022-09-13T07:14:31.564 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:14:31.578 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:14:31.653 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-4.18.0-408.el8.x86_64 Tue 13 Sep 2022 07:14:30 AM UTC 2022-09-13T07:14:31.654 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-408.el8.x86_64 2022-09-13T07:14:31.655 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-09-13T07:14:31.655 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2022-09-13T07:14:31.655 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-09-13T07:14:31.685 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1 2022-09-13T07:14:31.698 DEBUG:teuthology.parallel:result is None 2022-09-13T07:14:31.880 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:14:32.096 INFO:teuthology.orchestra.run.smithi036.stdout: Reinstalling : kernel-4.18.0-408.el8.x86_64 1/2 2022-09-13T07:14:32.096 INFO:teuthology.orchestra.run.smithi036.stdout: Cleanup : kernel-4.18.0-408.el8.x86_64 2/2 2022-09-13T07:14:32.534 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: kernel-4.18.0-408.el8.x86_64 2/2 2022-09-13T07:14:32.534 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 1/2 2022-09-13T07:14:33.050 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 2/2 2022-09-13T07:14:33.050 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:14:33.050 INFO:teuthology.orchestra.run.smithi036.stdout:Reinstalled: 2022-09-13T07:14:33.051 INFO:teuthology.orchestra.run.smithi036.stdout: kernel-4.18.0-408.el8.x86_64 2022-09-13T07:14:33.051 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:14:33.051 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:14:33.126 DEBUG:teuthology.orchestra.run.smithi036:> rpm -q kernel --last | head -n 1 2022-09-13T07:14:33.298 INFO:teuthology.orchestra.run.smithi036.stdout:kernel-4.18.0-408.el8.x86_64 Tue 13 Sep 2022 07:14:31 AM UTC 2022-09-13T07:14:33.299 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-408.el8.x86_64 2022-09-13T07:14:33.300 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-09-13T07:14:33.300 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2022-09-13T07:14:33.300 DEBUG:teuthology.orchestra.run.smithi036:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-09-13T07:14:33.337 INFO:teuthology.orchestra.run.smithi036.stdout:ttyS1 2022-09-13T07:14:33.351 DEBUG:teuthology.parallel:result is None 2022-09-13T07:14:33.351 INFO:teuthology.run_tasks:Running task internal.base... 2022-09-13T07:14:33.359 INFO:teuthology.task.internal:Creating test directory... 2022-09-13T07:14:33.359 DEBUG:teuthology.orchestra.run.smithi036:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-09-13T07:14:33.394 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-09-13T07:14:33.414 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-09-13T07:14:33.421 INFO:teuthology.run_tasks:Running task internal.archive... 2022-09-13T07:14:33.431 INFO:teuthology.task.internal:Creating archive directory... 2022-09-13T07:14:33.434 DEBUG:teuthology.orchestra.run.smithi036:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-09-13T07:14:33.451 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-09-13T07:14:33.491 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-09-13T07:14:33.500 INFO:teuthology.task.internal:Enabling coredump saving... 2022-09-13T07:14:33.500 DEBUG:teuthology.orchestra.run.smithi036:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-09-13T07:14:33.515 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-09-13T07:14:33.544 INFO:teuthology.orchestra.run.smithi036.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-13T07:14:33.554 INFO:teuthology.orchestra.run.smithi036.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-13T07:14:33.562 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-13T07:14:33.574 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-13T07:14:33.575 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-09-13T07:14:33.583 INFO:teuthology.task.internal:Configuring sudo... 2022-09-13T07:14:33.583 DEBUG:teuthology.orchestra.run.smithi036:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-09-13T07:14:33.598 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-09-13T07:14:33.644 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-09-13T07:14:33.653 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-09-13T07:14:33.654 DEBUG:teuthology.orchestra.run.smithi036:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-09-13T07:14:33.666 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-09-13T07:14:33.702 DEBUG:teuthology.orchestra.run.smithi036:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-13T07:14:33.767 DEBUG:teuthology.orchestra.run.smithi036:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-13T07:14:33.814 DEBUG:teuthology.orchestra.run.smithi036:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-13T07:14:33.898 DEBUG:teuthology.orchestra.run.smithi036:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-13T07:14:33.941 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:14:33.942 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-09-13T07:14:34.009 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-13T07:14:34.053 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-13T07:14:34.111 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-13T07:14:34.195 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-13T07:14:34.240 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:14:34.241 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-09-13T07:14:34.308 DEBUG:teuthology.orchestra.run.smithi036:> sudo service rsyslog restart 2022-09-13T07:14:34.310 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2022-09-13T07:14:34.375 INFO:teuthology.orchestra.run.smithi036.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-09-13T07:14:34.415 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-09-13T07:14:34.767 INFO:teuthology.run_tasks:Running task internal.timer... 2022-09-13T07:14:34.776 INFO:teuthology.task.internal:Starting timer... 2022-09-13T07:14:34.776 INFO:teuthology.run_tasks:Running task pcp... 2022-09-13T07:14:34.789 INFO:teuthology.run_tasks:Running task selinux... 2022-09-13T07:14:34.798 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-09-13T07:14:34.798 DEBUG:teuthology.orchestra.run.smithi036:> sudo service auditd rotate 2022-09-13T07:14:34.888 INFO:teuthology.orchestra.run.smithi036.stdout:Rotating logs: 2022-09-13T07:14:34.890 DEBUG:teuthology.orchestra.run.smithi083:> sudo service auditd rotate 2022-09-13T07:14:35.022 INFO:teuthology.orchestra.run.smithi083.stdout:Rotating logs: 2022-09-13T07:14:35.024 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-09-13T07:14:35.024 DEBUG:teuthology.orchestra.run.smithi036:> /usr/sbin/getenforce 2022-09-13T07:14:35.052 INFO:teuthology.orchestra.run.smithi036.stdout:Permissive 2022-09-13T07:14:35.053 DEBUG:teuthology.orchestra.run.smithi083:> /usr/sbin/getenforce 2022-09-13T07:14:35.082 INFO:teuthology.orchestra.run.smithi083.stdout:Permissive 2022-09-13T07:14:35.083 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi036.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi083.front.sepia.ceph.com': 'permissive'} 2022-09-13T07:14:35.083 DEBUG:teuthology.orchestra.run.smithi036:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-09-13T07:14:35.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T07:14:35.122 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-09-13T07:14:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T07:14:35.150 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-09-13T07:14:35.150 DEBUG:teuthology.orchestra.run.smithi036:> sudo /usr/sbin/setenforce permissive 2022-09-13T07:14:35.202 DEBUG:teuthology.orchestra.run.smithi083:> sudo /usr/sbin/setenforce permissive 2022-09-13T07:14:35.242 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-09-13T07:14:35.252 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-09-13T07:14:35.258 INFO:teuthology.repo_utils:Fetching main from origin 2022-09-13T07:14:35.290 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-09-13T07:14:35.311 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-09-13T07:14:35.313 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi036.front.sepia.ceph.com,smithi083.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-09-13T07:20:37.633 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi036.front.sepia.ceph.com'), Remote(name='ubuntu@smithi083.front.sepia.ceph.com')] 2022-09-13T07:20:37.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi036.front.sepia.ceph.com' 2022-09-13T07:20:37.685 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi036.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-13T07:20:37.771 DEBUG:teuthology.orchestra.run.smithi036:> true 2022-09-13T07:20:37.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2022-09-13T07:20:37.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-13T07:20:37.926 DEBUG:teuthology.orchestra.run.smithi083:> true 2022-09-13T07:20:37.992 INFO:teuthology.run_tasks:Running task clock... 2022-09-13T07:20:38.007 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-09-13T07:20:38.008 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-09-13T07:20:38.011 DEBUG:teuthology.orchestra.run.smithi036:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-09-13T07:20:38.013 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-09-13T07:20:38.014 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-09-13T07:20:38.044 INFO:teuthology.orchestra.run.smithi036.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-09-13T07:20:38.059 INFO:teuthology.orchestra.run.smithi036.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-09-13T07:20:38.064 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-09-13T07:20:38.084 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-09-13T07:20:38.134 INFO:teuthology.orchestra.run.smithi036.stderr:sudo: ntpd: command not found 2022-09-13T07:20:38.140 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: ntpd: command not found 2022-09-13T07:20:38.146 INFO:teuthology.orchestra.run.smithi036.stdout:506 Cannot talk to daemon 2022-09-13T07:20:38.152 INFO:teuthology.orchestra.run.smithi083.stdout:506 Cannot talk to daemon 2022-09-13T07:20:38.160 INFO:teuthology.orchestra.run.smithi036.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-09-13T07:20:38.166 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-09-13T07:20:38.174 INFO:teuthology.orchestra.run.smithi036.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-09-13T07:20:38.179 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-09-13T07:20:38.245 INFO:teuthology.orchestra.run.smithi036.stderr:bash: ntpq: command not found 2022-09-13T07:20:38.247 INFO:teuthology.orchestra.run.smithi083.stderr:bash: ntpq: command not found 2022-09-13T07:20:38.249 INFO:teuthology.orchestra.run.smithi036.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-09-13T07:20:38.250 INFO:teuthology.orchestra.run.smithi036.stdout:=============================================================================== 2022-09-13T07:20:38.250 INFO:teuthology.orchestra.run.smithi036.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-13T07:20:38.251 INFO:teuthology.orchestra.run.smithi036.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-13T07:20:38.251 INFO:teuthology.orchestra.run.smithi036.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-13T07:20:38.251 INFO:teuthology.orchestra.run.smithi036.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-13T07:20:38.253 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-09-13T07:20:38.254 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2022-09-13T07:20:38.255 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-13T07:20:38.255 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-13T07:20:38.256 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-13T07:20:38.256 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-13T07:20:38.257 INFO:teuthology.run_tasks:Running task install... 2022-09-13T07:20:38.266 DEBUG:teuthology.task.install:project ceph 2022-09-13T07:20:38.267 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '840d05418c459144c92fb2a90d74a2ded29610da'}} 2022-09-13T07:20:38.267 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '840d05418c459144c92fb2a90d74a2ded29610da'} 2022-09-13T07:20:38.267 INFO:teuthology.task.install:Using flavor: default 2022-09-13T07:20:38.275 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-09-13T07:20:38.276 INFO:teuthology.task.install:extra packages: [] 2022-09-13T07:20:38.277 DEBUG:teuthology.orchestra.run.smithi036:> sudo lsb_release -is 2022-09-13T07:20:38.327 INFO:teuthology.orchestra.run.smithi036.stdout:CentOSStream 2022-09-13T07:20:38.329 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-13T07:20:38.330 DEBUG:teuthology.orchestra.run.smithi083:> sudo lsb_release -is 2022-09-13T07:20:38.331 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '840d05418c459144c92fb2a90d74a2ded29610da', 'tag': None, 'wait_for_package': False} 2022-09-13T07:20:38.332 DEBUG:teuthology.orchestra.run.smithi036:> sudo lsb_release -is 2022-09-13T07:20:38.369 INFO:teuthology.orchestra.run.smithi083.stdout:CentOSStream 2022-09-13T07:20:38.371 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-13T07:20:38.372 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '840d05418c459144c92fb2a90d74a2ded29610da', 'tag': None, 'wait_for_package': False} 2022-09-13T07:20:38.372 DEBUG:teuthology.orchestra.run.smithi083:> sudo lsb_release -is 2022-09-13T07:20:38.407 INFO:teuthology.orchestra.run.smithi036.stdout:CentOSStream 2022-09-13T07:20:38.409 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-13T07:20:38.410 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=840d05418c459144c92fb2a90d74a2ded29610da 2022-09-13T07:20:38.453 INFO:teuthology.orchestra.run.smithi083.stdout:CentOSStream 2022-09-13T07:20:38.454 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-13T07:20:38.455 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=840d05418c459144c92fb2a90d74a2ded29610da 2022-09-13T07:20:38.572 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuval-fix-56572/840d05418c459144c92fb2a90d74a2ded29610da/centos/8/flavors/default/ 2022-09-13T07:20:38.573 INFO:teuthology.task.install.rpm:Package version is 17.0.0-14918.g840d0541 2022-09-13T07:20:38.598 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuval-fix-56572/840d05418c459144c92fb2a90d74a2ded29610da/centos/8/flavors/default/ 2022-09-13T07:20:38.599 INFO:teuthology.task.install.rpm:Package version is 17.0.0-14918.g840d0541 2022-09-13T07:20:38.697 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuval-fix-56572/840d05418c459144c92fb2a90d74a2ded29610da/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuval-fix-56572/840d05418c459144c92fb2a90d74a2ded29610da/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuval-fix-56572/840d05418c459144c92fb2a90d74a2ded29610da/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-09-13T07:20:38.698 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:20:38.699 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-09-13T07:20:38.714 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuval-fix-56572/840d05418c459144c92fb2a90d74a2ded29610da/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuval-fix-56572/840d05418c459144c92fb2a90d74a2ded29610da/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuval-fix-56572/840d05418c459144c92fb2a90d74a2ded29610da/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-09-13T07:20:38.714 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:20:38.715 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-09-13T07:20:38.727 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2022-09-13T07:20:38.727 DEBUG:teuthology.orchestra.run.smithi036:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/840d05418c459144c92fb2a90d74a2ded29610da/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-09-13T07:20:38.743 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2022-09-13T07:20:38.743 DEBUG:teuthology.orchestra.run.smithi083:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/840d05418c459144c92fb2a90d74a2ded29610da/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-09-13T07:20:38.795 DEBUG:teuthology.orchestra.run.smithi036:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-09-13T07:20:38.810 DEBUG:teuthology.orchestra.run.smithi083:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-09-13T07:20:38.874 DEBUG:teuthology.orchestra.run.smithi036:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-09-13T07:20:38.888 DEBUG:teuthology.orchestra.run.smithi083:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-09-13T07:20:38.942 INFO:teuthology.orchestra.run.smithi036.stdout:check_obsoletes = 1 2022-09-13T07:20:38.943 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum clean all 2022-09-13T07:20:38.954 INFO:teuthology.orchestra.run.smithi083.stdout:check_obsoletes = 1 2022-09-13T07:20:38.955 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum clean all 2022-09-13T07:20:39.201 INFO:teuthology.orchestra.run.smithi036.stdout:55 files removed 2022-09-13T07:20:39.202 INFO:teuthology.orchestra.run.smithi083.stdout:55 files removed 2022-09-13T07:20:39.226 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install ceph-radosgw 2022-09-13T07:20:39.228 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-radosgw 2022-09-13T07:20:39.834 INFO:teuthology.orchestra.run.smithi083.stdout:ceph packages for x86_64 309 kB/s | 81 kB 00:00 2022-09-13T07:20:39.888 INFO:teuthology.orchestra.run.smithi036.stdout:ceph packages for x86_64 277 kB/s | 81 kB 00:00 2022-09-13T07:20:40.102 INFO:teuthology.orchestra.run.smithi083.stdout:ceph noarch packages 40 kB/s | 9.4 kB 00:00 2022-09-13T07:20:40.180 INFO:teuthology.orchestra.run.smithi036.stdout:ceph noarch packages 37 kB/s | 9.4 kB 00:00 2022-09-13T07:20:40.352 INFO:teuthology.orchestra.run.smithi083.stdout:ceph source packages 6.7 kB/s | 1.5 kB 00:00 2022-09-13T07:20:40.464 INFO:teuthology.orchestra.run.smithi036.stdout:ceph source packages 5.9 kB/s | 1.5 kB 00:00 2022-09-13T07:20:41.072 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-8 - AppStream 35 MB/s | 25 MB 00:00 2022-09-13T07:20:41.300 INFO:teuthology.orchestra.run.smithi036.stdout:CentOS-8 - AppStream 30 MB/s | 25 MB 00:00 2022-09-13T07:20:46.022 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-8 - BaseOS 39 MB/s | 25 MB 00:00 2022-09-13T07:20:46.484 INFO:teuthology.orchestra.run.smithi036.stdout:CentOS-8 - BaseOS 42 MB/s | 25 MB 00:00 2022-09-13T07:20:49.838 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-8 - Extras 158 kB/s | 18 kB 00:00 2022-09-13T07:20:50.240 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-8 - PowerTools 18 MB/s | 5.0 MB 00:00 2022-09-13T07:20:50.557 INFO:teuthology.orchestra.run.smithi036.stdout:CentOS-8 - Extras 114 kB/s | 18 kB 00:00 2022-09-13T07:20:50.969 INFO:teuthology.orchestra.run.smithi036.stdout:CentOS-8 - PowerTools 17 MB/s | 5.0 MB 00:00 2022-09-13T07:20:53.606 INFO:teuthology.orchestra.run.smithi083.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2022-09-13T07:20:53.893 INFO:teuthology.orchestra.run.smithi036.stdout:Copr repo for python3-asyncssh owned by ceph 1.8 kB/s | 3.5 kB 00:01 2022-09-13T07:20:54.865 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 11 MB/s | 13 MB 00:01 2022-09-13T07:20:55.127 INFO:teuthology.orchestra.run.smithi036.stdout:Extra Packages for Enterprise Linux 12 MB/s | 13 MB 00:01 2022-09-13T07:20:57.890 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 411 kB/s | 24 kB 00:00 2022-09-13T07:20:58.494 INFO:teuthology.orchestra.run.smithi036.stdout:lab-extras 478 kB/s | 24 kB 00:00 2022-09-13T07:21:00.971 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:21:00.973 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:21:00.973 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2022-09-13T07:21:00.974 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:21:00.974 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:21:00.974 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw x86_64 2:17.0.0-14918.g840d0541.el8 ceph 18 M 2022-09-13T07:21:00.975 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrading: 2022-09-13T07:21:00.975 INFO:teuthology.orchestra.run.smithi083.stdout: librados2 x86_64 2:17.0.0-14918.g840d0541.el8 ceph 3.6 M 2022-09-13T07:21:00.975 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1 x86_64 2:17.0.0-14918.g840d0541.el8 ceph 3.6 M 2022-09-13T07:21:00.975 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2022-09-13T07:21:00.975 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base x86_64 2:17.0.0-14918.g840d0541.el8 ceph 5.2 M 2022-09-13T07:21:00.976 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common x86_64 2:17.0.0-14918.g840d0541.el8 ceph 24 M 2022-09-13T07:21:00.976 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux x86_64 2:17.0.0-14918.g840d0541.el8 ceph 24 k 2022-09-13T07:21:00.976 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-09-13T07:21:00.976 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2 x86_64 2:17.0.0-14918.g840d0541.el8 ceph 766 k 2022-09-13T07:21:00.977 INFO:teuthology.orchestra.run.smithi083.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-09-13T07:21:00.977 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2022-09-13T07:21:00.977 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1 x86_64 2:17.0.0-14918.g840d0541.el8 ceph 474 k 2022-09-13T07:21:00.977 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2022-09-13T07:21:00.978 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2 x86_64 2:17.0.0-14918.g840d0541.el8 ceph 6.4 M 2022-09-13T07:21:00.978 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-09-13T07:21:00.978 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-09-13T07:21:00.978 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse 2022-09-13T07:21:00.979 INFO:teuthology.orchestra.run.smithi083.stdout: x86_64 2:17.0.0-14918.g840d0541.el8 ceph 45 k 2022-09-13T07:21:00.979 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common x86_64 2:17.0.0-14918.g840d0541.el8 ceph 109 k 2022-09-13T07:21:00.979 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs x86_64 2:17.0.0-14918.g840d0541.el8 ceph 190 k 2022-09-13T07:21:00.979 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-09-13T07:21:00.979 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados x86_64 2:17.0.0-14918.g840d0541.el8 ceph 370 k 2022-09-13T07:21:00.980 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd x86_64 2:17.0.0-14918.g840d0541.el8 ceph 357 k 2022-09-13T07:21:00.980 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw x86_64 2:17.0.0-14918.g840d0541.el8 ceph 106 k 2022-09-13T07:21:00.980 INFO:teuthology.orchestra.run.smithi083.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-09-13T07:21:00.980 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:21:00.981 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:21:00.981 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:21:00.981 INFO:teuthology.orchestra.run.smithi083.stdout:Install 21 Packages 2022-09-13T07:21:00.981 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrade 2 Packages 2022-09-13T07:21:00.982 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:21:00.982 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 66 M 2022-09-13T07:21:00.983 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:21:01.500 INFO:teuthology.orchestra.run.smithi083.stdout:(1/23): ceph-base-17.0.0-14918.g840d0541.el8.x8 9.9 MB/s | 5.2 MB 00:00 2022-09-13T07:21:01.583 INFO:teuthology.orchestra.run.smithi083.stdout:(2/23): ceph-selinux-17.0.0-14918.g840d0541.el8 290 kB/s | 24 kB 00:00 2022-09-13T07:21:01.817 INFO:teuthology.orchestra.run.smithi083.stdout:(3/23): libcephfs2-17.0.0-14918.g840d0541.el8.x 3.2 MB/s | 766 kB 00:00 2022-09-13T07:21:01.900 INFO:teuthology.orchestra.run.smithi083.stdout:(4/23): libradosstriper1-17.0.0-14918.g840d0541 5.6 MB/s | 474 kB 00:00 2022-09-13T07:21:01.910 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:21:01.913 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:21:01.913 INFO:teuthology.orchestra.run.smithi036.stdout: Package Arch Version Repository Size 2022-09-13T07:21:01.913 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:21:01.914 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:21:01.914 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-radosgw x86_64 2:17.0.0-14918.g840d0541.el8 ceph 18 M 2022-09-13T07:21:01.914 INFO:teuthology.orchestra.run.smithi036.stdout:Upgrading: 2022-09-13T07:21:01.914 INFO:teuthology.orchestra.run.smithi036.stdout: librados2 x86_64 2:17.0.0-14918.g840d0541.el8 ceph 3.6 M 2022-09-13T07:21:01.915 INFO:teuthology.orchestra.run.smithi036.stdout: librbd1 x86_64 2:17.0.0-14918.g840d0541.el8 ceph 3.6 M 2022-09-13T07:21:01.915 INFO:teuthology.orchestra.run.smithi036.stdout:Installing dependencies: 2022-09-13T07:21:01.915 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-base x86_64 2:17.0.0-14918.g840d0541.el8 ceph 5.2 M 2022-09-13T07:21:01.915 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-common x86_64 2:17.0.0-14918.g840d0541.el8 ceph 24 M 2022-09-13T07:21:01.916 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-selinux x86_64 2:17.0.0-14918.g840d0541.el8 ceph 24 k 2022-09-13T07:21:01.916 INFO:teuthology.orchestra.run.smithi036.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-09-13T07:21:01.916 INFO:teuthology.orchestra.run.smithi036.stdout: libcephfs2 x86_64 2:17.0.0-14918.g840d0541.el8 ceph 766 k 2022-09-13T07:21:01.916 INFO:teuthology.orchestra.run.smithi036.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-09-13T07:21:01.917 INFO:teuthology.orchestra.run.smithi036.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2022-09-13T07:21:01.917 INFO:teuthology.orchestra.run.smithi036.stdout: libradosstriper1 x86_64 2:17.0.0-14918.g840d0541.el8 ceph 474 k 2022-09-13T07:21:01.917 INFO:teuthology.orchestra.run.smithi036.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2022-09-13T07:21:01.917 INFO:teuthology.orchestra.run.smithi036.stdout: librgw2 x86_64 2:17.0.0-14918.g840d0541.el8 ceph 6.4 M 2022-09-13T07:21:01.917 INFO:teuthology.orchestra.run.smithi036.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-09-13T07:21:01.918 INFO:teuthology.orchestra.run.smithi036.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-09-13T07:21:01.918 INFO:teuthology.orchestra.run.smithi036.stdout: python3-ceph-argparse 2022-09-13T07:21:01.918 INFO:teuthology.orchestra.run.smithi036.stdout: x86_64 2:17.0.0-14918.g840d0541.el8 ceph 45 k 2022-09-13T07:21:01.919 INFO:teuthology.orchestra.run.smithi036.stdout: python3-ceph-common x86_64 2:17.0.0-14918.g840d0541.el8 ceph 109 k 2022-09-13T07:21:01.919 INFO:teuthology.orchestra.run.smithi036.stdout: python3-cephfs x86_64 2:17.0.0-14918.g840d0541.el8 ceph 190 k 2022-09-13T07:21:01.919 INFO:teuthology.orchestra.run.smithi036.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-09-13T07:21:01.919 INFO:teuthology.orchestra.run.smithi036.stdout: python3-rados x86_64 2:17.0.0-14918.g840d0541.el8 ceph 370 k 2022-09-13T07:21:01.920 INFO:teuthology.orchestra.run.smithi036.stdout: python3-rbd x86_64 2:17.0.0-14918.g840d0541.el8 ceph 357 k 2022-09-13T07:21:01.920 INFO:teuthology.orchestra.run.smithi036.stdout: python3-rgw x86_64 2:17.0.0-14918.g840d0541.el8 ceph 106 k 2022-09-13T07:21:01.920 INFO:teuthology.orchestra.run.smithi036.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-09-13T07:21:01.920 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:21:01.920 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:21:01.921 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:21:01.921 INFO:teuthology.orchestra.run.smithi036.stdout:Install 21 Packages 2022-09-13T07:21:01.921 INFO:teuthology.orchestra.run.smithi036.stdout:Upgrade 2 Packages 2022-09-13T07:21:01.921 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:21:01.922 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 66 M 2022-09-13T07:21:01.922 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:21:02.770 INFO:teuthology.orchestra.run.smithi036.stdout:(1/23): ceph-base-17.0.0-14918.g840d0541.el8.x8 6.1 MB/s | 5.2 MB 00:00 2022-09-13T07:21:02.829 INFO:teuthology.orchestra.run.smithi036.stdout:(2/23): ceph-selinux-17.0.0-14918.g840d0541.el8 417 kB/s | 24 kB 00:00 2022-09-13T07:21:02.954 INFO:teuthology.orchestra.run.smithi036.stdout:(3/23): libcephfs2-17.0.0-14918.g840d0541.el8.x 6.0 MB/s | 766 kB 00:00 2022-09-13T07:21:03.063 INFO:teuthology.orchestra.run.smithi036.stdout:(4/23): libradosstriper1-17.0.0-14918.g840d0541 4.3 MB/s | 474 kB 00:00 2022-09-13T07:21:04.421 INFO:teuthology.orchestra.run.smithi036.stdout:(5/23): librgw2-17.0.0-14918.g840d0541.el8.x86_ 4.7 MB/s | 6.4 MB 00:01 2022-09-13T07:21:04.459 INFO:teuthology.orchestra.run.smithi083.stdout:(5/23): ceph-common-17.0.0-14918.g840d0541.el8. 6.9 MB/s | 24 MB 00:03 2022-09-13T07:21:04.489 INFO:teuthology.orchestra.run.smithi036.stdout:(6/23): python3-ceph-argparse-17.0.0-14918.g840 670 kB/s | 45 kB 00:00 2022-09-13T07:21:04.557 INFO:teuthology.orchestra.run.smithi036.stdout:(7/23): python3-ceph-common-17.0.0-14918.g840d0 1.6 MB/s | 109 kB 00:00 2022-09-13T07:21:04.577 INFO:teuthology.orchestra.run.smithi083.stdout:(6/23): librgw2-17.0.0-14918.g840d0541.el8.x86_ 2.4 MB/s | 6.4 MB 00:02 2022-09-13T07:21:04.601 INFO:teuthology.orchestra.run.smithi083.stdout:(7/23): python3-ceph-argparse-17.0.0-14918.g840 314 kB/s | 45 kB 00:00 2022-09-13T07:21:04.622 INFO:teuthology.orchestra.run.smithi036.stdout:(8/23): python3-cephfs-17.0.0-14918.g840d0541.e 2.8 MB/s | 190 kB 00:00 2022-09-13T07:21:04.635 INFO:teuthology.orchestra.run.smithi083.stdout:(8/23): python3-ceph-common-17.0.0-14918.g840d0 1.8 MB/s | 109 kB 00:00 2022-09-13T07:21:04.660 INFO:teuthology.orchestra.run.smithi083.stdout:(9/23): python3-cephfs-17.0.0-14918.g840d0541.e 3.2 MB/s | 190 kB 00:00 2022-09-13T07:21:04.756 INFO:teuthology.orchestra.run.smithi036.stdout:(9/23): python3-rados-17.0.0-14918.g840d0541.el 2.7 MB/s | 370 kB 00:00 2022-09-13T07:21:04.760 INFO:teuthology.orchestra.run.smithi083.stdout:(10/23): python3-rbd-17.0.0-14918.g840d0541.el8 3.5 MB/s | 357 kB 00:00 2022-09-13T07:21:04.794 INFO:teuthology.orchestra.run.smithi083.stdout:(11/23): python3-rados-17.0.0-14918.g840d0541.e 2.3 MB/s | 370 kB 00:00 2022-09-13T07:21:04.828 INFO:teuthology.orchestra.run.smithi083.stdout:(12/23): python3-rgw-17.0.0-14918.g840d0541.el8 1.6 MB/s | 106 kB 00:00 2022-09-13T07:21:04.870 INFO:teuthology.orchestra.run.smithi083.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 4.6 MB/s | 354 kB 00:00 2022-09-13T07:21:04.896 INFO:teuthology.orchestra.run.smithi083.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 1.8 MB/s | 44 kB 00:00 2022-09-13T07:21:04.920 INFO:teuthology.orchestra.run.smithi083.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2022-09-13T07:21:04.954 INFO:teuthology.orchestra.run.smithi083.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.0 MB/s | 259 kB 00:00 2022-09-13T07:21:05.006 INFO:teuthology.orchestra.run.smithi036.stdout:(10/23): ceph-radosgw-17.0.0-14918.g840d0541.el 5.8 MB/s | 18 MB 00:03 2022-09-13T07:21:05.032 INFO:teuthology.orchestra.run.smithi036.stdout:(11/23): python3-rbd-17.0.0-14918.g840d0541.el8 1.3 MB/s | 357 kB 00:00 2022-09-13T07:21:05.065 INFO:teuthology.orchestra.run.smithi036.stdout:(12/23): python3-rgw-17.0.0-14918.g840d0541.el8 1.8 MB/s | 106 kB 00:00 2022-09-13T07:21:05.099 INFO:teuthology.orchestra.run.smithi036.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 5.2 MB/s | 354 kB 00:00 2022-09-13T07:21:05.124 INFO:teuthology.orchestra.run.smithi036.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 1.8 MB/s | 44 kB 00:00 2022-09-13T07:21:05.129 INFO:teuthology.orchestra.run.smithi083.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 336 kB/s | 59 kB 00:00 2022-09-13T07:21:05.175 INFO:teuthology.orchestra.run.smithi036.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 932 kB/s | 47 kB 00:00 2022-09-13T07:21:05.200 INFO:teuthology.orchestra.run.smithi036.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.9 MB/s | 259 kB 00:00 2022-09-13T07:21:05.206 INFO:teuthology.orchestra.run.smithi083.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2022-09-13T07:21:05.230 INFO:teuthology.orchestra.run.smithi083.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 747 kB/s | 75 kB 00:00 2022-09-13T07:21:05.358 INFO:teuthology.orchestra.run.smithi036.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 372 kB/s | 59 kB 00:00 2022-09-13T07:21:05.448 INFO:teuthology.orchestra.run.smithi036.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2022-09-13T07:21:05.476 INFO:teuthology.orchestra.run.smithi036.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 641 kB/s | 75 kB 00:00 2022-09-13T07:21:05.522 INFO:teuthology.orchestra.run.smithi083.stdout:(20/23): ceph-radosgw-17.0.0-14918.g840d0541.el 3.9 MB/s | 18 MB 00:04 2022-09-13T07:21:05.577 INFO:teuthology.orchestra.run.smithi083.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 4.7 MB/s | 1.7 MB 00:00 2022-09-13T07:21:05.739 INFO:teuthology.orchestra.run.smithi083.stdout:(22/23): librados2-17.0.0-14918.g840d0541.el8.x 7.1 MB/s | 3.6 MB 00:00 2022-09-13T07:21:05.768 INFO:teuthology.orchestra.run.smithi036.stdout:(20/23): ceph-common-17.0.0-14918.g840d0541.el8 6.3 MB/s | 24 MB 00:03 2022-09-13T07:21:05.843 INFO:teuthology.orchestra.run.smithi036.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 4.4 MB/s | 1.7 MB 00:00 2022-09-13T07:21:05.939 INFO:teuthology.orchestra.run.smithi083.stdout:(23/23): librbd1-17.0.0-14918.g840d0541.el8.x86 8.6 MB/s | 3.6 MB 00:00 2022-09-13T07:21:05.943 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:21:05.944 INFO:teuthology.orchestra.run.smithi083.stdout:Total 13 MB/s | 66 MB 00:04 2022-09-13T07:21:05.981 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:21:06.032 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:21:06.032 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:21:06.043 INFO:teuthology.orchestra.run.smithi036.stdout:(22/23): librados2-17.0.0-14918.g840d0541.el8.x 6.4 MB/s | 3.6 MB 00:00 2022-09-13T07:21:06.203 INFO:teuthology.orchestra.run.smithi036.stdout:(23/23): librbd1-17.0.0-14918.g840d0541.el8.x86 8.2 MB/s | 3.6 MB 00:00 2022-09-13T07:21:06.206 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:21:06.206 INFO:teuthology.orchestra.run.smithi036.stdout:Total 15 MB/s | 66 MB 00:04 2022-09-13T07:21:06.245 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:21:06.302 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:21:06.302 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:21:06.721 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:21:06.724 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:21:06.999 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:21:07.003 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:21:07.894 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:21:08.063 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2022-09-13T07:21:08.303 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2022-09-13T07:21:08.312 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:21:08.490 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-09-13T07:21:08.495 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2022-09-13T07:21:08.739 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2022-09-13T07:21:08.871 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-09-13T07:21:08.938 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-09-13T07:21:09.090 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librados2-2:17.0.0-14918.g840d0541.el8.x86_64 4/25 2022-09-13T07:21:09.190 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:17.0.0-14918.g840d0541.el8.x86_64 4/25 2022-09-13T07:21:09.364 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-09-13T07:21:09.438 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rados-2:17.0.0-14918.g840d0541.el8.x86_64 5/25 2022-09-13T07:21:09.588 INFO:teuthology.orchestra.run.smithi036.stdout: Upgrading : librados2-2:17.0.0-14918.g840d0541.el8.x86_64 4/25 2022-09-13T07:21:09.648 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs2-2:17.0.0-14918.g840d0541.el8.x86_64 6/25 2022-09-13T07:21:09.672 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: librados2-2:17.0.0-14918.g840d0541.el8.x86_64 4/25 2022-09-13T07:21:09.966 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-rados-2:17.0.0-14918.g840d0541.el8.x86_64 5/25 2022-09-13T07:21:10.039 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs2-2:17.0.0-14918.g840d0541.el8.x86_64 6/25 2022-09-13T07:21:10.180 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : libcephfs2-2:17.0.0-14918.g840d0541.el8.x86_64 6/25 2022-09-13T07:21:10.207 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librbd1-2:17.0.0-14918.g840d0541.el8.x86_64 7/25 2022-09-13T07:21:10.271 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:17.0.0-14918.g840d0541.el8.x86_64 7/25 2022-09-13T07:21:10.471 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2022-09-13T07:21:10.592 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: libcephfs2-2:17.0.0-14918.g840d0541.el8.x86_64 6/25 2022-09-13T07:21:10.632 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2022-09-13T07:21:10.729 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2022-09-13T07:21:10.755 INFO:teuthology.orchestra.run.smithi036.stdout: Upgrading : librbd1-2:17.0.0-14918.g840d0541.el8.x86_64 7/25 2022-09-13T07:21:10.839 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: librbd1-2:17.0.0-14918.g840d0541.el8.x86_64 7/25 2022-09-13T07:21:10.879 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-argparse-2:17.0.0-14918.g840d0541.el8 10/25 2022-09-13T07:21:11.022 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2022-09-13T07:21:11.107 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cephfs-2:17.0.0-14918.g840d0541.el8.x86_64 11/25 2022-09-13T07:21:11.196 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2022-09-13T07:21:11.264 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2022-09-13T07:21:11.307 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rbd-2:17.0.0-14918.g840d0541.el8.x86_64 12/25 2022-09-13T07:21:11.410 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-ceph-argparse-2:17.0.0-14918.g840d0541.el8 10/25 2022-09-13T07:21:11.490 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libradosstriper1-2:17.0.0-14918.g840d0541.el8.x86_ 13/25 2022-09-13T07:21:11.548 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libradosstriper1-2:17.0.0-14918.g840d0541.el8.x86_ 13/25 2022-09-13T07:21:11.626 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-cephfs-2:17.0.0-14918.g840d0541.el8.x86_64 11/25 2022-09-13T07:21:11.766 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2022-09-13T07:21:11.884 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-rbd-2:17.0.0-14918.g840d0541.el8.x86_64 12/25 2022-09-13T07:21:12.046 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : libradosstriper1-2:17.0.0-14918.g840d0541.el8.x86_ 13/25 2022-09-13T07:21:12.120 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: libradosstriper1-2:17.0.0-14918.g840d0541.el8.x86_ 13/25 2022-09-13T07:21:12.325 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2022-09-13T07:21:12.503 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2022-09-13T07:21:12.657 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librgw2-2:17.0.0-14918.g840d0541.el8.x86_64 16/25 2022-09-13T07:21:12.728 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librgw2-2:17.0.0-14918.g840d0541.el8.x86_64 16/25 2022-09-13T07:21:12.932 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rgw-2:17.0.0-14918.g840d0541.el8.x86_64 17/25 2022-09-13T07:21:13.105 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2022-09-13T07:21:13.134 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2022-09-13T07:21:13.257 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-common-2:17.0.0-14918.g840d0541.el8.x 19/25 2022-09-13T07:21:13.321 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : librgw2-2:17.0.0-14918.g840d0541.el8.x86_64 16/25 2022-09-13T07:21:13.392 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: librgw2-2:17.0.0-14918.g840d0541.el8.x86_64 16/25 2022-09-13T07:21:13.621 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-rgw-2:17.0.0-14918.g840d0541.el8.x86_64 17/25 2022-09-13T07:21:13.816 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2022-09-13T07:21:13.980 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-ceph-common-2:17.0.0-14918.g840d0541.el8.x 19/25 2022-09-13T07:21:17.035 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:17.0.0-14918.g840d0541.el8.x86_64 20/25 2022-09-13T07:21:17.206 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-common-2:17.0.0-14918.g840d0541.el8.x86_64 20/25 2022-09-13T07:21:17.786 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:17.0.0-14918.g840d0541.el8.x86_64 20/25 2022-09-13T07:21:17.984 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-common-2:17.0.0-14918.g840d0541.el8.x86_64 20/25 2022-09-13T07:21:18.039 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-base-2:17.0.0-14918.g840d0541.el8.x86_64 21/25 2022-09-13T07:21:18.112 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-base-2:17.0.0-14918.g840d0541.el8.x86_64 21/25 2022-09-13T07:21:18.213 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-common-2:17.0.0-14918.g840d0541.el8.x86_64 20/25 2022-09-13T07:21:18.289 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-selinux-2:17.0.0-14918.g840d0541.el8.x86_64 22/25 2022-09-13T07:21:18.817 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-common-2:17.0.0-14918.g840d0541.el8.x86_64 20/25 2022-09-13T07:21:19.109 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-base-2:17.0.0-14918.g840d0541.el8.x86_64 21/25 2022-09-13T07:21:19.215 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-base-2:17.0.0-14918.g840d0541.el8.x86_64 21/25 2022-09-13T07:21:19.329 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-selinux-2:17.0.0-14918.g840d0541.el8.x86_64 22/25 2022-09-13T07:21:37.264 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-selinux-2:17.0.0-14918.g840d0541.el8.x86_64 22/25 2022-09-13T07:21:37.265 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /sys 2022-09-13T07:21:37.266 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /proc 2022-09-13T07:21:37.266 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /dev 2022-09-13T07:21:37.266 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /run 2022-09-13T07:21:37.267 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /mnt 2022-09-13T07:21:37.267 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /var/tmp 2022-09-13T07:21:37.268 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /home 2022-09-13T07:21:37.268 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /tmp 2022-09-13T07:21:37.268 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /dev 2022-09-13T07:21:37.269 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:21:39.011 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-radosgw-2:17.0.0-14918.g840d0541.el8.x86_64 23/25 2022-09-13T07:21:39.031 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-14918.g840d0541.el8.x86_64 23/25 2022-09-13T07:21:39.031 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-09-13T07:21:39.144 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-selinux-2:17.0.0-14918.g840d0541.el8.x86_64 22/25 2022-09-13T07:21:39.144 INFO:teuthology.orchestra.run.smithi036.stdout:skipping the directory /sys 2022-09-13T07:21:39.145 INFO:teuthology.orchestra.run.smithi036.stdout:skipping the directory /proc 2022-09-13T07:21:39.145 INFO:teuthology.orchestra.run.smithi036.stdout:skipping the directory /dev 2022-09-13T07:21:39.145 INFO:teuthology.orchestra.run.smithi036.stdout:skipping the directory /run 2022-09-13T07:21:39.145 INFO:teuthology.orchestra.run.smithi036.stdout:skipping the directory /mnt 2022-09-13T07:21:39.146 INFO:teuthology.orchestra.run.smithi036.stdout:skipping the directory /var/tmp 2022-09-13T07:21:39.146 INFO:teuthology.orchestra.run.smithi036.stdout:skipping the directory /home 2022-09-13T07:21:39.146 INFO:teuthology.orchestra.run.smithi036.stdout:skipping the directory /tmp 2022-09-13T07:21:39.146 INFO:teuthology.orchestra.run.smithi036.stdout:skipping the directory /dev 2022-09-13T07:21:39.147 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:21:39.255 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-09-13T07:21:39.255 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2022-09-13T07:21:39.869 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2022-09-13T07:21:39.870 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-base-2:17.0.0-14918.g840d0541.el8.x86_64 1/25 2022-09-13T07:21:39.870 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-common-2:17.0.0-14918.g840d0541.el8.x86_64 2/25 2022-09-13T07:21:39.870 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-radosgw-2:17.0.0-14918.g840d0541.el8.x86_64 3/25 2022-09-13T07:21:39.871 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-selinux-2:17.0.0-14918.g840d0541.el8.x86_64 4/25 2022-09-13T07:21:39.871 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs2-2:17.0.0-14918.g840d0541.el8.x86_64 5/25 2022-09-13T07:21:39.871 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libradosstriper1-2:17.0.0-14918.g840d0541.el8.x86_ 6/25 2022-09-13T07:21:39.872 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librgw2-2:17.0.0-14918.g840d0541.el8.x86_64 7/25 2022-09-13T07:21:39.872 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-argparse-2:17.0.0-14918.g840d0541.el8 8/25 2022-09-13T07:21:39.872 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-common-2:17.0.0-14918.g840d0541.el8.x 9/25 2022-09-13T07:21:39.872 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cephfs-2:17.0.0-14918.g840d0541.el8.x86_64 10/25 2022-09-13T07:21:39.873 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rados-2:17.0.0-14918.g840d0541.el8.x86_64 11/25 2022-09-13T07:21:39.873 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rbd-2:17.0.0-14918.g840d0541.el8.x86_64 12/25 2022-09-13T07:21:39.873 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rgw-2:17.0.0-14918.g840d0541.el8.x86_64 13/25 2022-09-13T07:21:39.873 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2022-09-13T07:21:39.873 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2022-09-13T07:21:39.874 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2022-09-13T07:21:39.874 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2022-09-13T07:21:39.874 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2022-09-13T07:21:39.874 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2022-09-13T07:21:39.875 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2022-09-13T07:21:39.875 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2022-09-13T07:21:39.875 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:17.0.0-14918.g840d0541.el8.x86_64 22/25 2022-09-13T07:21:39.876 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2022-09-13T07:21:39.876 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:17.0.0-14918.g840d0541.el8.x86_64 24/25 2022-09-13T07:21:40.668 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2022-09-13T07:21:40.669 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:21:40.669 INFO:teuthology.orchestra.run.smithi083.stdout:Upgraded: 2022-09-13T07:21:40.669 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:40.669 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:40.670 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:21:40.670 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:40.670 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:40.670 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:40.671 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:40.671 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-09-13T07:21:40.671 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:40.671 INFO:teuthology.orchestra.run.smithi083.stdout: liboath-2.6.2-3.el8.x86_64 2022-09-13T07:21:40.671 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-09-13T07:21:40.672 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:40.672 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-09-13T07:21:40.672 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:40.672 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind-1.3.1-3.el8.x86_64 2022-09-13T07:21:40.673 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-09-13T07:21:40.673 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:40.673 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:40.673 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:40.673 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-09-13T07:21:40.674 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:40.674 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:40.674 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:40.675 INFO:teuthology.orchestra.run.smithi083.stdout: thrift-0.13.0-2.el8.x86_64 2022-09-13T07:21:40.675 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:21:40.675 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:21:40.880 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-test 2022-09-13T07:21:41.048 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-radosgw-2:17.0.0-14918.g840d0541.el8.x86_64 23/25 2022-09-13T07:21:41.070 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-14918.g840d0541.el8.x86_64 23/25 2022-09-13T07:21:41.070 INFO:teuthology.orchestra.run.smithi036.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-09-13T07:21:41.270 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-09-13T07:21:41.270 INFO:teuthology.orchestra.run.smithi036.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2022-09-13T07:21:41.480 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:44 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:21:41.926 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2022-09-13T07:21:41.927 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-base-2:17.0.0-14918.g840d0541.el8.x86_64 1/25 2022-09-13T07:21:41.927 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-common-2:17.0.0-14918.g840d0541.el8.x86_64 2/25 2022-09-13T07:21:41.927 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-radosgw-2:17.0.0-14918.g840d0541.el8.x86_64 3/25 2022-09-13T07:21:41.927 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-selinux-2:17.0.0-14918.g840d0541.el8.x86_64 4/25 2022-09-13T07:21:41.928 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : libcephfs2-2:17.0.0-14918.g840d0541.el8.x86_64 5/25 2022-09-13T07:21:41.928 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : libradosstriper1-2:17.0.0-14918.g840d0541.el8.x86_ 6/25 2022-09-13T07:21:41.928 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : librgw2-2:17.0.0-14918.g840d0541.el8.x86_64 7/25 2022-09-13T07:21:41.928 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-ceph-argparse-2:17.0.0-14918.g840d0541.el8 8/25 2022-09-13T07:21:41.929 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-ceph-common-2:17.0.0-14918.g840d0541.el8.x 9/25 2022-09-13T07:21:41.929 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-cephfs-2:17.0.0-14918.g840d0541.el8.x86_64 10/25 2022-09-13T07:21:41.929 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-rados-2:17.0.0-14918.g840d0541.el8.x86_64 11/25 2022-09-13T07:21:41.929 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-rbd-2:17.0.0-14918.g840d0541.el8.x86_64 12/25 2022-09-13T07:21:41.930 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-rgw-2:17.0.0-14918.g840d0541.el8.x86_64 13/25 2022-09-13T07:21:41.930 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2022-09-13T07:21:41.930 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2022-09-13T07:21:41.930 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2022-09-13T07:21:41.930 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2022-09-13T07:21:41.931 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2022-09-13T07:21:41.931 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2022-09-13T07:21:41.931 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2022-09-13T07:21:41.931 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2022-09-13T07:21:41.932 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : librados2-2:17.0.0-14918.g840d0541.el8.x86_64 22/25 2022-09-13T07:21:41.932 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2022-09-13T07:21:41.932 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : librbd1-2:17.0.0-14918.g840d0541.el8.x86_64 24/25 2022-09-13T07:21:42.539 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:21:42.540 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:21:42.540 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2022-09-13T07:21:42.541 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:21:42.541 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:21:42.541 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test x86_64 2:17.0.0-14918.g840d0541.el8 ceph 54 M 2022-09-13T07:21:42.541 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2022-09-13T07:21:42.542 INFO:teuthology.orchestra.run.smithi083.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2022-09-13T07:21:42.542 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite x86_64 2:17.0.0-14918.g840d0541.el8 ceph 137 k 2022-09-13T07:21:42.542 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-09-13T07:21:42.542 INFO:teuthology.orchestra.run.smithi083.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-09-13T07:21:42.543 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-PowerTools 71 k 2022-09-13T07:21:42.543 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:21:42.543 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:21:42.543 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:21:42.544 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2022-09-13T07:21:42.544 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:21:42.545 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 55 M 2022-09-13T07:21:42.545 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 238 M 2022-09-13T07:21:42.546 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:21:42.628 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 2.4 MB/s | 202 kB 00:00 2022-09-13T07:21:42.677 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2022-09-13T07:21:42.711 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 9.4 MB/s | 323 kB 00:00 2022-09-13T07:21:42.762 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2022-09-13T07:21:42.829 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): libcephsqlite-17.0.0-14918.g840d0541.el8 478 kB/s | 137 kB 00:00 2022-09-13T07:21:42.862 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2022-09-13T07:21:42.863 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:21:42.863 INFO:teuthology.orchestra.run.smithi036.stdout:Upgraded: 2022-09-13T07:21:42.864 INFO:teuthology.orchestra.run.smithi036.stdout: librados2-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:42.864 INFO:teuthology.orchestra.run.smithi036.stdout: librbd1-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:42.864 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:21:42.865 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-base-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:42.865 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-common-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:42.866 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-radosgw-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:42.866 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-selinux-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:42.866 INFO:teuthology.orchestra.run.smithi036.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-09-13T07:21:42.867 INFO:teuthology.orchestra.run.smithi036.stdout: libcephfs2-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:42.867 INFO:teuthology.orchestra.run.smithi036.stdout: liboath-2.6.2-3.el8.x86_64 2022-09-13T07:21:42.867 INFO:teuthology.orchestra.run.smithi036.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-09-13T07:21:42.868 INFO:teuthology.orchestra.run.smithi036.stdout: libradosstriper1-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:42.868 INFO:teuthology.orchestra.run.smithi036.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-09-13T07:21:42.869 INFO:teuthology.orchestra.run.smithi036.stdout: librgw2-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:42.869 INFO:teuthology.orchestra.run.smithi036.stdout: libunwind-1.3.1-3.el8.x86_64 2022-09-13T07:21:42.869 INFO:teuthology.orchestra.run.smithi036.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-09-13T07:21:42.870 INFO:teuthology.orchestra.run.smithi036.stdout: python3-ceph-argparse-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:42.870 INFO:teuthology.orchestra.run.smithi036.stdout: python3-ceph-common-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:42.870 INFO:teuthology.orchestra.run.smithi036.stdout: python3-cephfs-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:42.871 INFO:teuthology.orchestra.run.smithi036.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-09-13T07:21:42.871 INFO:teuthology.orchestra.run.smithi036.stdout: python3-rados-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:42.871 INFO:teuthology.orchestra.run.smithi036.stdout: python3-rbd-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:42.871 INFO:teuthology.orchestra.run.smithi036.stdout: python3-rgw-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:42.872 INFO:teuthology.orchestra.run.smithi036.stdout: thrift-0.13.0-2.el8.x86_64 2022-09-13T07:21:42.872 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:21:42.872 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:21:43.055 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install ceph-test 2022-09-13T07:21:43.680 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:00:45 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:21:44.770 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:21:44.771 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:21:44.771 INFO:teuthology.orchestra.run.smithi036.stdout: Package Arch Version Repository Size 2022-09-13T07:21:44.771 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:21:44.772 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:21:44.772 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-test x86_64 2:17.0.0-14918.g840d0541.el8 ceph 54 M 2022-09-13T07:21:44.772 INFO:teuthology.orchestra.run.smithi036.stdout:Installing dependencies: 2022-09-13T07:21:44.772 INFO:teuthology.orchestra.run.smithi036.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2022-09-13T07:21:44.773 INFO:teuthology.orchestra.run.smithi036.stdout: libcephsqlite x86_64 2:17.0.0-14918.g840d0541.el8 ceph 137 k 2022-09-13T07:21:44.773 INFO:teuthology.orchestra.run.smithi036.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-09-13T07:21:44.773 INFO:teuthology.orchestra.run.smithi036.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-09-13T07:21:44.773 INFO:teuthology.orchestra.run.smithi036.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-PowerTools 71 k 2022-09-13T07:21:44.774 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:21:44.774 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:21:44.774 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:21:44.774 INFO:teuthology.orchestra.run.smithi036.stdout:Install 6 Packages 2022-09-13T07:21:44.775 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:21:44.776 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 55 M 2022-09-13T07:21:44.776 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 238 M 2022-09-13T07:21:44.777 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:21:44.849 INFO:teuthology.orchestra.run.smithi036.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 2.7 MB/s | 202 kB 00:00 2022-09-13T07:21:44.899 INFO:teuthology.orchestra.run.smithi036.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2022-09-13T07:21:44.958 INFO:teuthology.orchestra.run.smithi036.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2022-09-13T07:21:44.984 INFO:teuthology.orchestra.run.smithi036.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 71 kB 00:00 2022-09-13T07:21:45.042 INFO:teuthology.orchestra.run.smithi036.stdout:(5/6): libcephsqlite-17.0.0-14918.g840d0541.el8 513 kB/s | 137 kB 00:00 2022-09-13T07:21:47.789 INFO:teuthology.orchestra.run.smithi036.stdout:(6/6): ceph-test-17.0.0-14918.g840d0541.el8.x86 18 MB/s | 54 MB 00:03 2022-09-13T07:21:47.792 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:21:47.792 INFO:teuthology.orchestra.run.smithi036.stdout:Total 18 MB/s | 55 MB 00:03 2022-09-13T07:21:47.834 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:21:47.856 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:21:47.857 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:21:48.224 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:21:48.227 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:21:48.779 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:21:48.981 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2022-09-13T07:21:49.146 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2022-09-13T07:21:49.316 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2022-09-13T07:21:49.394 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2022-09-13T07:21:49.605 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2022-09-13T07:21:49.750 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : libcephsqlite-2:17.0.0-14918.g840d0541.el8.x86_64 5/6 2022-09-13T07:21:50.491 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): ceph-test-17.0.0-14918.g840d0541.el8.x86 6.8 MB/s | 54 MB 00:07 2022-09-13T07:21:50.494 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:21:50.494 INFO:teuthology.orchestra.run.smithi083.stdout:Total 6.9 MB/s | 55 MB 00:07 2022-09-13T07:21:50.535 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:21:50.556 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:21:50.556 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:21:50.880 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:21:50.883 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:21:51.354 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:21:51.588 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2022-09-13T07:21:51.745 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2022-09-13T07:21:51.901 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2022-09-13T07:21:51.988 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2022-09-13T07:21:52.171 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2022-09-13T07:21:52.334 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephsqlite-2:17.0.0-14918.g840d0541.el8.x86_64 5/6 2022-09-13T07:21:55.244 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: libcephsqlite-2:17.0.0-14918.g840d0541.el8.x86_64 5/6 2022-09-13T07:21:55.514 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-test-2:17.0.0-14918.g840d0541.el8.x86_64 6/6 2022-09-13T07:21:55.780 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-test-2:17.0.0-14918.g840d0541.el8.x86_64 6/6 2022-09-13T07:21:55.781 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-test-2:17.0.0-14918.g840d0541.el8.x86_64 1/6 2022-09-13T07:21:55.781 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : libcephsqlite-2:17.0.0-14918.g840d0541.el8.x86_64 2/6 2022-09-13T07:21:55.782 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2022-09-13T07:21:55.782 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2022-09-13T07:21:55.782 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2022-09-13T07:21:56.236 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2022-09-13T07:21:56.237 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:21:56.237 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:21:56.238 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-test-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:56.238 INFO:teuthology.orchestra.run.smithi036.stdout: jq-1.6-3.el8.x86_64 2022-09-13T07:21:56.238 INFO:teuthology.orchestra.run.smithi036.stdout: libcephsqlite-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:56.238 INFO:teuthology.orchestra.run.smithi036.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-09-13T07:21:56.239 INFO:teuthology.orchestra.run.smithi036.stdout: socat-1.7.4.1-1.el8.x86_64 2022-09-13T07:21:56.239 INFO:teuthology.orchestra.run.smithi036.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2022-09-13T07:21:56.239 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:21:56.239 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:21:56.455 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install ceph 2022-09-13T07:21:57.049 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:00:59 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:21:57.482 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephsqlite-2:17.0.0-14918.g840d0541.el8.x86_64 5/6 2022-09-13T07:21:57.731 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-test-2:17.0.0-14918.g840d0541.el8.x86_64 6/6 2022-09-13T07:21:57.977 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-test-2:17.0.0-14918.g840d0541.el8.x86_64 6/6 2022-09-13T07:21:57.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-test-2:17.0.0-14918.g840d0541.el8.x86_64 1/6 2022-09-13T07:21:57.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephsqlite-2:17.0.0-14918.g840d0541.el8.x86_64 2/6 2022-09-13T07:21:57.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2022-09-13T07:21:57.979 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2022-09-13T07:21:57.979 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2022-09-13T07:21:58.134 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:21:58.137 INFO:teuthology.orchestra.run.smithi036.stdout:======================================================================================== 2022-09-13T07:21:58.138 INFO:teuthology.orchestra.run.smithi036.stdout: Package Arch Version Repository Size 2022-09-13T07:21:58.138 INFO:teuthology.orchestra.run.smithi036.stdout:======================================================================================== 2022-09-13T07:21:58.138 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:21:58.138 INFO:teuthology.orchestra.run.smithi036.stdout: ceph x86_64 2:17.0.0-14918.g840d0541.el8 ceph 6.4 k 2022-09-13T07:21:58.139 INFO:teuthology.orchestra.run.smithi036.stdout:Installing dependencies: 2022-09-13T07:21:58.139 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mds x86_64 2:17.0.0-14918.g840d0541.el8 ceph 2.3 M 2022-09-13T07:21:58.139 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr x86_64 2:17.0.0-14918.g840d0541.el8 ceph 1.5 M 2022-09-13T07:21:58.139 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-modules-core noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 233 k 2022-09-13T07:21:58.139 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mon x86_64 2:17.0.0-14918.g840d0541.el8 ceph 3.9 M 2022-09-13T07:21:58.140 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-osd x86_64 2:17.0.0-14918.g840d0541.el8 ceph 17 M 2022-09-13T07:21:58.140 INFO:teuthology.orchestra.run.smithi036.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-09-13T07:21:58.140 INFO:teuthology.orchestra.run.smithi036.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-09-13T07:21:58.140 INFO:teuthology.orchestra.run.smithi036.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2022-09-13T07:21:58.141 INFO:teuthology.orchestra.run.smithi036.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-09-13T07:21:58.141 INFO:teuthology.orchestra.run.smithi036.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-09-13T07:21:58.141 INFO:teuthology.orchestra.run.smithi036.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2022-09-13T07:21:58.141 INFO:teuthology.orchestra.run.smithi036.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-09-13T07:21:58.141 INFO:teuthology.orchestra.run.smithi036.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-09-13T07:21:58.142 INFO:teuthology.orchestra.run.smithi036.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-09-13T07:21:58.142 INFO:teuthology.orchestra.run.smithi036.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-09-13T07:21:58.142 INFO:teuthology.orchestra.run.smithi036.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-09-13T07:21:58.143 INFO:teuthology.orchestra.run.smithi036.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-09-13T07:21:58.143 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-09-13T07:21:58.143 INFO:teuthology.orchestra.run.smithi036.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-09-13T07:21:58.144 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-09-13T07:21:58.144 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2022-09-13T07:21:58.144 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-09-13T07:21:58.144 INFO:teuthology.orchestra.run.smithi036.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-09-13T07:21:58.145 INFO:teuthology.orchestra.run.smithi036.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-09-13T07:21:58.145 INFO:teuthology.orchestra.run.smithi036.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-09-13T07:21:58.145 INFO:teuthology.orchestra.run.smithi036.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-09-13T07:21:58.145 INFO:teuthology.orchestra.run.smithi036.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-09-13T07:21:58.145 INFO:teuthology.orchestra.run.smithi036.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-09-13T07:21:58.146 INFO:teuthology.orchestra.run.smithi036.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-09-13T07:21:58.146 INFO:teuthology.orchestra.run.smithi036.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-09-13T07:21:58.146 INFO:teuthology.orchestra.run.smithi036.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-09-13T07:21:58.146 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:21:58.147 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:21:58.147 INFO:teuthology.orchestra.run.smithi036.stdout:======================================================================================== 2022-09-13T07:21:58.147 INFO:teuthology.orchestra.run.smithi036.stdout:Install 32 Packages 2022-09-13T07:21:58.147 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:21:58.148 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 28 M 2022-09-13T07:21:58.148 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 110 M 2022-09-13T07:21:58.149 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:21:58.265 INFO:teuthology.orchestra.run.smithi036.stdout:(1/32): ceph-17.0.0-14918.g840d0541.el8.x86_64. 53 kB/s | 6.4 kB 00:00 2022-09-13T07:21:58.421 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2022-09-13T07:21:58.422 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:21:58.422 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:21:58.422 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:58.422 INFO:teuthology.orchestra.run.smithi083.stdout: jq-1.6-3.el8.x86_64 2022-09-13T07:21:58.423 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:21:58.423 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-09-13T07:21:58.423 INFO:teuthology.orchestra.run.smithi083.stdout: socat-1.7.4.1-1.el8.x86_64 2022-09-13T07:21:58.424 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2022-09-13T07:21:58.424 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:21:58.424 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:21:58.599 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph 2022-09-13T07:21:59.107 INFO:teuthology.orchestra.run.smithi036.stdout:(2/32): ceph-mgr-17.0.0-14918.g840d0541.el8.x86 1.6 MB/s | 1.5 MB 00:00 2022-09-13T07:21:59.166 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:02 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:21:59.315 INFO:teuthology.orchestra.run.smithi036.stdout:(3/32): ceph-mds-17.0.0-14918.g840d0541.el8.x86 1.9 MB/s | 2.3 MB 00:01 2022-09-13T07:21:59.849 INFO:teuthology.orchestra.run.smithi036.stdout:(4/32): ceph-mon-17.0.0-14918.g840d0541.el8.x86 2.4 MB/s | 3.9 MB 00:01 2022-09-13T07:21:59.891 INFO:teuthology.orchestra.run.smithi036.stdout:(5/32): ceph-mgr-modules-core-17.0.0-14918.g840 405 kB/s | 233 kB 00:00 2022-09-13T07:21:59.925 INFO:teuthology.orchestra.run.smithi036.stdout:(6/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.0 MB/s | 157 kB 00:00 2022-09-13T07:21:59.951 INFO:teuthology.orchestra.run.smithi036.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.2 MB/s | 103 kB 00:00 2022-09-13T07:21:59.967 INFO:teuthology.orchestra.run.smithi036.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 521 kB/s | 39 kB 00:00 2022-09-13T07:22:00.011 INFO:teuthology.orchestra.run.smithi036.stdout:(9/32): python3-pytz-2017.2-9.el8.noarch.rpm 888 kB/s | 54 kB 00:00 2022-09-13T07:22:00.043 INFO:teuthology.orchestra.run.smithi036.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 7.5 MB/s | 237 kB 00:00 2022-09-13T07:22:00.068 INFO:teuthology.orchestra.run.smithi036.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 4.4 MB/s | 457 kB 00:00 2022-09-13T07:22:00.093 INFO:teuthology.orchestra.run.smithi036.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2022-09-13T07:22:00.119 INFO:teuthology.orchestra.run.smithi036.stdout:(13/32): python3-cryptography-3.2.1-5.el8.x86_6 7.2 MB/s | 559 kB 00:00 2022-09-13T07:22:00.179 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:22:00.183 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2022-09-13T07:22:00.183 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2022-09-13T07:22:00.183 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2022-09-13T07:22:00.184 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:22:00.184 INFO:teuthology.orchestra.run.smithi083.stdout: ceph x86_64 2:17.0.0-14918.g840d0541.el8 ceph 6.4 k 2022-09-13T07:22:00.184 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2022-09-13T07:22:00.185 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds x86_64 2:17.0.0-14918.g840d0541.el8 ceph 2.3 M 2022-09-13T07:22:00.185 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr x86_64 2:17.0.0-14918.g840d0541.el8 ceph 1.5 M 2022-09-13T07:22:00.185 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 233 k 2022-09-13T07:22:00.185 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon x86_64 2:17.0.0-14918.g840d0541.el8 ceph 3.9 M 2022-09-13T07:22:00.186 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd x86_64 2:17.0.0-14918.g840d0541.el8 ceph 17 M 2022-09-13T07:22:00.186 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-09-13T07:22:00.186 INFO:teuthology.orchestra.run.smithi083.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-09-13T07:22:00.186 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2022-09-13T07:22:00.187 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-09-13T07:22:00.187 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-09-13T07:22:00.188 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2022-09-13T07:22:00.188 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-09-13T07:22:00.189 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-09-13T07:22:00.189 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-09-13T07:22:00.189 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-09-13T07:22:00.189 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-09-13T07:22:00.190 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-09-13T07:22:00.190 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-09-13T07:22:00.190 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-09-13T07:22:00.191 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-09-13T07:22:00.191 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2022-09-13T07:22:00.191 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-09-13T07:22:00.192 INFO:teuthology.orchestra.run.smithi083.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-09-13T07:22:00.192 INFO:teuthology.orchestra.run.smithi083.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-09-13T07:22:00.192 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-09-13T07:22:00.192 INFO:teuthology.orchestra.run.smithi083.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-09-13T07:22:00.193 INFO:teuthology.orchestra.run.smithi083.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-09-13T07:22:00.193 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-09-13T07:22:00.193 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-09-13T07:22:00.193 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-09-13T07:22:00.194 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-09-13T07:22:00.194 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:00.194 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:22:00.195 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2022-09-13T07:22:00.195 INFO:teuthology.orchestra.run.smithi083.stdout:Install 32 Packages 2022-09-13T07:22:00.195 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:00.196 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 28 M 2022-09-13T07:22:00.197 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 110 M 2022-09-13T07:22:00.197 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:22:00.252 INFO:teuthology.orchestra.run.smithi036.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 275 kB/s | 44 kB 00:00 2022-09-13T07:22:00.319 INFO:teuthology.orchestra.run.smithi036.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 926 kB/s | 185 kB 00:00 2022-09-13T07:22:00.360 INFO:teuthology.orchestra.run.smithi083.stdout:(1/32): ceph-17.0.0-14918.g840d0541.el8.x86_64. 37 kB/s | 6.4 kB 00:00 2022-09-13T07:22:00.378 INFO:teuthology.orchestra.run.smithi036.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2022-09-13T07:22:00.595 INFO:teuthology.orchestra.run.smithi036.stdout:(17/32): ceph-osd-17.0.0-14918.g840d0541.el8.x8 11 MB/s | 17 MB 00:01 2022-09-13T07:22:00.602 INFO:teuthology.orchestra.run.smithi083.stdout:(2/32): ceph-mds-17.0.0-14918.g840d0541.el8.x86 5.5 MB/s | 2.3 MB 00:00 2022-09-13T07:22:00.620 INFO:teuthology.orchestra.run.smithi036.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 46 kB/s | 11 kB 00:00 2022-09-13T07:22:00.646 INFO:teuthology.orchestra.run.smithi036.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.2 MB/s | 384 kB 00:00 2022-09-13T07:22:00.671 INFO:teuthology.orchestra.run.smithi036.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 233 kB/s | 18 kB 00:00 2022-09-13T07:22:00.704 INFO:teuthology.orchestra.run.smithi036.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2022-09-13T07:22:00.763 INFO:teuthology.orchestra.run.smithi036.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2022-09-13T07:22:00.780 INFO:teuthology.orchestra.run.smithi036.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 207 kB/s | 16 kB 00:00 2022-09-13T07:22:00.796 INFO:teuthology.orchestra.run.smithi036.stdout:(24/32): python3-logutils-0.3.5-11.el8.noarch.r 276 kB/s | 49 kB 00:00 2022-09-13T07:22:00.816 INFO:teuthology.orchestra.run.smithi036.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 371 kB/s | 19 kB 00:00 2022-09-13T07:22:00.830 INFO:teuthology.orchestra.run.smithi036.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 474 kB/s | 24 kB 00:00 2022-09-13T07:22:00.881 INFO:teuthology.orchestra.run.smithi036.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 348 kB/s | 29 kB 00:00 2022-09-13T07:22:00.885 INFO:teuthology.orchestra.run.smithi083.stdout:(3/32): ceph-mon-17.0.0-14918.g840d0541.el8.x86 7.3 MB/s | 3.9 MB 00:00 2022-09-13T07:22:00.898 INFO:teuthology.orchestra.run.smithi036.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 328 kB/s | 27 kB 00:00 2022-09-13T07:22:00.923 INFO:teuthology.orchestra.run.smithi036.stdout:(29/32): python3-waitress-1.2.1-2.el8.1.noarch. 2.6 MB/s | 241 kB 00:00 2022-09-13T07:22:00.956 INFO:teuthology.orchestra.run.smithi036.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.4 MB/s | 85 kB 00:00 2022-09-13T07:22:00.973 INFO:teuthology.orchestra.run.smithi036.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 459 kB/s | 23 kB 00:00 2022-09-13T07:22:00.986 INFO:teuthology.orchestra.run.smithi083.stdout:(4/32): ceph-mgr-modules-core-17.0.0-14918.g840 2.3 MB/s | 233 kB 00:00 2022-09-13T07:22:01.015 INFO:teuthology.orchestra.run.smithi036.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2022-09-13T07:22:01.019 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:01.019 INFO:teuthology.orchestra.run.smithi036.stdout:Total 9.8 MB/s | 28 MB 00:02 2022-09-13T07:22:01.044 INFO:teuthology.orchestra.run.smithi083.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2022-09-13T07:22:01.070 INFO:teuthology.orchestra.run.smithi083.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.5 MB/s | 39 kB 00:00 2022-09-13T07:22:01.095 INFO:teuthology.orchestra.run.smithi083.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.0 MB/s | 103 kB 00:00 2022-09-13T07:22:01.104 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:22:01.121 INFO:teuthology.orchestra.run.smithi083.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2022-09-13T07:22:01.145 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:22:01.146 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:22:01.155 INFO:teuthology.orchestra.run.smithi083.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 13 MB/s | 457 kB 00:00 2022-09-13T07:22:01.179 INFO:teuthology.orchestra.run.smithi083.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.3 MB/s | 237 kB 00:00 2022-09-13T07:22:01.222 INFO:teuthology.orchestra.run.smithi083.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 13 MB/s | 559 kB 00:00 2022-09-13T07:22:01.263 INFO:teuthology.orchestra.run.smithi083.stdout:(12/32): ceph-mgr-17.0.0-14918.g840d0541.el8.x8 1.4 MB/s | 1.5 MB 00:01 2022-09-13T07:22:01.288 INFO:teuthology.orchestra.run.smithi083.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.6 MB/s | 109 kB 00:00 2022-09-13T07:22:01.472 INFO:teuthology.orchestra.run.smithi083.stdout:(14/32): ceph-osd-17.0.0-14918.g840d0541.el8.x8 19 MB/s | 17 MB 00:00 2022-09-13T07:22:01.488 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:22:01.489 INFO:teuthology.orchestra.run.smithi083.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 193 kB/s | 44 kB 00:00 2022-09-13T07:22:01.491 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:22:01.639 INFO:teuthology.orchestra.run.smithi083.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 528 kB/s | 185 kB 00:00 2022-09-13T07:22:01.667 INFO:teuthology.orchestra.run.smithi083.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.1 MB/s | 384 kB 00:00 2022-09-13T07:22:01.690 INFO:teuthology.orchestra.run.smithi083.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 795 kB/s | 173 kB 00:00 2022-09-13T07:22:01.707 INFO:teuthology.orchestra.run.smithi083.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 165 kB/s | 11 kB 00:00 2022-09-13T07:22:01.724 INFO:teuthology.orchestra.run.smithi083.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 299 kB/s | 18 kB 00:00 2022-09-13T07:22:01.741 INFO:teuthology.orchestra.run.smithi083.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 966 kB/s | 49 kB 00:00 2022-09-13T07:22:01.766 INFO:teuthology.orchestra.run.smithi083.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 998 kB/s | 59 kB 00:00 2022-09-13T07:22:01.791 INFO:teuthology.orchestra.run.smithi083.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2022-09-13T07:22:01.808 INFO:teuthology.orchestra.run.smithi083.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.3 MB/s | 283 kB 00:00 2022-09-13T07:22:01.825 INFO:teuthology.orchestra.run.smithi083.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 317 kB/s | 19 kB 00:00 2022-09-13T07:22:01.842 INFO:teuthology.orchestra.run.smithi083.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 472 kB/s | 24 kB 00:00 2022-09-13T07:22:01.859 INFO:teuthology.orchestra.run.smithi083.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 577 kB/s | 29 kB 00:00 2022-09-13T07:22:01.876 INFO:teuthology.orchestra.run.smithi083.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 544 kB/s | 27 kB 00:00 2022-09-13T07:22:01.902 INFO:teuthology.orchestra.run.smithi083.stdout:(29/32): python3-waitress-1.2.1-2.el8.1.noarch. 4.0 MB/s | 241 kB 00:00 2022-09-13T07:22:01.919 INFO:teuthology.orchestra.run.smithi083.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.2 MB/s | 251 kB 00:00 2022-09-13T07:22:01.944 INFO:teuthology.orchestra.run.smithi083.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.2 MB/s | 85 kB 00:00 2022-09-13T07:22:01.969 INFO:teuthology.orchestra.run.smithi083.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 343 kB/s | 23 kB 00:00 2022-09-13T07:22:01.972 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:01.972 INFO:teuthology.orchestra.run.smithi083.stdout:Total 16 MB/s | 28 MB 00:01 2022-09-13T07:22:02.050 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:22:02.086 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:22:02.087 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:22:02.424 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:22:02.424 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:22:02.837 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:22:03.083 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2022-09-13T07:22:03.335 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2022-09-13T07:22:03.550 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:22:03.594 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2022-09-13T07:22:03.813 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2022-09-13T07:22:03.822 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2022-09-13T07:22:03.970 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2022-09-13T07:22:04.024 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2022-09-13T07:22:04.169 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2022-09-13T07:22:04.253 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2022-09-13T07:22:04.443 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2022-09-13T07:22:04.473 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2022-09-13T07:22:04.629 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2022-09-13T07:22:04.655 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2022-09-13T07:22:04.792 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2022-09-13T07:22:04.850 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2022-09-13T07:22:05.019 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2022-09-13T07:22:05.086 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2022-09-13T07:22:05.221 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2022-09-13T07:22:05.316 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2022-09-13T07:22:05.430 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2022-09-13T07:22:05.565 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2022-09-13T07:22:05.650 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2022-09-13T07:22:05.873 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2022-09-13T07:22:06.022 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2022-09-13T07:22:06.114 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2022-09-13T07:22:06.260 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2022-09-13T07:22:06.464 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2022-09-13T07:22:06.521 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2022-09-13T07:22:06.700 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2022-09-13T07:22:06.751 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2022-09-13T07:22:06.906 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2022-09-13T07:22:06.949 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-09-13T07:22:07.180 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2022-09-13T07:22:07.203 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2022-09-13T07:22:07.375 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-09-13T07:22:07.396 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2022-09-13T07:22:07.600 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2022-09-13T07:22:07.673 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2022-09-13T07:22:07.819 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2022-09-13T07:22:07.854 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2022-09-13T07:22:08.026 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2022-09-13T07:22:08.071 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2022-09-13T07:22:08.212 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2022-09-13T07:22:08.269 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2022-09-13T07:22:08.479 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2022-09-13T07:22:08.480 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2022-09-13T07:22:08.685 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2022-09-13T07:22:08.856 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2022-09-13T07:22:08.875 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2022-09-13T07:22:09.181 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2022-09-13T07:22:09.231 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2022-09-13T07:22:09.521 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-14918.g840d0541.el8 27/32 2022-09-13T07:22:09.528 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2022-09-13T07:22:09.669 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-mgr-2:17.0.0-14918.g840d0541.el8.x86_64 28/32 2022-09-13T07:22:09.875 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-14918.g840d0541.el8 27/32 2022-09-13T07:22:10.023 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-2:17.0.0-14918.g840d0541.el8.x86_64 28/32 2022-09-13T07:22:11.266 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-mgr-2:17.0.0-14918.g840d0541.el8.x86_64 28/32 2022-09-13T07:22:11.516 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-2:17.0.0-14918.g840d0541.el8.x86_64 28/32 2022-09-13T07:22:11.594 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-osd-2:17.0.0-14918.g840d0541.el8.x86_64 29/32 2022-09-13T07:22:11.706 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-osd-2:17.0.0-14918.g840d0541.el8.x86_64 29/32 2022-09-13T07:22:12.002 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-osd-2:17.0.0-14918.g840d0541.el8.x86_64 29/32 2022-09-13T07:22:12.090 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-osd-2:17.0.0-14918.g840d0541.el8.x86_64 29/32 2022-09-13T07:22:12.189 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-mon-2:17.0.0-14918.g840d0541.el8.x86_64 30/32 2022-09-13T07:22:12.414 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-mon-2:17.0.0-14918.g840d0541.el8.x86_64 30/32 2022-09-13T07:22:12.580 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-mds-2:17.0.0-14918.g840d0541.el8.x86_64 31/32 2022-09-13T07:22:12.599 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-mds-2:17.0.0-14918.g840d0541.el8.x86_64 31/32 2022-09-13T07:22:12.746 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-2:17.0.0-14918.g840d0541.el8.x86_64 32/32 2022-09-13T07:22:12.891 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mon-2:17.0.0-14918.g840d0541.el8.x86_64 30/32 2022-09-13T07:22:13.108 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mon-2:17.0.0-14918.g840d0541.el8.x86_64 30/32 2022-09-13T07:22:13.138 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-2:17.0.0-14918.g840d0541.el8.x86_64 32/32 2022-09-13T07:22:13.139 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-2:17.0.0-14918.g840d0541.el8.x86_64 1/32 2022-09-13T07:22:13.139 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-mds-2:17.0.0-14918.g840d0541.el8.x86_64 2/32 2022-09-13T07:22:13.140 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-mgr-2:17.0.0-14918.g840d0541.el8.x86_64 3/32 2022-09-13T07:22:13.140 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-mon-2:17.0.0-14918.g840d0541.el8.x86_64 4/32 2022-09-13T07:22:13.141 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-osd-2:17.0.0-14918.g840d0541.el8.x86_64 5/32 2022-09-13T07:22:13.141 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-14918.g840d0541.el8 6/32 2022-09-13T07:22:13.141 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2022-09-13T07:22:13.142 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2022-09-13T07:22:13.142 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2022-09-13T07:22:13.142 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2022-09-13T07:22:13.142 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2022-09-13T07:22:13.143 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2022-09-13T07:22:13.143 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2022-09-13T07:22:13.143 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2022-09-13T07:22:13.143 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2022-09-13T07:22:13.144 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2022-09-13T07:22:13.144 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-09-13T07:22:13.144 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2022-09-13T07:22:13.144 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2022-09-13T07:22:13.145 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2022-09-13T07:22:13.145 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2022-09-13T07:22:13.145 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2022-09-13T07:22:13.146 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2022-09-13T07:22:13.146 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2022-09-13T07:22:13.146 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2022-09-13T07:22:13.146 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2022-09-13T07:22:13.147 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2022-09-13T07:22:13.147 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2022-09-13T07:22:13.147 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2022-09-13T07:22:13.147 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2022-09-13T07:22:13.148 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2022-09-13T07:22:13.265 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mds-2:17.0.0-14918.g840d0541.el8.x86_64 31/32 2022-09-13T07:22:13.282 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mds-2:17.0.0-14918.g840d0541.el8.x86_64 31/32 2022-09-13T07:22:13.405 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-2:17.0.0-14918.g840d0541.el8.x86_64 32/32 2022-09-13T07:22:13.783 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-2:17.0.0-14918.g840d0541.el8.x86_64 32/32 2022-09-13T07:22:13.784 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-2:17.0.0-14918.g840d0541.el8.x86_64 1/32 2022-09-13T07:22:13.784 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mds-2:17.0.0-14918.g840d0541.el8.x86_64 2/32 2022-09-13T07:22:13.784 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-2:17.0.0-14918.g840d0541.el8.x86_64 3/32 2022-09-13T07:22:13.785 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mon-2:17.0.0-14918.g840d0541.el8.x86_64 4/32 2022-09-13T07:22:13.785 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-osd-2:17.0.0-14918.g840d0541.el8.x86_64 5/32 2022-09-13T07:22:13.785 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-14918.g840d0541.el8 6/32 2022-09-13T07:22:13.785 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2022-09-13T07:22:13.785 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2022-09-13T07:22:13.786 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2022-09-13T07:22:13.786 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2022-09-13T07:22:13.786 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2022-09-13T07:22:13.786 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2022-09-13T07:22:13.787 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2022-09-13T07:22:13.787 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2022-09-13T07:22:13.787 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2022-09-13T07:22:13.787 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2022-09-13T07:22:13.788 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-09-13T07:22:13.788 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2022-09-13T07:22:13.788 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2022-09-13T07:22:13.788 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2022-09-13T07:22:13.788 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2022-09-13T07:22:13.789 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2022-09-13T07:22:13.789 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2022-09-13T07:22:13.789 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2022-09-13T07:22:13.789 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2022-09-13T07:22:13.789 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2022-09-13T07:22:13.790 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2022-09-13T07:22:13.790 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2022-09-13T07:22:13.790 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2022-09-13T07:22:13.790 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2022-09-13T07:22:13.791 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2022-09-13T07:22:14.658 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2022-09-13T07:22:14.659 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:14.659 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:22:14.659 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:22:14.659 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:22:14.660 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:22:14.660 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:14.660 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:22:14.660 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:22:14.661 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-09-13T07:22:14.661 INFO:teuthology.orchestra.run.smithi083.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-09-13T07:22:14.661 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-09-13T07:22:14.661 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-09-13T07:22:14.661 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-09-13T07:22:14.662 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-09-13T07:22:14.662 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-6.2-6.el8.noarch 2022-09-13T07:22:14.662 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-09-13T07:22:14.662 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-09-13T07:22:14.663 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako-1.0.6-13.el8.noarch 2022-09-13T07:22:14.663 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-09-13T07:22:14.663 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-09-13T07:22:14.663 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-09-13T07:22:14.664 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend-2.6-1.el8.noarch 2022-09-13T07:22:14.664 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-09-13T07:22:14.664 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser-2.14-14.el8.noarch 2022-09-13T07:22:14.664 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz-2017.2-9.el8.noarch 2022-09-13T07:22:14.665 INFO:teuthology.orchestra.run.smithi083.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-09-13T07:22:14.665 INFO:teuthology.orchestra.run.smithi083.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-09-13T07:22:14.665 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-09-13T07:22:14.665 INFO:teuthology.orchestra.run.smithi083.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-09-13T07:22:14.665 INFO:teuthology.orchestra.run.smithi083.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-09-13T07:22:14.666 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-09-13T07:22:14.666 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-09-13T07:22:14.666 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-09-13T07:22:14.666 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-09-13T07:22:14.667 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:14.667 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:22:14.760 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2022-09-13T07:22:14.760 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:14.760 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:22:14.761 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:22:14.761 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mds-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:22:14.761 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:22:14.762 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-modules-core-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:14.762 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mon-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:22:14.762 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-osd-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:22:14.762 INFO:teuthology.orchestra.run.smithi036.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-09-13T07:22:14.763 INFO:teuthology.orchestra.run.smithi036.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-09-13T07:22:14.763 INFO:teuthology.orchestra.run.smithi036.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-09-13T07:22:14.763 INFO:teuthology.orchestra.run.smithi036.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-09-13T07:22:14.763 INFO:teuthology.orchestra.run.smithi036.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-09-13T07:22:14.764 INFO:teuthology.orchestra.run.smithi036.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-09-13T07:22:14.764 INFO:teuthology.orchestra.run.smithi036.stdout: python3-jaraco-6.2-6.el8.noarch 2022-09-13T07:22:14.764 INFO:teuthology.orchestra.run.smithi036.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-09-13T07:22:14.765 INFO:teuthology.orchestra.run.smithi036.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-09-13T07:22:14.765 INFO:teuthology.orchestra.run.smithi036.stdout: python3-mako-1.0.6-13.el8.noarch 2022-09-13T07:22:14.765 INFO:teuthology.orchestra.run.smithi036.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-09-13T07:22:14.765 INFO:teuthology.orchestra.run.smithi036.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-09-13T07:22:14.766 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-09-13T07:22:14.766 INFO:teuthology.orchestra.run.smithi036.stdout: python3-portend-2.6-1.el8.noarch 2022-09-13T07:22:14.766 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-09-13T07:22:14.766 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pycparser-2.14-14.el8.noarch 2022-09-13T07:22:14.766 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pytz-2017.2-9.el8.noarch 2022-09-13T07:22:14.767 INFO:teuthology.orchestra.run.smithi036.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-09-13T07:22:14.767 INFO:teuthology.orchestra.run.smithi036.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-09-13T07:22:14.767 INFO:teuthology.orchestra.run.smithi036.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-09-13T07:22:14.767 INFO:teuthology.orchestra.run.smithi036.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-09-13T07:22:14.768 INFO:teuthology.orchestra.run.smithi036.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-09-13T07:22:14.768 INFO:teuthology.orchestra.run.smithi036.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-09-13T07:22:14.768 INFO:teuthology.orchestra.run.smithi036.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-09-13T07:22:14.769 INFO:teuthology.orchestra.run.smithi036.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-09-13T07:22:14.769 INFO:teuthology.orchestra.run.smithi036.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-09-13T07:22:14.769 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:14.769 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:22:14.864 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-base 2022-09-13T07:22:15.038 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install ceph-base 2022-09-13T07:22:15.433 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:18 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:22:15.635 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:01:17 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:22:16.369 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-base-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:22:16.453 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:22:16.455 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2022-09-13T07:22:16.455 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:22:16.529 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install cephadm 2022-09-13T07:22:16.628 INFO:teuthology.orchestra.run.smithi036.stdout:Package ceph-base-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:22:16.712 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:22:16.713 INFO:teuthology.orchestra.run.smithi036.stdout:Nothing to do. 2022-09-13T07:22:16.713 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:22:16.787 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install cephadm 2022-09-13T07:22:17.093 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:20 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:22:17.384 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:01:19 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:22:18.100 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:22:18.102 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:22:18.103 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2022-09-13T07:22:18.103 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:22:18.104 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:22:18.104 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 81 k 2022-09-13T07:22:18.105 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:18.105 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:22:18.106 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:22:18.106 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2022-09-13T07:22:18.107 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:18.107 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 81 k 2022-09-13T07:22:18.107 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 367 k 2022-09-13T07:22:18.108 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:22:18.278 INFO:teuthology.orchestra.run.smithi083.stdout:cephadm-17.0.0-14918.g840d0541.el8.noarch.rpm 461 kB/s | 81 kB 00:00 2022-09-13T07:22:18.279 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:18.279 INFO:teuthology.orchestra.run.smithi083.stdout:Total 458 kB/s | 81 kB 00:00 2022-09-13T07:22:18.279 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:22:18.287 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:22:18.287 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:22:18.288 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:22:18.291 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:22:18.331 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:22:18.465 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:22:18.465 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:22:18.466 INFO:teuthology.orchestra.run.smithi036.stdout: Package Arch Version Repository Size 2022-09-13T07:22:18.467 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:22:18.468 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:22:18.468 INFO:teuthology.orchestra.run.smithi036.stdout: cephadm noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 81 k 2022-09-13T07:22:18.469 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:18.469 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:22:18.469 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:22:18.470 INFO:teuthology.orchestra.run.smithi036.stdout:Install 1 Package 2022-09-13T07:22:18.470 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:18.471 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 81 k 2022-09-13T07:22:18.471 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 367 k 2022-09-13T07:22:18.471 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:22:18.647 INFO:teuthology.orchestra.run.smithi036.stdout:cephadm-17.0.0-14918.g840d0541.el8.noarch.rpm 445 kB/s | 81 kB 00:00 2022-09-13T07:22:18.649 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:18.649 INFO:teuthology.orchestra.run.smithi036.stdout:Total 442 kB/s | 81 kB 00:00 2022-09-13T07:22:18.649 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:22:18.657 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:22:18.657 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:22:18.659 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:22:18.662 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:22:18.702 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:22:19.806 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:17.0.0-14918.g840d0541.el8.noarch 1/1 2022-09-13T07:22:19.976 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephadm-2:17.0.0-14918.g840d0541.el8.noarch 1/1 2022-09-13T07:22:20.160 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:17.0.0-14918.g840d0541.el8.noarch 1/1 2022-09-13T07:22:20.548 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephadm-2:17.0.0-14918.g840d0541.el8.noarch 1/1 2022-09-13T07:22:20.549 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:20.549 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:22:20.549 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:20.550 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:20.550 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:22:20.566 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: cephadm-2:17.0.0-14918.g840d0541.el8.noarch 1/1 2022-09-13T07:22:20.707 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-immutable-object-cache 2022-09-13T07:22:20.713 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : cephadm-2:17.0.0-14918.g840d0541.el8.noarch 1/1 2022-09-13T07:22:20.916 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: cephadm-2:17.0.0-14918.g840d0541.el8.noarch 1/1 2022-09-13T07:22:21.283 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:24 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:22:21.359 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : cephadm-2:17.0.0-14918.g840d0541.el8.noarch 1/1 2022-09-13T07:22:21.360 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:21.360 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:22:21.361 INFO:teuthology.orchestra.run.smithi036.stdout: cephadm-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:21.361 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:21.362 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:22:21.567 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install ceph-immutable-object-cache 2022-09-13T07:22:22.167 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:01:24 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:22:22.302 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:22:22.303 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:22:22.303 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2022-09-13T07:22:22.303 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:22:22.304 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:22:22.304 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-14918.g840d0541.el8 ceph 165 k 2022-09-13T07:22:22.304 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:22.304 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:22:22.305 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:22:22.305 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2022-09-13T07:22:22.305 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:22.305 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 165 k 2022-09-13T07:22:22.305 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 503 k 2022-09-13T07:22:22.306 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:22:22.507 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-immutable-object-cache-17.0.0-14918.g840d0 810 kB/s | 165 kB 00:00 2022-09-13T07:22:22.508 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:22.509 INFO:teuthology.orchestra.run.smithi083.stdout:Total 806 kB/s | 165 kB 00:00 2022-09-13T07:22:22.509 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:22:22.519 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:22:22.520 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:22:22.565 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:22:22.567 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:22:22.677 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:22:22.839 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-14918.g840d0541 1/1 2022-09-13T07:22:23.217 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-14918.g840d0541 1/1 2022-09-13T07:22:23.254 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:22:23.255 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:22:23.256 INFO:teuthology.orchestra.run.smithi036.stdout: Package Arch Version Repo Size 2022-09-13T07:22:23.256 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:22:23.256 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:22:23.256 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-14918.g840d0541.el8 ceph 165 k 2022-09-13T07:22:23.257 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:23.257 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:22:23.257 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:22:23.257 INFO:teuthology.orchestra.run.smithi036.stdout:Install 1 Package 2022-09-13T07:22:23.257 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:23.258 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 165 k 2022-09-13T07:22:23.258 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 503 k 2022-09-13T07:22:23.258 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:22:23.444 INFO:teuthology.orchestra.run.smithi036.stdout:ceph-immutable-object-cache-17.0.0-14918.g840d0 880 kB/s | 165 kB 00:00 2022-09-13T07:22:23.445 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:23.446 INFO:teuthology.orchestra.run.smithi036.stdout:Total 874 kB/s | 165 kB 00:00 2022-09-13T07:22:23.446 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:22:23.457 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:22:23.457 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:22:23.507 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:22:23.510 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:22:23.654 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:22:23.812 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-14918.g840d0541 1/1 2022-09-13T07:22:24.217 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-14918.g840d0541 1/1 2022-09-13T07:22:24.486 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-14918.g840d0541 1/1 2022-09-13T07:22:24.486 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:24.486 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:22:24.487 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:22:24.487 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:24.487 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:22:24.633 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-14918.g840d0541 1/1 2022-09-13T07:22:24.633 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:24.634 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:22:24.634 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-immutable-object-cache-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:22:24.634 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:24.634 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:22:24.681 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr 2022-09-13T07:22:24.851 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install ceph-mgr 2022-09-13T07:22:25.251 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:28 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:22:25.473 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:01:27 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:22:26.189 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-mgr-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:22:26.270 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:22:26.271 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2022-09-13T07:22:26.271 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:22:26.353 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-dashboard 2022-09-13T07:22:26.471 INFO:teuthology.orchestra.run.smithi036.stdout:Package ceph-mgr-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:22:26.559 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:22:26.561 INFO:teuthology.orchestra.run.smithi036.stdout:Nothing to do. 2022-09-13T07:22:26.561 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:22:26.916 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:29 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:22:26.945 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install ceph-mgr-dashboard 2022-09-13T07:22:27.542 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:01:29 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:22:27.923 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:22:27.924 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:22:27.925 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2022-09-13T07:22:27.925 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:22:27.925 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:22:27.926 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 1.6 M 2022-09-13T07:22:27.926 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2022-09-13T07:22:27.927 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards 2022-09-13T07:22:27.927 INFO:teuthology.orchestra.run.smithi083.stdout: noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 23 k 2022-09-13T07:22:27.928 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 14 k 2022-09-13T07:22:27.928 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2022-09-13T07:22:27.928 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-09-13T07:22:27.930 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-09-13T07:22:27.931 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:27.931 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:22:27.931 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:22:27.932 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2022-09-13T07:22:27.932 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:27.934 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 1.9 M 2022-09-13T07:22:27.934 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 7.8 M 2022-09-13T07:22:27.935 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:22:28.070 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): ceph-grafana-dashboards-17.0.0-14918.g84 160 kB/s | 23 kB 00:00 2022-09-13T07:22:28.087 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): ceph-prometheus-alerts-17.0.0-14918.g840 90 kB/s | 14 kB 00:00 2022-09-13T07:22:28.129 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 725 kB/s | 43 kB 00:00 2022-09-13T07:22:28.246 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): ceph-mgr-dashboard-17.0.0-14918.g840d054 5.2 MB/s | 1.6 MB 00:00 2022-09-13T07:22:28.429 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 98 kB/s | 34 kB 00:00 2022-09-13T07:22:28.537 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 481 kB/s | 196 kB 00:00 2022-09-13T07:22:28.540 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:28.540 INFO:teuthology.orchestra.run.smithi083.stdout:Total 3.2 MB/s | 1.9 MB 00:00 2022-09-13T07:22:28.550 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:22:28.559 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:22:28.560 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:22:28.633 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:22:28.634 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:22:28.634 INFO:teuthology.orchestra.run.smithi036.stdout: Package Arch Version Repository Size 2022-09-13T07:22:28.635 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:22:28.635 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:22:28.635 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-dashboard noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 1.6 M 2022-09-13T07:22:28.635 INFO:teuthology.orchestra.run.smithi036.stdout:Installing dependencies: 2022-09-13T07:22:28.636 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-grafana-dashboards 2022-09-13T07:22:28.636 INFO:teuthology.orchestra.run.smithi036.stdout: noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 23 k 2022-09-13T07:22:28.636 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-prometheus-alerts noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 14 k 2022-09-13T07:22:28.636 INFO:teuthology.orchestra.run.smithi036.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2022-09-13T07:22:28.637 INFO:teuthology.orchestra.run.smithi036.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-09-13T07:22:28.637 INFO:teuthology.orchestra.run.smithi036.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-09-13T07:22:28.637 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:28.637 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:22:28.638 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:22:28.638 INFO:teuthology.orchestra.run.smithi036.stdout:Install 6 Packages 2022-09-13T07:22:28.638 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:28.639 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 1.9 M 2022-09-13T07:22:28.640 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 7.8 M 2022-09-13T07:22:28.640 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:22:28.640 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:22:28.643 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:22:28.789 INFO:teuthology.orchestra.run.smithi036.stdout:(1/6): ceph-grafana-dashboards-17.0.0-14918.g84 150 kB/s | 23 kB 00:00 2022-09-13T07:22:28.815 INFO:teuthology.orchestra.run.smithi036.stdout:(2/6): ceph-prometheus-alerts-17.0.0-14918.g840 81 kB/s | 14 kB 00:00 2022-09-13T07:22:28.865 INFO:teuthology.orchestra.run.smithi036.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 566 kB/s | 43 kB 00:00 2022-09-13T07:22:28.960 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:22:28.963 INFO:teuthology.orchestra.run.smithi036.stdout:(4/6): ceph-mgr-dashboard-17.0.0-14918.g840d054 5.1 MB/s | 1.6 MB 00:00 2022-09-13T07:22:28.990 INFO:teuthology.orchestra.run.smithi036.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 191 kB/s | 34 kB 00:00 2022-09-13T07:22:29.057 INFO:teuthology.orchestra.run.smithi036.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2022-09-13T07:22:29.059 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:29.060 INFO:teuthology.orchestra.run.smithi036.stdout:Total 4.6 MB/s | 1.9 MB 00:00 2022-09-13T07:22:29.070 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:22:29.079 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:22:29.080 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:22:29.160 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:22:29.162 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:22:29.169 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-09-13T07:22:29.347 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-09-13T07:22:29.503 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:22:29.524 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-09-13T07:22:29.719 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-09-13T07:22:29.938 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-09-13T07:22:30.164 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-09-13T07:22:30.318 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-14918.g840d0541.el8. 4/6 2022-09-13T07:22:30.668 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-14918.g840d0541.el8 5/6 2022-09-13T07:22:30.853 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-14918.g840d0541.el8. 4/6 2022-09-13T07:22:30.882 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-14918.g840d0541.el8.noar 6/6 2022-09-13T07:22:31.162 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-14918.g840d0541.el8.noar 6/6 2022-09-13T07:22:31.163 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-14918.g840d0541.el8 1/6 2022-09-13T07:22:31.163 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-14918.g840d0541.el8.noar 2/6 2022-09-13T07:22:31.163 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-14918.g840d0541.el8. 3/6 2022-09-13T07:22:31.164 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-09-13T07:22:31.164 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-09-13T07:22:31.175 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-14918.g840d0541.el8 5/6 2022-09-13T07:22:31.420 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-14918.g840d0541.el8.noar 6/6 2022-09-13T07:22:31.579 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-09-13T07:22:31.580 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:31.580 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:22:31.580 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-grafana-dashboards-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:31.580 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-dashboard-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:31.581 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-prometheus-alerts-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:31.581 INFO:teuthology.orchestra.run.smithi036.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-09-13T07:22:31.581 INFO:teuthology.orchestra.run.smithi036.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-09-13T07:22:31.581 INFO:teuthology.orchestra.run.smithi036.stdout: python3-routes-2.4.1-12.el8.noarch 2022-09-13T07:22:31.582 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:31.582 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:22:31.648 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-14918.g840d0541.el8.noar 6/6 2022-09-13T07:22:31.648 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-14918.g840d0541.el8 1/6 2022-09-13T07:22:31.649 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-14918.g840d0541.el8.noar 2/6 2022-09-13T07:22:31.649 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-14918.g840d0541.el8. 3/6 2022-09-13T07:22:31.649 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-09-13T07:22:31.650 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-09-13T07:22:31.674 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install ceph-mgr-diskprediction-local 2022-09-13T07:22:32.026 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-09-13T07:22:32.027 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:32.027 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:22:32.028 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:32.028 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:32.029 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:32.029 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-09-13T07:22:32.030 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-09-13T07:22:32.030 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes-2.4.1-12.el8.noarch 2022-09-13T07:22:32.031 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:32.031 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:22:32.133 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-diskprediction-local 2022-09-13T07:22:32.275 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:01:34 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:22:32.711 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:35 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:22:33.356 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:22:33.357 INFO:teuthology.orchestra.run.smithi036.stdout:===================================================================================================== 2022-09-13T07:22:33.358 INFO:teuthology.orchestra.run.smithi036.stdout: Package Arch Version Repository Size 2022-09-13T07:22:33.358 INFO:teuthology.orchestra.run.smithi036.stdout:===================================================================================================== 2022-09-13T07:22:33.358 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:22:33.358 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 7.4 M 2022-09-13T07:22:33.359 INFO:teuthology.orchestra.run.smithi036.stdout:Installing dependencies: 2022-09-13T07:22:33.359 INFO:teuthology.orchestra.run.smithi036.stdout: libgfortran x86_64 8.5.0-15.el8 CentOS-BaseOS 644 k 2022-09-13T07:22:33.359 INFO:teuthology.orchestra.run.smithi036.stdout: libquadmath x86_64 8.5.0-15.el8 CentOS-BaseOS 172 k 2022-09-13T07:22:33.359 INFO:teuthology.orchestra.run.smithi036.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2022-09-13T07:22:33.359 INFO:teuthology.orchestra.run.smithi036.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2022-09-13T07:22:33.360 INFO:teuthology.orchestra.run.smithi036.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-09-13T07:22:33.360 INFO:teuthology.orchestra.run.smithi036.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-09-13T07:22:33.360 INFO:teuthology.orchestra.run.smithi036.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-09-13T07:22:33.360 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:33.361 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:22:33.361 INFO:teuthology.orchestra.run.smithi036.stdout:===================================================================================================== 2022-09-13T07:22:33.361 INFO:teuthology.orchestra.run.smithi036.stdout:Install 8 Packages 2022-09-13T07:22:33.361 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:33.362 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 35 M 2022-09-13T07:22:33.362 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 210 M 2022-09-13T07:22:33.362 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:22:33.597 INFO:teuthology.orchestra.run.smithi036.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 21 MB/s | 4.9 MB 00:00 2022-09-13T07:22:33.705 INFO:teuthology.orchestra.run.smithi036.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 35 MB/s | 3.7 MB 00:00 2022-09-13T07:22:33.737 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:22:33.738 INFO:teuthology.orchestra.run.smithi083.stdout:===================================================================================================== 2022-09-13T07:22:33.738 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2022-09-13T07:22:33.738 INFO:teuthology.orchestra.run.smithi083.stdout:===================================================================================================== 2022-09-13T07:22:33.738 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:22:33.739 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 7.4 M 2022-09-13T07:22:33.739 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2022-09-13T07:22:33.739 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran x86_64 8.5.0-15.el8 CentOS-BaseOS 644 k 2022-09-13T07:22:33.739 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath x86_64 8.5.0-15.el8 CentOS-BaseOS 172 k 2022-09-13T07:22:33.740 INFO:teuthology.orchestra.run.smithi083.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2022-09-13T07:22:33.740 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2022-09-13T07:22:33.740 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-09-13T07:22:33.741 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-09-13T07:22:33.741 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-09-13T07:22:33.741 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:33.741 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:22:33.742 INFO:teuthology.orchestra.run.smithi083.stdout:===================================================================================================== 2022-09-13T07:22:33.742 INFO:teuthology.orchestra.run.smithi083.stdout:Install 8 Packages 2022-09-13T07:22:33.742 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:33.743 INFO:teuthology.orchestra.run.smithi036.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.7 MB/s | 225 kB 00:00 2022-09-13T07:22:33.743 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 35 M 2022-09-13T07:22:33.744 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 210 M 2022-09-13T07:22:33.744 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:22:33.831 INFO:teuthology.orchestra.run.smithi036.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 10 MB/s | 4.7 MB 00:00 2022-09-13T07:22:33.873 INFO:teuthology.orchestra.run.smithi036.stdout:(5/8): libgfortran-8.5.0-15.el8.x86_64.rpm 15 MB/s | 644 kB 00:00 2022-09-13T07:22:33.906 INFO:teuthology.orchestra.run.smithi036.stdout:(6/8): libquadmath-8.5.0-15.el8.x86_64.rpm 5.0 MB/s | 172 kB 00:00 2022-09-13T07:22:33.944 INFO:teuthology.orchestra.run.smithi083.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2022-09-13T07:22:34.103 INFO:teuthology.orchestra.run.smithi083.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 13 MB/s | 4.9 MB 00:00 2022-09-13T07:22:34.137 INFO:teuthology.orchestra.run.smithi083.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2022-09-13T07:22:34.203 INFO:teuthology.orchestra.run.smithi083.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 14 MB/s | 3.7 MB 00:00 2022-09-13T07:22:34.230 INFO:teuthology.orchestra.run.smithi036.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 28 MB/s | 14 MB 00:00 2022-09-13T07:22:34.246 INFO:teuthology.orchestra.run.smithi083.stdout:(5/8): libgfortran-8.5.0-15.el8.x86_64.rpm 15 MB/s | 644 kB 00:00 2022-09-13T07:22:34.271 INFO:teuthology.orchestra.run.smithi083.stdout:(6/8): libquadmath-8.5.0-15.el8.x86_64.rpm 6.8 MB/s | 172 kB 00:00 2022-09-13T07:22:34.359 INFO:teuthology.orchestra.run.smithi036.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-149 7.5 MB/s | 7.4 MB 00:00 2022-09-13T07:22:34.359 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:34.360 INFO:teuthology.orchestra.run.smithi036.stdout:Total 36 MB/s | 35 MB 00:00 2022-09-13T07:22:34.389 INFO:teuthology.orchestra.run.smithi083.stdout:(7/8): ceph-mgr-diskprediction-local-17.0.0-149 12 MB/s | 7.4 MB 00:00 2022-09-13T07:22:34.588 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:22:34.607 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:22:34.608 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:22:34.622 INFO:teuthology.orchestra.run.smithi083.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 28 MB/s | 14 MB 00:00 2022-09-13T07:22:34.624 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:34.624 INFO:teuthology.orchestra.run.smithi083.stdout:Total 40 MB/s | 35 MB 00:00 2022-09-13T07:22:34.859 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:22:34.873 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:22:34.874 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:22:34.991 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:22:34.995 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:22:35.248 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:22:35.250 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:22:35.597 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:22:35.710 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : libquadmath-8.5.0-15.el8.x86_64 1/8 2022-09-13T07:22:35.781 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:22:35.872 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: libquadmath-8.5.0-15.el8.x86_64 1/8 2022-09-13T07:22:35.903 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libquadmath-8.5.0-15.el8.x86_64 1/8 2022-09-13T07:22:36.035 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : libgfortran-8.5.0-15.el8.x86_64 2/8 2022-09-13T07:22:36.049 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libquadmath-8.5.0-15.el8.x86_64 1/8 2022-09-13T07:22:36.240 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libgfortran-8.5.0-15.el8.x86_64 2/8 2022-09-13T07:22:36.690 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: libgfortran-8.5.0-15.el8.x86_64 2/8 2022-09-13T07:22:36.846 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2022-09-13T07:22:36.896 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libgfortran-8.5.0-15.el8.x86_64 2/8 2022-09-13T07:22:37.058 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2022-09-13T07:22:37.517 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2022-09-13T07:22:37.668 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2022-09-13T07:22:37.673 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2022-09-13T07:22:37.820 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2022-09-13T07:22:38.402 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2022-09-13T07:22:38.501 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2022-09-13T07:22:38.735 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2022-09-13T07:22:38.781 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2022-09-13T07:22:38.986 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-09-13T07:22:39.053 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-09-13T07:22:40.816 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-09-13T07:22:40.870 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-09-13T07:22:42.049 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2022-09-13T07:22:42.130 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2022-09-13T07:22:42.302 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-14918.g840d05 8/8 2022-09-13T07:22:42.581 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-14918.g840d05 8/8 2022-09-13T07:22:42.582 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-14918.g840d05 1/8 2022-09-13T07:22:42.582 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2022-09-13T07:22:42.582 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2022-09-13T07:22:42.583 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2022-09-13T07:22:42.583 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2022-09-13T07:22:42.583 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2022-09-13T07:22:42.583 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : libgfortran-8.5.0-15.el8.x86_64 7/8 2022-09-13T07:22:42.702 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-14918.g840d05 8/8 2022-09-13T07:22:42.965 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-14918.g840d05 8/8 2022-09-13T07:22:42.966 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-14918.g840d05 1/8 2022-09-13T07:22:42.967 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2022-09-13T07:22:42.967 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2022-09-13T07:22:42.968 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2022-09-13T07:22:42.968 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2022-09-13T07:22:42.968 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2022-09-13T07:22:42.968 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libgfortran-8.5.0-15.el8.x86_64 7/8 2022-09-13T07:22:43.024 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : libquadmath-8.5.0-15.el8.x86_64 8/8 2022-09-13T07:22:43.025 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:43.025 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:22:43.026 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-diskprediction-local-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:43.026 INFO:teuthology.orchestra.run.smithi036.stdout: libgfortran-8.5.0-15.el8.x86_64 2022-09-13T07:22:43.027 INFO:teuthology.orchestra.run.smithi036.stdout: libquadmath-8.5.0-15.el8.x86_64 2022-09-13T07:22:43.027 INFO:teuthology.orchestra.run.smithi036.stdout: openblas-0.3.15-4.el8.x86_64 2022-09-13T07:22:43.027 INFO:teuthology.orchestra.run.smithi036.stdout: openblas-threads-0.3.15-4.el8.x86_64 2022-09-13T07:22:43.028 INFO:teuthology.orchestra.run.smithi036.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-09-13T07:22:43.029 INFO:teuthology.orchestra.run.smithi036.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-09-13T07:22:43.029 INFO:teuthology.orchestra.run.smithi036.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-09-13T07:22:43.030 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:43.030 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:22:43.375 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libquadmath-8.5.0-15.el8.x86_64 8/8 2022-09-13T07:22:43.376 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:43.376 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:22:43.376 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:43.377 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran-8.5.0-15.el8.x86_64 2022-09-13T07:22:43.377 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath-8.5.0-15.el8.x86_64 2022-09-13T07:22:43.377 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-0.3.15-4.el8.x86_64 2022-09-13T07:22:43.377 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-threads-0.3.15-4.el8.x86_64 2022-09-13T07:22:43.377 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-09-13T07:22:43.378 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-09-13T07:22:43.378 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-09-13T07:22:43.378 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:43.378 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:22:43.480 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-rook 2022-09-13T07:22:44.048 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:47 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:22:44.350 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install ceph-mgr-rook 2022-09-13T07:22:44.957 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:01:46 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:22:45.045 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:22:45.046 INFO:teuthology.orchestra.run.smithi083.stdout:========================================================================================= 2022-09-13T07:22:45.047 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2022-09-13T07:22:45.047 INFO:teuthology.orchestra.run.smithi083.stdout:========================================================================================= 2022-09-13T07:22:45.047 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:22:45.048 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 47 k 2022-09-13T07:22:45.048 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2022-09-13T07:22:45.048 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-09-13T07:22:45.048 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-09-13T07:22:45.049 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-09-13T07:22:45.049 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-09-13T07:22:45.050 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-09-13T07:22:45.050 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-09-13T07:22:45.050 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2022-09-13T07:22:45.051 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-09-13T07:22:45.051 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-09-13T07:22:45.051 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2022-09-13T07:22:45.052 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-09-13T07:22:45.052 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-09-13T07:22:45.052 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:45.052 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:22:45.053 INFO:teuthology.orchestra.run.smithi083.stdout:========================================================================================= 2022-09-13T07:22:45.053 INFO:teuthology.orchestra.run.smithi083.stdout:Install 13 Packages 2022-09-13T07:22:45.063 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:45.064 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 1.9 M 2022-09-13T07:22:45.064 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 25 M 2022-09-13T07:22:45.064 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:22:45.122 INFO:teuthology.orchestra.run.smithi083.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 370 kB/s | 27 kB 00:00 2022-09-13T07:22:45.149 INFO:teuthology.orchestra.run.smithi083.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 206 kB/s | 20 kB 00:00 2022-09-13T07:22:45.173 INFO:teuthology.orchestra.run.smithi083.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2022-09-13T07:22:45.197 INFO:teuthology.orchestra.run.smithi083.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2022-09-13T07:22:45.248 INFO:teuthology.orchestra.run.smithi083.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.0 MB/s | 155 kB 00:00 2022-09-13T07:22:45.274 INFO:teuthology.orchestra.run.smithi083.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 564 kB/s | 43 kB 00:00 2022-09-13T07:22:45.382 INFO:teuthology.orchestra.run.smithi083.stdout:(7/13): ceph-mgr-rook-17.0.0-14918.g840d0541.el 141 kB/s | 47 kB 00:00 2022-09-13T07:22:45.408 INFO:teuthology.orchestra.run.smithi083.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 122 kB/s | 16 kB 00:00 2022-09-13T07:22:45.433 INFO:teuthology.orchestra.run.smithi083.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 179 kB/s | 33 kB 00:00 2022-09-13T07:22:45.517 INFO:teuthology.orchestra.run.smithi083.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 607 kB/s | 81 kB 00:00 2022-09-13T07:22:45.542 INFO:teuthology.orchestra.run.smithi083.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 566 kB/s | 61 kB 00:00 2022-09-13T07:22:45.593 INFO:teuthology.orchestra.run.smithi083.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 816 kB/s | 61 kB 00:00 2022-09-13T07:22:45.767 INFO:teuthology.orchestra.run.smithi083.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.2 MB/s | 1.2 MB 00:00 2022-09-13T07:22:45.769 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:45.769 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.7 MB/s | 1.9 MB 00:00 2022-09-13T07:22:45.821 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:22:45.837 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:22:45.837 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:22:45.917 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:22:45.918 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:22:46.053 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:22:46.057 INFO:teuthology.orchestra.run.smithi036.stdout:========================================================================================= 2022-09-13T07:22:46.057 INFO:teuthology.orchestra.run.smithi036.stdout: Package Arch Version Repository Size 2022-09-13T07:22:46.057 INFO:teuthology.orchestra.run.smithi036.stdout:========================================================================================= 2022-09-13T07:22:46.058 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:22:46.058 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-rook noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 47 k 2022-09-13T07:22:46.058 INFO:teuthology.orchestra.run.smithi036.stdout:Installing dependencies: 2022-09-13T07:22:46.058 INFO:teuthology.orchestra.run.smithi036.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-09-13T07:22:46.059 INFO:teuthology.orchestra.run.smithi036.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-09-13T07:22:46.059 INFO:teuthology.orchestra.run.smithi036.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-09-13T07:22:46.059 INFO:teuthology.orchestra.run.smithi036.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-09-13T07:22:46.059 INFO:teuthology.orchestra.run.smithi036.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-09-13T07:22:46.060 INFO:teuthology.orchestra.run.smithi036.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-09-13T07:22:46.060 INFO:teuthology.orchestra.run.smithi036.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2022-09-13T07:22:46.060 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-09-13T07:22:46.060 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-09-13T07:22:46.061 INFO:teuthology.orchestra.run.smithi036.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2022-09-13T07:22:46.061 INFO:teuthology.orchestra.run.smithi036.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-09-13T07:22:46.061 INFO:teuthology.orchestra.run.smithi036.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-09-13T07:22:46.061 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:46.062 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:22:46.062 INFO:teuthology.orchestra.run.smithi036.stdout:========================================================================================= 2022-09-13T07:22:46.062 INFO:teuthology.orchestra.run.smithi036.stdout:Install 13 Packages 2022-09-13T07:22:46.062 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:46.063 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 1.9 M 2022-09-13T07:22:46.063 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 25 M 2022-09-13T07:22:46.063 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:22:46.123 INFO:teuthology.orchestra.run.smithi036.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 313 kB/s | 20 kB 00:00 2022-09-13T07:22:46.173 INFO:teuthology.orchestra.run.smithi036.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 233 kB/s | 27 kB 00:00 2022-09-13T07:22:46.198 INFO:teuthology.orchestra.run.smithi036.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.7 MB/s | 126 kB 00:00 2022-09-13T07:22:46.231 INFO:teuthology.orchestra.run.smithi036.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 3.2 MB/s | 110 kB 00:00 2022-09-13T07:22:46.259 INFO:teuthology.orchestra.run.smithi036.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.6 MB/s | 155 kB 00:00 2022-09-13T07:22:46.274 INFO:teuthology.orchestra.run.smithi036.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.0 MB/s | 43 kB 00:00 2022-09-13T07:22:46.291 INFO:teuthology.orchestra.run.smithi036.stdout:(7/13): ceph-mgr-rook-17.0.0-14918.g840d0541.el 202 kB/s | 47 kB 00:00 2022-09-13T07:22:46.395 INFO:teuthology.orchestra.run.smithi036.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 139 kB/s | 16 kB 00:00 2022-09-13T07:22:46.408 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:22:46.409 INFO:teuthology.orchestra.run.smithi036.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 219 kB/s | 33 kB 00:00 2022-09-13T07:22:46.450 INFO:teuthology.orchestra.run.smithi036.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 511 kB/s | 81 kB 00:00 2022-09-13T07:22:46.501 INFO:teuthology.orchestra.run.smithi036.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 667 kB/s | 61 kB 00:00 2022-09-13T07:22:46.526 INFO:teuthology.orchestra.run.smithi036.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 816 kB/s | 61 kB 00:00 2022-09-13T07:22:46.604 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-09-13T07:22:46.634 INFO:teuthology.orchestra.run.smithi036.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.8 MB/s | 1.2 MB 00:00 2022-09-13T07:22:46.636 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:46.636 INFO:teuthology.orchestra.run.smithi036.stdout:Total 3.3 MB/s | 1.9 MB 00:00 2022-09-13T07:22:46.691 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:22:46.707 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:22:46.708 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:22:46.800 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:22:46.802 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-09-13T07:22:46.803 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:22:47.003 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-09-13T07:22:47.206 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-09-13T07:22:47.309 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:22:47.374 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-09-13T07:22:47.514 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-09-13T07:22:47.581 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-09-13T07:22:47.729 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-09-13T07:22:47.813 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-09-13T07:22:47.955 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-09-13T07:22:48.081 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-09-13T07:22:48.157 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-09-13T07:22:48.316 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-09-13T07:22:48.560 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-09-13T07:22:48.829 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-09-13T07:22:49.058 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-09-13T07:22:49.083 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-09-13T07:22:49.303 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-09-13T07:22:49.458 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-09-13T07:22:49.611 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-09-13T07:22:49.768 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-rook-2:17.0.0-14918.g840d0541.el8.noarch 13/13 2022-09-13T07:22:50.051 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-14918.g840d0541.el8.noarch 13/13 2022-09-13T07:22:50.051 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-rook-2:17.0.0-14918.g840d0541.el8.noarch 1/13 2022-09-13T07:22:50.051 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-09-13T07:22:50.052 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-09-13T07:22:50.052 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-09-13T07:22:50.052 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-09-13T07:22:50.052 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-09-13T07:22:50.052 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-09-13T07:22:50.053 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-09-13T07:22:50.053 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-09-13T07:22:50.053 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-09-13T07:22:50.053 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-09-13T07:22:50.054 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-09-13T07:22:50.140 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-09-13T07:22:50.366 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-09-13T07:22:50.501 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-09-13T07:22:50.501 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:50.501 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:22:50.502 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:50.502 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-09-13T07:22:50.502 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-09-13T07:22:50.503 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-09-13T07:22:50.503 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-09-13T07:22:50.503 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-09-13T07:22:50.503 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-09-13T07:22:50.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-09-13T07:22:50.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-09-13T07:22:50.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-09-13T07:22:50.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-09-13T07:22:50.505 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa-4.8-1.el8.noarch 2022-09-13T07:22:50.505 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-09-13T07:22:50.505 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:50.506 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:22:50.524 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-09-13T07:22:50.615 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-cephadm 2022-09-13T07:22:50.660 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-09-13T07:22:50.825 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-mgr-rook-2:17.0.0-14918.g840d0541.el8.noarch 13/13 2022-09-13T07:22:51.078 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-14918.g840d0541.el8.noarch 13/13 2022-09-13T07:22:51.079 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-mgr-rook-2:17.0.0-14918.g840d0541.el8.noarch 1/13 2022-09-13T07:22:51.079 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-09-13T07:22:51.080 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-09-13T07:22:51.080 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-09-13T07:22:51.081 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-09-13T07:22:51.081 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-09-13T07:22:51.081 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-09-13T07:22:51.082 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-09-13T07:22:51.082 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-09-13T07:22:51.083 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-09-13T07:22:51.083 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-09-13T07:22:51.083 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-09-13T07:22:51.179 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:54 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:22:51.564 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-09-13T07:22:51.565 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:51.566 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:22:51.566 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-rook-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:51.566 INFO:teuthology.orchestra.run.smithi036.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-09-13T07:22:51.567 INFO:teuthology.orchestra.run.smithi036.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-09-13T07:22:51.567 INFO:teuthology.orchestra.run.smithi036.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-09-13T07:22:51.567 INFO:teuthology.orchestra.run.smithi036.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-09-13T07:22:51.567 INFO:teuthology.orchestra.run.smithi036.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-09-13T07:22:51.567 INFO:teuthology.orchestra.run.smithi036.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-09-13T07:22:51.568 INFO:teuthology.orchestra.run.smithi036.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-09-13T07:22:51.569 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-09-13T07:22:51.569 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-09-13T07:22:51.569 INFO:teuthology.orchestra.run.smithi036.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-09-13T07:22:51.569 INFO:teuthology.orchestra.run.smithi036.stdout: python3-rsa-4.8-1.el8.noarch 2022-09-13T07:22:51.570 INFO:teuthology.orchestra.run.smithi036.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-09-13T07:22:51.570 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:51.571 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:22:51.674 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install ceph-mgr-cephadm 2022-09-13T07:22:52.185 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:22:52.187 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:22:52.187 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2022-09-13T07:22:52.188 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:22:52.188 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:22:52.189 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 115 k 2022-09-13T07:22:52.189 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2022-09-13T07:22:52.189 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-09-13T07:22:52.190 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-09-13T07:22:52.190 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-09-13T07:22:52.191 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-09-13T07:22:52.191 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:52.192 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:22:52.192 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:22:52.193 INFO:teuthology.orchestra.run.smithi083.stdout:Install 5 Packages 2022-09-13T07:22:52.193 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:52.194 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 5.9 M 2022-09-13T07:22:52.194 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 26 M 2022-09-13T07:22:52.194 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:22:52.274 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:01:54 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:22:52.282 INFO:teuthology.orchestra.run.smithi083.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 5.6 MB/s | 538 kB 00:00 2022-09-13T07:22:52.366 INFO:teuthology.orchestra.run.smithi083.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 27 MB/s | 4.8 MB 00:00 2022-09-13T07:22:52.409 INFO:teuthology.orchestra.run.smithi083.stdout:(3/5): ceph-mgr-cephadm-17.0.0-14918.g840d0541. 524 kB/s | 115 kB 00:00 2022-09-13T07:22:52.658 INFO:teuthology.orchestra.run.smithi083.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.2 MB/s | 476 kB 00:00 2022-09-13T07:22:52.675 INFO:teuthology.orchestra.run.smithi083.stdout:(5/5): python3-natsort-7.1.1-2.el8.noarch.rpm 195 kB/s | 60 kB 00:00 2022-09-13T07:22:52.676 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:52.676 INFO:teuthology.orchestra.run.smithi083.stdout:Total 12 MB/s | 5.9 MB 00:00 2022-09-13T07:22:52.727 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:22:52.738 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:22:52.738 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:22:52.823 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:22:52.826 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:22:53.110 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:22:53.360 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:22:53.361 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:22:53.362 INFO:teuthology.orchestra.run.smithi036.stdout: Package Arch Version Repository Size 2022-09-13T07:22:53.362 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:22:53.362 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:22:53.362 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-cephadm noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 115 k 2022-09-13T07:22:53.363 INFO:teuthology.orchestra.run.smithi036.stdout:Installing dependencies: 2022-09-13T07:22:53.363 INFO:teuthology.orchestra.run.smithi036.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-09-13T07:22:53.363 INFO:teuthology.orchestra.run.smithi036.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-09-13T07:22:53.364 INFO:teuthology.orchestra.run.smithi036.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-09-13T07:22:53.364 INFO:teuthology.orchestra.run.smithi036.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-09-13T07:22:53.364 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:53.364 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:22:53.365 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:22:53.365 INFO:teuthology.orchestra.run.smithi036.stdout:Install 5 Packages 2022-09-13T07:22:53.365 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:53.366 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 5.9 M 2022-09-13T07:22:53.366 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 26 M 2022-09-13T07:22:53.366 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:22:53.449 INFO:teuthology.orchestra.run.smithi036.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 6.2 MB/s | 538 kB 00:00 2022-09-13T07:22:53.576 INFO:teuthology.orchestra.run.smithi036.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 23 MB/s | 4.8 MB 00:00 2022-09-13T07:22:53.624 INFO:teuthology.orchestra.run.smithi036.stdout:(3/5): ceph-mgr-cephadm-17.0.0-14918.g840d0541. 443 kB/s | 115 kB 00:00 2022-09-13T07:22:53.729 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2022-09-13T07:22:53.750 INFO:teuthology.orchestra.run.smithi036.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 344 kB/s | 60 kB 00:00 2022-09-13T07:22:53.841 INFO:teuthology.orchestra.run.smithi036.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.2 MB/s | 476 kB 00:00 2022-09-13T07:22:53.843 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:53.843 INFO:teuthology.orchestra.run.smithi036.stdout:Total 12 MB/s | 5.9 MB 00:00 2022-09-13T07:22:53.895 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:22:53.908 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:22:53.908 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:22:54.005 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:22:54.008 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:22:54.298 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:22:54.604 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2022-09-13T07:22:54.641 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2022-09-13T07:22:54.964 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2022-09-13T07:22:55.179 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2022-09-13T07:22:55.409 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-14918.g840d0541.el8.noarch 5/5 2022-09-13T07:22:55.655 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-14918.g840d0541.el8.noarch 5/5 2022-09-13T07:22:55.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-14918.g840d0541.el8.noarch 1/5 2022-09-13T07:22:55.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2022-09-13T07:22:55.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2022-09-13T07:22:55.657 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2022-09-13T07:22:55.849 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2022-09-13T07:22:56.040 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2022-09-13T07:22:56.041 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:56.041 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:22:56.041 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:56.041 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-09-13T07:22:56.042 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel-2.5.1-7.el8.noarch 2022-09-13T07:22:56.042 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-09-13T07:22:56.042 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-09-13T07:22:56.043 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:56.043 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:22:56.139 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-fuse 2022-09-13T07:22:56.211 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2022-09-13T07:22:56.421 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2022-09-13T07:22:56.624 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-14918.g840d0541.el8.noarch 5/5 2022-09-13T07:22:56.702 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:59 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:22:56.867 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-14918.g840d0541.el8.noarch 5/5 2022-09-13T07:22:56.868 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-14918.g840d0541.el8.noarch 1/5 2022-09-13T07:22:56.868 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2022-09-13T07:22:56.868 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2022-09-13T07:22:56.869 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2022-09-13T07:22:57.269 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2022-09-13T07:22:57.269 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:57.270 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:22:57.270 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-cephadm-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:22:57.270 INFO:teuthology.orchestra.run.smithi036.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-09-13T07:22:57.271 INFO:teuthology.orchestra.run.smithi036.stdout: python3-babel-2.5.1-7.el8.noarch 2022-09-13T07:22:57.271 INFO:teuthology.orchestra.run.smithi036.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-09-13T07:22:57.271 INFO:teuthology.orchestra.run.smithi036.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-09-13T07:22:57.272 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:57.272 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:22:57.376 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install ceph-fuse 2022-09-13T07:22:57.707 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:22:57.709 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:22:57.709 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2022-09-13T07:22:57.710 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:22:57.710 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:22:57.710 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse x86_64 2:17.0.0-14918.g840d0541.el8 ceph 844 k 2022-09-13T07:22:57.711 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2022-09-13T07:22:57.711 INFO:teuthology.orchestra.run.smithi083.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2022-09-13T07:22:57.711 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:57.712 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:22:57.712 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:22:57.713 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2022-09-13T07:22:57.713 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:57.713 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 926 k 2022-09-13T07:22:57.714 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 2.9 M 2022-09-13T07:22:57.714 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:22:57.780 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 1.3 MB/s | 83 kB 00:00 2022-09-13T07:22:57.979 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:01:59 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:22:58.241 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): ceph-fuse-17.0.0-14918.g840d0541.el8.x86 1.6 MB/s | 844 kB 00:00 2022-09-13T07:22:58.242 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:58.243 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.7 MB/s | 926 kB 00:00 2022-09-13T07:22:58.252 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:22:58.273 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:22:58.274 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:22:58.345 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:22:58.348 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:22:58.511 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:22:58.748 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2022-09-13T07:22:58.882 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 2/2 2022-09-13T07:22:59.065 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:22:59.066 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:22:59.067 INFO:teuthology.orchestra.run.smithi036.stdout: Package Arch Version Repository Size 2022-09-13T07:22:59.067 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:22:59.068 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:22:59.068 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-fuse x86_64 2:17.0.0-14918.g840d0541.el8 ceph 844 k 2022-09-13T07:22:59.069 INFO:teuthology.orchestra.run.smithi036.stdout:Installing dependencies: 2022-09-13T07:22:59.069 INFO:teuthology.orchestra.run.smithi036.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2022-09-13T07:22:59.070 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:59.070 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:22:59.071 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:22:59.071 INFO:teuthology.orchestra.run.smithi036.stdout:Install 2 Packages 2022-09-13T07:22:59.071 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:22:59.071 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 926 k 2022-09-13T07:22:59.072 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 2.9 M 2022-09-13T07:22:59.072 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:22:59.134 INFO:teuthology.orchestra.run.smithi036.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 1.3 MB/s | 83 kB 00:00 2022-09-13T07:22:59.233 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 2/2 2022-09-13T07:22:59.234 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 1/2 2022-09-13T07:22:59.340 INFO:teuthology.orchestra.run.smithi036.stdout:(2/2): ceph-fuse-17.0.0-14918.g840d0541.el8.x86 3.0 MB/s | 844 kB 00:00 2022-09-13T07:22:59.341 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:22:59.342 INFO:teuthology.orchestra.run.smithi036.stdout:Total 3.3 MB/s | 926 kB 00:00 2022-09-13T07:22:59.352 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:22:59.375 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:22:59.375 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:22:59.454 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:22:59.457 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:22:59.618 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2022-09-13T07:22:59.619 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:59.619 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:22:59.619 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2022-09-13T07:22:59.620 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:22:59.620 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:22:59.666 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:22:59.814 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-volume 2022-09-13T07:22:59.945 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2022-09-13T07:23:00.248 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 2/2 2022-09-13T07:23:00.395 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:03 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:23:00.630 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 2/2 2022-09-13T07:23:00.630 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 1/2 2022-09-13T07:23:01.044 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2022-09-13T07:23:01.045 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:01.045 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:23:01.045 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2022-09-13T07:23:01.045 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:01.046 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:23:01.208 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install ceph-volume 2022-09-13T07:23:01.408 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:23:01.409 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:01.410 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2022-09-13T07:23:01.410 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:01.410 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:23:01.410 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 263 k 2022-09-13T07:23:01.410 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:01.411 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:23:01.411 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:01.411 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2022-09-13T07:23:01.411 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:01.412 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 263 k 2022-09-13T07:23:01.412 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.3 M 2022-09-13T07:23:01.412 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:23:01.627 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-volume-17.0.0-14918.g840d0541.el8.noarch.r 1.2 MB/s | 263 kB 00:00 2022-09-13T07:23:01.629 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:23:01.630 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.2 MB/s | 263 kB 00:00 2022-09-13T07:23:01.639 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:23:01.640 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:23:01.641 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:23:01.668 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:23:01.668 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:23:01.810 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:02:03 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:23:01.824 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:23:02.058 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-volume-2:17.0.0-14918.g840d0541.el8.noarch 1/1 2022-09-13T07:23:02.466 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-volume-2:17.0.0-14918.g840d0541.el8.noarch 1/1 2022-09-13T07:23:02.840 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-volume-2:17.0.0-14918.g840d0541.el8.noarch 1/1 2022-09-13T07:23:02.840 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:02.840 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:23:02.841 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:23:02.841 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:02.841 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:23:02.895 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:23:02.896 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:02.897 INFO:teuthology.orchestra.run.smithi036.stdout: Package Arch Version Repository Size 2022-09-13T07:23:02.897 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:02.897 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:23:02.898 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-volume noarch 2:17.0.0-14918.g840d0541.el8 ceph-noarch 263 k 2022-09-13T07:23:02.898 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:02.898 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:23:02.899 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:02.899 INFO:teuthology.orchestra.run.smithi036.stdout:Install 1 Package 2022-09-13T07:23:02.900 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:02.900 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 263 k 2022-09-13T07:23:02.900 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 1.3 M 2022-09-13T07:23:02.901 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:23:03.043 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados-devel 2022-09-13T07:23:03.128 INFO:teuthology.orchestra.run.smithi036.stdout:ceph-volume-17.0.0-14918.g840d0541.el8.noarch.r 1.1 MB/s | 263 kB 00:00 2022-09-13T07:23:03.129 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:23:03.129 INFO:teuthology.orchestra.run.smithi036.stdout:Total 1.1 MB/s | 263 kB 00:00 2022-09-13T07:23:03.130 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:23:03.140 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:23:03.141 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:23:03.157 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:23:03.160 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:23:03.332 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:23:03.540 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : ceph-volume-2:17.0.0-14918.g840d0541.el8.noarch 1/1 2022-09-13T07:23:03.628 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:06 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:23:03.956 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: ceph-volume-2:17.0.0-14918.g840d0541.el8.noarch 1/1 2022-09-13T07:23:04.383 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : ceph-volume-2:17.0.0-14918.g840d0541.el8.noarch 1/1 2022-09-13T07:23:04.384 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:04.384 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:23:04.385 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-volume-2:17.0.0-14918.g840d0541.el8.noarch 2022-09-13T07:23:04.385 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:04.385 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:23:04.608 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install librados-devel 2022-09-13T07:23:04.640 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:23:04.641 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:04.642 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2022-09-13T07:23:04.642 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:04.643 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:23:04.643 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel x86_64 2:17.0.0-14918.g840d0541.el8 ceph 139 k 2022-09-13T07:23:04.643 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:04.644 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:23:04.644 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:04.644 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2022-09-13T07:23:04.644 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:04.645 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 139 k 2022-09-13T07:23:04.645 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 512 k 2022-09-13T07:23:04.645 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:23:04.832 INFO:teuthology.orchestra.run.smithi083.stdout:librados-devel-17.0.0-14918.g840d0541.el8.x86_6 731 kB/s | 139 kB 00:00 2022-09-13T07:23:04.833 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:23:04.833 INFO:teuthology.orchestra.run.smithi083.stdout:Total 726 kB/s | 139 kB 00:00 2022-09-13T07:23:04.833 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:23:04.838 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:23:04.839 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:23:04.887 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:23:04.890 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:23:05.012 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:23:05.164 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados-devel-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:05.214 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:02:07 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:23:05.430 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados-devel-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:05.798 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados-devel-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:05.798 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:05.799 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:23:05.799 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:23:05.799 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:05.799 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:23:05.986 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs2 2022-09-13T07:23:06.306 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:23:06.308 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:06.308 INFO:teuthology.orchestra.run.smithi036.stdout: Package Arch Version Repo Size 2022-09-13T07:23:06.308 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:06.309 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:23:06.309 INFO:teuthology.orchestra.run.smithi036.stdout: librados-devel x86_64 2:17.0.0-14918.g840d0541.el8 ceph 139 k 2022-09-13T07:23:06.310 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:06.310 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:23:06.310 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:06.311 INFO:teuthology.orchestra.run.smithi036.stdout:Install 1 Package 2022-09-13T07:23:06.311 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:06.312 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 139 k 2022-09-13T07:23:06.312 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 512 k 2022-09-13T07:23:06.312 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:23:06.515 INFO:teuthology.orchestra.run.smithi036.stdout:librados-devel-17.0.0-14918.g840d0541.el8.x86_6 673 kB/s | 139 kB 00:00 2022-09-13T07:23:06.516 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:23:06.516 INFO:teuthology.orchestra.run.smithi036.stdout:Total 669 kB/s | 139 kB 00:00 2022-09-13T07:23:06.516 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:23:06.521 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:23:06.522 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:23:06.563 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:09 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:23:06.575 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:23:06.578 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:23:06.703 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:23:06.880 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : librados-devel-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:07.146 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: librados-devel-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:07.498 INFO:teuthology.orchestra.run.smithi083.stdout:Package libcephfs2-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:23:07.558 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : librados-devel-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:07.558 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:07.559 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:23:07.559 INFO:teuthology.orchestra.run.smithi036.stdout: librados-devel-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:23:07.559 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:07.559 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:23:07.573 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:23:07.574 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2022-09-13T07:23:07.574 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:23:07.669 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs-devel 2022-09-13T07:23:07.760 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install libcephfs2 2022-09-13T07:23:08.240 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:11 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:23:08.366 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:02:10 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:23:09.252 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:23:09.253 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:09.253 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2022-09-13T07:23:09.253 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:09.254 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:23:09.254 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel x86_64 2:17.0.0-14918.g840d0541.el8 ceph 30 k 2022-09-13T07:23:09.254 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:09.254 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:23:09.254 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:09.255 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2022-09-13T07:23:09.255 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:09.255 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 30 k 2022-09-13T07:23:09.255 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 137 k 2022-09-13T07:23:09.256 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:23:09.363 INFO:teuthology.orchestra.run.smithi036.stdout:Package libcephfs2-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:23:09.446 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:23:09.447 INFO:teuthology.orchestra.run.smithi036.stdout:Nothing to do. 2022-09-13T07:23:09.447 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:23:09.531 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install libcephfs-devel 2022-09-13T07:23:09.789 INFO:teuthology.orchestra.run.smithi083.stdout:libcephfs-devel-17.0.0-14918.g840d0541.el8.x86_ 57 kB/s | 30 kB 00:00 2022-09-13T07:23:09.790 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:23:09.790 INFO:teuthology.orchestra.run.smithi083.stdout:Total 57 kB/s | 30 kB 00:00 2022-09-13T07:23:09.791 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:23:09.794 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:23:09.795 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:23:09.915 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:23:09.918 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:23:10.129 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:02:12 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:23:10.241 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:23:10.589 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs-devel-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:10.851 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:11.195 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs-devel-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:11.195 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:11.195 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:23:11.196 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:23:11.196 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:11.196 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:23:11.215 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:23:11.216 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:11.217 INFO:teuthology.orchestra.run.smithi036.stdout: Package Arch Version Repo Size 2022-09-13T07:23:11.217 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:11.217 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:23:11.218 INFO:teuthology.orchestra.run.smithi036.stdout: libcephfs-devel x86_64 2:17.0.0-14918.g840d0541.el8 ceph 30 k 2022-09-13T07:23:11.218 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:11.218 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:23:11.218 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:11.219 INFO:teuthology.orchestra.run.smithi036.stdout:Install 1 Package 2022-09-13T07:23:11.219 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:11.219 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 30 k 2022-09-13T07:23:11.219 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 137 k 2022-09-13T07:23:11.219 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:23:11.288 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados2 2022-09-13T07:23:11.854 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:14 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:23:12.043 INFO:teuthology.orchestra.run.smithi036.stdout:libcephfs-devel-17.0.0-14918.g840d0541.el8.x86_ 37 kB/s | 30 kB 00:00 2022-09-13T07:23:12.043 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:23:12.044 INFO:teuthology.orchestra.run.smithi036.stdout:Total 37 kB/s | 30 kB 00:00 2022-09-13T07:23:12.044 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:23:12.048 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:23:12.049 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:23:12.192 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:23:12.196 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:23:12.570 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:23:12.738 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : libcephfs-devel-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:12.791 INFO:teuthology.orchestra.run.smithi083.stdout:Package librados2-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:23:12.865 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:23:12.866 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2022-09-13T07:23:12.866 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:23:12.956 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librbd1 2022-09-13T07:23:12.980 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:13.361 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : libcephfs-devel-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:13.362 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:13.363 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:23:13.363 INFO:teuthology.orchestra.run.smithi036.stdout: libcephfs-devel-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:23:13.364 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:13.364 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:23:13.461 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install librados2 2022-09-13T07:23:13.526 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:16 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:23:14.057 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:02:16 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:23:14.459 INFO:teuthology.orchestra.run.smithi083.stdout:Package librbd1-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:23:14.533 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:23:14.534 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2022-09-13T07:23:14.535 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:23:14.618 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rados 2022-09-13T07:23:15.060 INFO:teuthology.orchestra.run.smithi036.stdout:Package librados2-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:23:15.143 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:23:15.144 INFO:teuthology.orchestra.run.smithi036.stdout:Nothing to do. 2022-09-13T07:23:15.144 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:23:15.187 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:18 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:23:15.219 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install librbd1 2022-09-13T07:23:15.819 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:02:17 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:23:16.101 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rados-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:23:16.174 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:23:16.176 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2022-09-13T07:23:16.176 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:23:16.549 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rgw 2022-09-13T07:23:16.814 INFO:teuthology.orchestra.run.smithi036.stdout:Package librbd1-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:23:16.896 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:23:16.897 INFO:teuthology.orchestra.run.smithi036.stdout:Nothing to do. 2022-09-13T07:23:16.898 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:23:16.993 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install python3-rados 2022-09-13T07:23:17.111 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:20 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:23:17.591 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:02:19 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:23:18.053 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rgw-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:23:18.127 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:23:18.129 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2022-09-13T07:23:18.129 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:23:18.204 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-cephfs 2022-09-13T07:23:18.599 INFO:teuthology.orchestra.run.smithi036.stdout:Package python3-rados-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:23:18.681 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:23:18.683 INFO:teuthology.orchestra.run.smithi036.stdout:Nothing to do. 2022-09-13T07:23:18.683 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:23:18.765 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install python3-rgw 2022-09-13T07:23:18.770 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:21 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:23:19.365 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:02:21 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:23:19.703 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-cephfs-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:23:19.778 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:23:19.779 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2022-09-13T07:23:19.779 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:23:19.859 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rbd 2022-09-13T07:23:20.377 INFO:teuthology.orchestra.run.smithi036.stdout:Package python3-rgw-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:23:20.426 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:23 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:23:20.461 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:23:20.463 INFO:teuthology.orchestra.run.smithi036.stdout:Nothing to do. 2022-09-13T07:23:20.463 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:23:20.541 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install python3-cephfs 2022-09-13T07:23:21.141 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:02:23 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:23:21.356 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rbd-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:23:21.431 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:23:21.432 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2022-09-13T07:23:21.432 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:23:21.668 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-fuse 2022-09-13T07:23:22.147 INFO:teuthology.orchestra.run.smithi036.stdout:Package python3-cephfs-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:23:22.230 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:23:22.231 INFO:teuthology.orchestra.run.smithi036.stdout:Nothing to do. 2022-09-13T07:23:22.231 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:23:22.236 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:25 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:23:22.336 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install python3-rbd 2022-09-13T07:23:22.934 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:02:24 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:23:23.240 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:23:23.241 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:23.241 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2022-09-13T07:23:23.241 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:23.241 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:23:23.242 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse x86_64 2:17.0.0-14918.g840d0541.el8 ceph 95 k 2022-09-13T07:23:23.242 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:23.242 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:23:23.242 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:23.243 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2022-09-13T07:23:23.243 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:23.243 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 95 k 2022-09-13T07:23:23.244 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 280 k 2022-09-13T07:23:23.244 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:23:23.435 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-fuse-17.0.0-14918.g840d0541.el8.x86_64.rpm 488 kB/s | 95 kB 00:00 2022-09-13T07:23:23.436 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:23:23.437 INFO:teuthology.orchestra.run.smithi083.stdout:Total 485 kB/s | 95 kB 00:00 2022-09-13T07:23:23.437 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:23:23.447 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:23:23.448 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:23:23.494 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:23:23.497 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:23:23.606 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:23:23.770 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:23.931 INFO:teuthology.orchestra.run.smithi036.stdout:Package python3-rbd-2:17.0.0-14918.g840d0541.el8.x86_64 is already installed. 2022-09-13T07:23:24.015 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:23:24.016 INFO:teuthology.orchestra.run.smithi036.stdout:Nothing to do. 2022-09-13T07:23:24.016 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:23:24.017 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:24.092 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install rbd-fuse 2022-09-13T07:23:24.398 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:24.398 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:24.399 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:23:24.399 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:23:24.400 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:24.400 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:23:24.614 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-mirror 2022-09-13T07:23:24.692 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:02:26 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:23:25.192 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:28 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:23:25.769 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:23:25.770 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:25.770 INFO:teuthology.orchestra.run.smithi036.stdout: Package Arch Version Repo Size 2022-09-13T07:23:25.771 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:25.771 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:23:25.772 INFO:teuthology.orchestra.run.smithi036.stdout: rbd-fuse x86_64 2:17.0.0-14918.g840d0541.el8 ceph 95 k 2022-09-13T07:23:25.772 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:25.773 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:23:25.774 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:25.774 INFO:teuthology.orchestra.run.smithi036.stdout:Install 1 Package 2022-09-13T07:23:25.774 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:25.775 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 95 k 2022-09-13T07:23:25.775 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 280 k 2022-09-13T07:23:25.776 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:23:25.967 INFO:teuthology.orchestra.run.smithi036.stdout:rbd-fuse-17.0.0-14918.g840d0541.el8.x86_64.rpm 483 kB/s | 95 kB 00:00 2022-09-13T07:23:25.968 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:23:25.968 INFO:teuthology.orchestra.run.smithi036.stdout:Total 479 kB/s | 95 kB 00:00 2022-09-13T07:23:25.969 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:23:25.980 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:23:25.980 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:23:26.032 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:23:26.035 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:23:26.155 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:23:26.200 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:23:26.201 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:26.202 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2022-09-13T07:23:26.202 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:26.202 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:23:26.202 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror x86_64 2:17.0.0-14918.g840d0541.el8 ceph 3.5 M 2022-09-13T07:23:26.203 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:26.203 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:23:26.203 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:26.203 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2022-09-13T07:23:26.204 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:26.204 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 3.5 M 2022-09-13T07:23:26.204 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 16 M 2022-09-13T07:23:26.204 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:23:26.327 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : rbd-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:26.572 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-mirror-17.0.0-14918.g840d0541.el8.x86_64.rp 9.4 MB/s | 3.5 MB 00:00 2022-09-13T07:23:26.574 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:23:26.575 INFO:teuthology.orchestra.run.smithi083.stdout:Total 9.3 MB/s | 3.5 MB 00:00 2022-09-13T07:23:26.575 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:23:26.584 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:23:26.585 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:23:26.647 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:23:26.650 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:23:26.690 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: rbd-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:27.093 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:23:27.097 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : rbd-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:27.097 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:27.098 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:23:27.098 INFO:teuthology.orchestra.run.smithi036.stdout: rbd-fuse-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:23:27.098 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:27.098 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:23:27.283 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install rbd-mirror 2022-09-13T07:23:27.321 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-mirror-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:27.699 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-mirror-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:27.884 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:02:29 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:23:28.077 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-mirror-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:28.078 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:28.078 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:23:28.078 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:23:28.079 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:28.079 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:23:28.278 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-nbd 2022-09-13T07:23:28.859 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:31 ago on Tue 13 Sep 2022 07:20:57 AM UTC. 2022-09-13T07:23:28.971 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:23:28.972 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:28.972 INFO:teuthology.orchestra.run.smithi036.stdout: Package Arch Version Repo Size 2022-09-13T07:23:28.972 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:28.973 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:23:28.973 INFO:teuthology.orchestra.run.smithi036.stdout: rbd-mirror x86_64 2:17.0.0-14918.g840d0541.el8 ceph 3.5 M 2022-09-13T07:23:28.974 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:28.974 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:23:28.974 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:28.975 INFO:teuthology.orchestra.run.smithi036.stdout:Install 1 Package 2022-09-13T07:23:28.975 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:28.976 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 3.5 M 2022-09-13T07:23:28.976 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 16 M 2022-09-13T07:23:28.976 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:23:29.421 INFO:teuthology.orchestra.run.smithi036.stdout:rbd-mirror-17.0.0-14918.g840d0541.el8.x86_64.rp 7.7 MB/s | 3.5 MB 00:00 2022-09-13T07:23:29.422 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:23:29.423 INFO:teuthology.orchestra.run.smithi036.stdout:Total 7.6 MB/s | 3.5 MB 00:00 2022-09-13T07:23:29.423 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:23:29.436 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:23:29.436 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:23:29.500 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:23:29.503 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:23:29.868 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2022-09-13T07:23:29.869 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:29.870 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2022-09-13T07:23:29.870 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:29.870 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2022-09-13T07:23:29.870 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd x86_64 2:17.0.0-14918.g840d0541.el8 ceph 175 k 2022-09-13T07:23:29.871 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:29.871 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2022-09-13T07:23:29.871 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2022-09-13T07:23:29.871 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2022-09-13T07:23:29.871 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:29.872 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 175 k 2022-09-13T07:23:29.872 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 556 k 2022-09-13T07:23:29.872 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2022-09-13T07:23:29.966 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:23:30.068 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-nbd-17.0.0-14918.g840d0541.el8.x86_64.rpm 886 kB/s | 175 kB 00:00 2022-09-13T07:23:30.068 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:23:30.069 INFO:teuthology.orchestra.run.smithi083.stdout:Total 881 kB/s | 175 kB 00:00 2022-09-13T07:23:30.069 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2022-09-13T07:23:30.081 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2022-09-13T07:23:30.082 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2022-09-13T07:23:30.128 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2022-09-13T07:23:30.131 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2022-09-13T07:23:30.243 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2022-09-13T07:23:30.279 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : rbd-mirror-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:30.394 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-nbd-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:30.650 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-nbd-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:30.661 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: rbd-mirror-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:31.063 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : rbd-mirror-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:31.064 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:31.064 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:23:31.065 INFO:teuthology.orchestra.run.smithi036.stdout: rbd-mirror-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:23:31.065 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:31.066 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:23:31.077 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-nbd-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:31.077 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:31.077 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2022-09-13T07:23:31.078 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:23:31.078 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:31.079 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2022-09-13T07:23:31.238 DEBUG:teuthology.parallel:result is None 2022-09-13T07:23:31.262 DEBUG:teuthology.orchestra.run.smithi036:> sudo yum -y install rbd-nbd 2022-09-13T07:23:31.869 INFO:teuthology.orchestra.run.smithi036.stdout:Last metadata expiration check: 0:02:33 ago on Tue 13 Sep 2022 07:20:58 AM UTC. 2022-09-13T07:23:32.971 INFO:teuthology.orchestra.run.smithi036.stdout:Dependencies resolved. 2022-09-13T07:23:32.973 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:32.974 INFO:teuthology.orchestra.run.smithi036.stdout: Package Architecture Version Repository Size 2022-09-13T07:23:32.974 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:32.974 INFO:teuthology.orchestra.run.smithi036.stdout:Installing: 2022-09-13T07:23:32.975 INFO:teuthology.orchestra.run.smithi036.stdout: rbd-nbd x86_64 2:17.0.0-14918.g840d0541.el8 ceph 175 k 2022-09-13T07:23:32.976 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:32.976 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction Summary 2022-09-13T07:23:32.977 INFO:teuthology.orchestra.run.smithi036.stdout:================================================================================ 2022-09-13T07:23:32.977 INFO:teuthology.orchestra.run.smithi036.stdout:Install 1 Package 2022-09-13T07:23:32.978 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:32.978 INFO:teuthology.orchestra.run.smithi036.stdout:Total download size: 175 k 2022-09-13T07:23:32.979 INFO:teuthology.orchestra.run.smithi036.stdout:Installed size: 556 k 2022-09-13T07:23:32.980 INFO:teuthology.orchestra.run.smithi036.stdout:Downloading Packages: 2022-09-13T07:23:33.182 INFO:teuthology.orchestra.run.smithi036.stdout:rbd-nbd-17.0.0-14918.g840d0541.el8.x86_64.rpm 840 kB/s | 175 kB 00:00 2022-09-13T07:23:33.183 INFO:teuthology.orchestra.run.smithi036.stdout:-------------------------------------------------------------------------------- 2022-09-13T07:23:33.183 INFO:teuthology.orchestra.run.smithi036.stdout:Total 835 kB/s | 175 kB 00:00 2022-09-13T07:23:33.183 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction check 2022-09-13T07:23:33.195 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction check succeeded. 2022-09-13T07:23:33.195 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction test 2022-09-13T07:23:33.249 INFO:teuthology.orchestra.run.smithi036.stdout:Transaction test succeeded. 2022-09-13T07:23:33.252 INFO:teuthology.orchestra.run.smithi036.stdout:Running transaction 2022-09-13T07:23:33.370 INFO:teuthology.orchestra.run.smithi036.stdout: Preparing : 1/1 2022-09-13T07:23:33.584 INFO:teuthology.orchestra.run.smithi036.stdout: Installing : rbd-nbd-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:33.848 INFO:teuthology.orchestra.run.smithi036.stdout: Running scriptlet: rbd-nbd-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:34.228 INFO:teuthology.orchestra.run.smithi036.stdout: Verifying : rbd-nbd-2:17.0.0-14918.g840d0541.el8.x86_64 1/1 2022-09-13T07:23:34.229 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:34.229 INFO:teuthology.orchestra.run.smithi036.stdout:Installed: 2022-09-13T07:23:34.229 INFO:teuthology.orchestra.run.smithi036.stdout: rbd-nbd-2:17.0.0-14918.g840d0541.el8.x86_64 2022-09-13T07:23:34.230 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:34.230 INFO:teuthology.orchestra.run.smithi036.stdout:Complete! 2022-09-13T07:23:34.445 DEBUG:teuthology.parallel:result is None 2022-09-13T07:23:34.446 DEBUG:teuthology.orchestra.run.smithi036:> sudo lsb_release -is 2022-09-13T07:23:34.485 INFO:teuthology.orchestra.run.smithi036.stdout:CentOSStream 2022-09-13T07:23:34.487 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-13T07:23:34.488 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=840d05418c459144c92fb2a90d74a2ded29610da 2022-09-13T07:23:34.660 DEBUG:teuthology.orchestra.run.smithi036:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-09-13T07:23:34.684 INFO:teuthology.orchestra.run.smithi036.stdout:17.0.0-14918.g840d0541.el8 2022-09-13T07:23:34.686 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-14918.g840d0541.el8 2022-09-13T07:23:34.686 INFO:teuthology.task.install:The correct ceph version 17.0.0-14918.g840d0541 is installed. 2022-09-13T07:23:34.687 DEBUG:teuthology.orchestra.run.smithi083:> sudo lsb_release -is 2022-09-13T07:23:34.726 INFO:teuthology.orchestra.run.smithi083.stdout:CentOSStream 2022-09-13T07:23:34.728 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-13T07:23:34.728 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=840d05418c459144c92fb2a90d74a2ded29610da 2022-09-13T07:23:34.887 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-09-13T07:23:34.911 INFO:teuthology.orchestra.run.smithi083.stdout:17.0.0-14918.g840d0541.el8 2022-09-13T07:23:34.911 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-14918.g840d0541.el8 2022-09-13T07:23:34.912 INFO:teuthology.task.install:The correct ceph version 17.0.0-14918.g840d0541 is installed. 2022-09-13T07:23:34.912 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-09-13T07:23:34.913 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:34.913 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-09-13T07:23:34.941 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:23:34.942 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-09-13T07:23:34.979 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-09-13T07:23:34.980 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:34.980 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd of=/usr/bin/daemon-helper 2022-09-13T07:23:35.008 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-09-13T07:23:35.076 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:23:35.076 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/daemon-helper 2022-09-13T07:23:35.104 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-09-13T07:23:35.173 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-09-13T07:23:35.174 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:35.175 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd of=/usr/bin/adjust-ulimits 2022-09-13T07:23:35.203 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-09-13T07:23:35.271 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:23:35.272 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/adjust-ulimits 2022-09-13T07:23:35.300 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-09-13T07:23:35.367 INFO:teuthology.run_tasks:Running task ceph... 2022-09-13T07:23:35.457 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-09-13T07:23:35.458 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 777 /var/log/ceph 2022-09-13T07:23:35.461 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /var/log/ceph 2022-09-13T07:23:35.488 INFO:tasks.ceph:Disabling ceph logrotate... 2022-09-13T07:23:35.489 DEBUG:teuthology.orchestra.run.smithi036:> sudo rm -f -- /etc/logrotate.d/ceph 2022-09-13T07:23:35.530 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/logrotate.d/ceph 2022-09-13T07:23:35.558 INFO:tasks.ceph:Creating extra log directories... 2022-09-13T07:23:35.558 DEBUG:teuthology.orchestra.run.smithi036:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-09-13T07:23:35.599 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-09-13T07:23:35.648 INFO:tasks.ceph:Creating ceph cluster c1... 2022-09-13T07:23:35.649 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-09-13T07:23:35.650 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuvalif-2022-09-13_07:01:40-rgw:multisite-wip-yuval-fix-56572-distro-default-smithi/7030425', 'branch': 'wip-yuval-fix-56572', 'description': 'rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7030425', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuvalif-2022-09-13_07:01:40-rgw:multisite-wip-yuval-fix-56572-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuval-fix-56572'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '840d05418c459144c92fb2a90d74a2ded29610da', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '840d05418c459144c92fb2a90d74a2ded29610da'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuval-fix-56572', 'sha1': '840d05418c459144c92fb2a90d74a2ded29610da'}}, 'owner': 'scheduled_yuvalif@teuthology', 'priority': 50, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '840d05418c459144c92fb2a90d74a2ded29610da', 'sleep_before_teardown': 0, 'suite': 'rgw:multisite', 'suite_branch': 'wip-yuval-fix-56572', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '840d05418c459144c92fb2a90d74a2ded29610da', 'targets': {'smithi036.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwZ10DhqfIPFcUwNSkeR9qLDxnG1HoxeKhmdLj9JcAN5J/ZJ7gkiU4iMSbViuINM2NTvcD2/EL0PGkYz+BhowcJPciculEA/wU67kHrJUdjcxBug8sEgbDHF6t7iR47g0ksrysKN+YpSUseVtgCSrHRlawbpy2GHby45dGTVM8J6XB9KpPP+733LuNmL5/ZsSCi1zu3D/WkIqHeTHWXBBkUlJB+hjOiWhXS20e9hlBWTxv13p/IQ0fK+byN48zP5uxcHlFqapgOfXbI3SBhXlBJgRke/pcipC7+UyukyvFgi0sIsHyoFTgl31Ihw1qZ7Vfiok+f04z5Z01pbbrdvuh56cjrIUxZKunLxoSTnZhDIWNeTwb3Wx5VxyOkmsucjcXgUOXRI8DpIEYwBPpiVI6QfJpQ8pLMYerFKkvxqYibT7ypK66xRruGMRK8IDy5RAM7z3TRrF8ICNPVUiTxPbm5lVhI3zD35JoLewYboMo6mLRe9MBYo2ploAK0cGynf0=', 'smithi083.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKhIgPHWHpUJcIGujf2yUi90hlD6vf5UNtX5WGaN0K+n6B8OUcfmnejuPWM21jMK7T/xfUT+nYNdWaHs/me21DI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '840d05418c459144c92fb2a90d74a2ded29610da', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'main', 'teuthology_sha1': 'b58f2c18636eb10faa77ed3614abd00cb85dfc2c', 'timestamp': '2022-09-13_07:01:40', 'tube': 'smithi', 'user': 'yuvalif', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.20382'} 2022-09-13T07:23:35.651 DEBUG:teuthology.orchestra.run.smithi036:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2022-09-13T07:23:35.667 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2022-09-13T07:23:35.686 DEBUG:teuthology.orchestra.run.smithi036:> sudo install -d -m0777 -- /var/run/ceph 2022-09-13T07:23:35.725 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/run/ceph 2022-09-13T07:23:35.752 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:35.752 DEBUG:teuthology.orchestra.run.smithi036:> dd if=/scratch_devs of=/dev/stdout 2022-09-13T07:23:35.809 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-09-13T07:23:35.810 DEBUG:teuthology.orchestra.run.smithi036:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-09-13T07:23:35.868 INFO:teuthology.orchestra.run.smithi036.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-09-13T07:23:35.868 INFO:teuthology.orchestra.run.smithi036.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-13T07:23:35.868 INFO:teuthology.orchestra.run.smithi036.stdout:Device: 6h/6d Inode: 364893 Links: 1 2022-09-13T07:23:35.869 INFO:teuthology.orchestra.run.smithi036.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-13T07:23:35.869 INFO:teuthology.orchestra.run.smithi036.stdout:Context: system_u:object_r:device_t:s0 2022-09-13T07:23:35.869 INFO:teuthology.orchestra.run.smithi036.stdout:Access: 2022-09-13 07:23:30.438094300 +0000 2022-09-13T07:23:35.869 INFO:teuthology.orchestra.run.smithi036.stdout:Modify: 2022-09-13 07:20:08.268079774 +0000 2022-09-13T07:23:35.870 INFO:teuthology.orchestra.run.smithi036.stdout:Change: 2022-09-13 07:20:08.268079774 +0000 2022-09-13T07:23:35.870 INFO:teuthology.orchestra.run.smithi036.stdout: Birth: - 2022-09-13T07:23:35.877 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records in 2022-09-13T07:23:35.878 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records out 2022-09-13T07:23:35.878 INFO:teuthology.orchestra.run.smithi036.stderr:512 bytes copied, 0.00024381 s, 2.1 MB/s 2022-09-13T07:23:35.880 DEBUG:teuthology.orchestra.run.smithi036:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-09-13T07:23:35.939 INFO:teuthology.orchestra.run.smithi036.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-09-13T07:23:35.939 INFO:teuthology.orchestra.run.smithi036.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-13T07:23:35.939 INFO:teuthology.orchestra.run.smithi036.stdout:Device: 6h/6d Inode: 363712 Links: 1 2022-09-13T07:23:35.940 INFO:teuthology.orchestra.run.smithi036.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-13T07:23:35.940 INFO:teuthology.orchestra.run.smithi036.stdout:Context: system_u:object_r:device_t:s0 2022-09-13T07:23:35.940 INFO:teuthology.orchestra.run.smithi036.stdout:Access: 2022-09-13 07:23:30.438094300 +0000 2022-09-13T07:23:35.941 INFO:teuthology.orchestra.run.smithi036.stdout:Modify: 2022-09-13 07:20:08.827073873 +0000 2022-09-13T07:23:35.941 INFO:teuthology.orchestra.run.smithi036.stdout:Change: 2022-09-13 07:20:08.827073873 +0000 2022-09-13T07:23:35.941 INFO:teuthology.orchestra.run.smithi036.stdout: Birth: - 2022-09-13T07:23:35.949 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records in 2022-09-13T07:23:35.949 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records out 2022-09-13T07:23:35.949 INFO:teuthology.orchestra.run.smithi036.stderr:512 bytes copied, 0.000203891 s, 2.5 MB/s 2022-09-13T07:23:35.952 DEBUG:teuthology.orchestra.run.smithi036:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-09-13T07:23:36.010 INFO:teuthology.orchestra.run.smithi036.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-09-13T07:23:36.010 INFO:teuthology.orchestra.run.smithi036.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-13T07:23:36.011 INFO:teuthology.orchestra.run.smithi036.stdout:Device: 6h/6d Inode: 367765 Links: 1 2022-09-13T07:23:36.011 INFO:teuthology.orchestra.run.smithi036.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-13T07:23:36.011 INFO:teuthology.orchestra.run.smithi036.stdout:Context: system_u:object_r:device_t:s0 2022-09-13T07:23:36.012 INFO:teuthology.orchestra.run.smithi036.stdout:Access: 2022-09-13 07:23:30.438094300 +0000 2022-09-13T07:23:36.012 INFO:teuthology.orchestra.run.smithi036.stdout:Modify: 2022-09-13 07:20:09.401067815 +0000 2022-09-13T07:23:36.013 INFO:teuthology.orchestra.run.smithi036.stdout:Change: 2022-09-13 07:20:09.401067815 +0000 2022-09-13T07:23:36.013 INFO:teuthology.orchestra.run.smithi036.stdout: Birth: - 2022-09-13T07:23:36.019 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records in 2022-09-13T07:23:36.020 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records out 2022-09-13T07:23:36.020 INFO:teuthology.orchestra.run.smithi036.stderr:512 bytes copied, 0.000197475 s, 2.6 MB/s 2022-09-13T07:23:36.022 DEBUG:teuthology.orchestra.run.smithi036:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-09-13T07:23:36.080 INFO:teuthology.orchestra.run.smithi036.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-09-13T07:23:36.080 INFO:teuthology.orchestra.run.smithi036.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-13T07:23:36.080 INFO:teuthology.orchestra.run.smithi036.stdout:Device: 6h/6d Inode: 363024 Links: 1 2022-09-13T07:23:36.081 INFO:teuthology.orchestra.run.smithi036.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-13T07:23:36.081 INFO:teuthology.orchestra.run.smithi036.stdout:Context: system_u:object_r:device_t:s0 2022-09-13T07:23:36.081 INFO:teuthology.orchestra.run.smithi036.stdout:Access: 2022-09-13 07:23:30.438094300 +0000 2022-09-13T07:23:36.081 INFO:teuthology.orchestra.run.smithi036.stdout:Modify: 2022-09-13 07:20:09.964061872 +0000 2022-09-13T07:23:36.082 INFO:teuthology.orchestra.run.smithi036.stdout:Change: 2022-09-13 07:20:09.964061872 +0000 2022-09-13T07:23:36.082 INFO:teuthology.orchestra.run.smithi036.stdout: Birth: - 2022-09-13T07:23:36.090 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records in 2022-09-13T07:23:36.090 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records out 2022-09-13T07:23:36.091 INFO:teuthology.orchestra.run.smithi036.stderr:512 bytes copied, 0.000224555 s, 2.3 MB/s 2022-09-13T07:23:36.093 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-13T07:23:36.094 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi036.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-13T07:23:36.094 INFO:tasks.ceph:Generating config... 2022-09-13T07:23:36.097 INFO:tasks.ceph:[client] debug rgw = 20 2022-09-13T07:23:36.097 INFO:tasks.ceph:[client] debug rgw sync = 20 2022-09-13T07:23:36.098 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2022-09-13T07:23:36.098 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2022-09-13T07:23:36.099 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-13T07:23:36.099 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2022-09-13T07:23:36.100 INFO:tasks.ceph:[client] rgw data log num shards = 4 2022-09-13T07:23:36.100 INFO:tasks.ceph:[client] rgw md log max shards = 4 2022-09-13T07:23:36.101 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2022-09-13T07:23:36.102 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2022-09-13T07:23:36.102 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2022-09-13T07:23:36.103 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2022-09-13T07:23:36.103 INFO:tasks.ceph:[client] setgroup = ceph 2022-09-13T07:23:36.104 INFO:tasks.ceph:[client] setuser = ceph 2022-09-13T07:23:36.104 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2022-09-13T07:23:36.105 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-09-13T07:23:36.106 INFO:tasks.ceph:[mgr] debug ms = 1 2022-09-13T07:23:36.106 INFO:tasks.ceph:[mon] debug mon = 20 2022-09-13T07:23:36.107 INFO:tasks.ceph:[mon] debug ms = 1 2022-09-13T07:23:36.107 INFO:tasks.ceph:[mon] debug paxos = 20 2022-09-13T07:23:36.108 INFO:tasks.ceph:[osd] debug ms = 1 2022-09-13T07:23:36.108 INFO:tasks.ceph:[osd] debug osd = 20 2022-09-13T07:23:36.108 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2022-09-13T07:23:36.109 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2022-09-13T07:23:36.109 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2022-09-13T07:23:36.109 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2022-09-13T07:23:36.110 INFO:tasks.ceph:Setting up c1.mon.a... 2022-09-13T07:23:36.110 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2022-09-13T07:23:36.186 INFO:teuthology.orchestra.run.smithi036.stdout:creating /etc/ceph/c1.keyring 2022-09-13T07:23:36.189 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2022-09-13T07:23:36.236 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-09-13T07:23:36.264 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.36')] 2022-09-13T07:23:36.265 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.36'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2022-09-13T07:23:36.266 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:36.267 DEBUG:teuthology.orchestra.run.smithi036:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-09-13T07:23:36.322 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.36 --print /home/ubuntu/cephtest/c1.monmap 2022-09-13T07:23:36.405 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:23:36.405 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:23:36.407 INFO:teuthology.orchestra.run.smithi036.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2022-09-13T07:23:36.407 INFO:teuthology.orchestra.run.smithi036.stdout:monmaptool: generated fsid 5cb3badf-0576-4aaa-b80a-741bd397fa0b 2022-09-13T07:23:36.407 INFO:teuthology.orchestra.run.smithi036.stdout:setting min_mon_release = octopus 2022-09-13T07:23:36.407 INFO:teuthology.orchestra.run.smithi036.stdout:epoch 0 2022-09-13T07:23:36.408 INFO:teuthology.orchestra.run.smithi036.stdout:fsid 5cb3badf-0576-4aaa-b80a-741bd397fa0b 2022-09-13T07:23:36.408 INFO:teuthology.orchestra.run.smithi036.stdout:last_changed 2022-09-13T07:23:36.403801+0000 2022-09-13T07:23:36.408 INFO:teuthology.orchestra.run.smithi036.stdout:created 2022-09-13T07:23:36.403801+0000 2022-09-13T07:23:36.408 INFO:teuthology.orchestra.run.smithi036.stdout:min_mon_release 15 (octopus) 2022-09-13T07:23:36.409 INFO:teuthology.orchestra.run.smithi036.stdout:election_strategy: 1 2022-09-13T07:23:36.409 INFO:teuthology.orchestra.run.smithi036.stdout:0: [v2:172.21.15.36:3300/0,v1:172.21.15.36:6789/0] mon.a 2022-09-13T07:23:36.409 INFO:teuthology.orchestra.run.smithi036.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2022-09-13T07:23:36.410 DEBUG:teuthology.orchestra.run.smithi036:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-09-13T07:23:36.465 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 5cb3badf-0576-4aaa-b80a-741bd397fa0b... 2022-09-13T07:23:36.466 DEBUG:teuthology.orchestra.run.smithi036:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2022-09-13T07:23:36.507 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2022-09-13T07:23:36.550 INFO:teuthology.orchestra.run.smithi036.stdout:[global] 2022-09-13T07:23:36.551 INFO:teuthology.orchestra.run.smithi036.stdout: chdir = "" 2022-09-13T07:23:36.551 INFO:teuthology.orchestra.run.smithi036.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-13T07:23:36.552 INFO:teuthology.orchestra.run.smithi036.stdout: auth supported = cephx 2022-09-13T07:23:36.552 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.553 INFO:teuthology.orchestra.run.smithi036.stdout: filestore xattr use omap = true 2022-09-13T07:23:36.553 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.554 INFO:teuthology.orchestra.run.smithi036.stdout: mon clock drift allowed = 1.000 2022-09-13T07:23:36.554 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.554 INFO:teuthology.orchestra.run.smithi036.stdout: osd crush chooseleaf type = 0 2022-09-13T07:23:36.554 INFO:teuthology.orchestra.run.smithi036.stdout: auth debug = true 2022-09-13T07:23:36.555 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.555 INFO:teuthology.orchestra.run.smithi036.stdout: ms die on old message = true 2022-09-13T07:23:36.555 INFO:teuthology.orchestra.run.smithi036.stdout: ms die on bug = true 2022-09-13T07:23:36.555 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.555 INFO:teuthology.orchestra.run.smithi036.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-13T07:23:36.556 INFO:teuthology.orchestra.run.smithi036.stdout: mon pg warn max object skew = 0 2022-09-13T07:23:36.556 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.557 INFO:teuthology.orchestra.run.smithi036.stdout: # disable pg_autoscaler by default for new pools 2022-09-13T07:23:36.557 INFO:teuthology.orchestra.run.smithi036.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-13T07:23:36.557 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.557 INFO:teuthology.orchestra.run.smithi036.stdout: osd pool default size = 2 2022-09-13T07:23:36.558 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.558 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd allow primary affinity = true 2022-09-13T07:23:36.558 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd allow pg remap = true 2022-09-13T07:23:36.559 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on legacy crush tunables = false 2022-09-13T07:23:36.559 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on crush straw calc version zero = false 2022-09-13T07:23:36.559 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on no sortbitwise = false 2022-09-13T07:23:36.559 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on osd down out interval zero = false 2022-09-13T07:23:36.560 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on too few osds = false 2022-09-13T07:23:36.560 INFO:teuthology.orchestra.run.smithi036.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-13T07:23:36.560 INFO:teuthology.orchestra.run.smithi036.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-13T07:23:36.560 INFO:teuthology.orchestra.run.smithi036.stdout: mon_allow_pool_size_one = true 2022-09-13T07:23:36.560 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.561 INFO:teuthology.orchestra.run.smithi036.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-13T07:23:36.561 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.561 INFO:teuthology.orchestra.run.smithi036.stdout: osd default data pool replay window = 5 2022-09-13T07:23:36.561 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.562 INFO:teuthology.orchestra.run.smithi036.stdout: mon allow pool delete = true 2022-09-13T07:23:36.562 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.562 INFO:teuthology.orchestra.run.smithi036.stdout: mon cluster log file level = debug 2022-09-13T07:23:36.562 INFO:teuthology.orchestra.run.smithi036.stdout: debug asserts on shutdown = true 2022-09-13T07:23:36.562 INFO:teuthology.orchestra.run.smithi036.stdout: mon health detail to clog = false 2022-09-13T07:23:36.563 INFO:teuthology.orchestra.run.smithi036.stdout: mon host = 172.21.15.36 2022-09-13T07:23:36.563 INFO:teuthology.orchestra.run.smithi036.stdout: fsid = 5cb3badf-0576-4aaa-b80a-741bd397fa0b 2022-09-13T07:23:36.563 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.563 INFO:teuthology.orchestra.run.smithi036.stdout:[osd] 2022-09-13T07:23:36.564 INFO:teuthology.orchestra.run.smithi036.stdout: osd journal size = 100 2022-09-13T07:23:36.564 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.564 INFO:teuthology.orchestra.run.smithi036.stdout: osd scrub load threshold = 5.0 2022-09-13T07:23:36.564 INFO:teuthology.orchestra.run.smithi036.stdout: osd scrub max interval = 600 2022-09-13T07:23:36.564 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.565 INFO:teuthology.orchestra.run.smithi036.stdout: osd recover clone overlap = true 2022-09-13T07:23:36.565 INFO:teuthology.orchestra.run.smithi036.stdout: osd recovery max chunk = 1048576 2022-09-13T07:23:36.565 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.565 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug shutdown = true 2022-09-13T07:23:36.565 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug op order = true 2022-09-13T07:23:36.566 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug verify stray on activate = true 2022-09-13T07:23:36.566 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.566 INFO:teuthology.orchestra.run.smithi036.stdout: osd open classes on start = true 2022-09-13T07:23:36.566 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug pg log writeout = true 2022-09-13T07:23:36.567 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.567 INFO:teuthology.orchestra.run.smithi036.stdout: osd deep scrub update digest min age = 30 2022-09-13T07:23:36.567 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.567 INFO:teuthology.orchestra.run.smithi036.stdout: osd map max advance = 10 2022-09-13T07:23:36.567 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.568 INFO:teuthology.orchestra.run.smithi036.stdout: journal zero on create = true 2022-09-13T07:23:36.568 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.568 INFO:teuthology.orchestra.run.smithi036.stdout: filestore ondisk finisher threads = 3 2022-09-13T07:23:36.569 INFO:teuthology.orchestra.run.smithi036.stdout: filestore apply finisher threads = 3 2022-09-13T07:23:36.569 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.569 INFO:teuthology.orchestra.run.smithi036.stdout: bdev debug aio = true 2022-09-13T07:23:36.569 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug misdirected ops = true 2022-09-13T07:23:36.569 INFO:teuthology.orchestra.run.smithi036.stdout: debug ms = 1 2022-09-13T07:23:36.570 INFO:teuthology.orchestra.run.smithi036.stdout: debug osd = 20 2022-09-13T07:23:36.570 INFO:teuthology.orchestra.run.smithi036.stdout: rocksdb delete range threshold = 0 2022-09-13T07:23:36.570 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.570 INFO:teuthology.orchestra.run.smithi036.stdout:[mgr] 2022-09-13T07:23:36.571 INFO:teuthology.orchestra.run.smithi036.stdout: debug ms = 1 2022-09-13T07:23:36.571 INFO:teuthology.orchestra.run.smithi036.stdout: debug mgr = 20 2022-09-13T07:23:36.571 INFO:teuthology.orchestra.run.smithi036.stdout: debug mon = 20 2022-09-13T07:23:36.571 INFO:teuthology.orchestra.run.smithi036.stdout: debug auth = 20 2022-09-13T07:23:36.572 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min pgs per osd = 4 2022-09-13T07:23:36.572 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min bytes per osd = 10 2022-09-13T07:23:36.573 INFO:teuthology.orchestra.run.smithi036.stdout: mgr/telemetry/nag = false 2022-09-13T07:23:36.573 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.573 INFO:teuthology.orchestra.run.smithi036.stdout:[mon] 2022-09-13T07:23:36.574 INFO:teuthology.orchestra.run.smithi036.stdout: debug ms = 1 2022-09-13T07:23:36.574 INFO:teuthology.orchestra.run.smithi036.stdout: debug mon = 20 2022-09-13T07:23:36.575 INFO:teuthology.orchestra.run.smithi036.stdout: debug paxos = 20 2022-09-13T07:23:36.576 INFO:teuthology.orchestra.run.smithi036.stdout: debug auth = 20 2022-09-13T07:23:36.576 INFO:teuthology.orchestra.run.smithi036.stdout: mon data avail warn = 5 2022-09-13T07:23:36.577 INFO:teuthology.orchestra.run.smithi036.stdout: mon mgr mkfs grace = 240 2022-09-13T07:23:36.577 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min pgs per osd = 4 2022-09-13T07:23:36.577 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd reporter subtree level = osd 2022-09-13T07:23:36.578 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd prime pg temp = true 2022-09-13T07:23:36.578 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min bytes per osd = 10 2022-09-13T07:23:36.579 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.579 INFO:teuthology.orchestra.run.smithi036.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-13T07:23:36.579 INFO:teuthology.orchestra.run.smithi036.stdout: auth mon ticket ttl = 660 # 11m 2022-09-13T07:23:36.580 INFO:teuthology.orchestra.run.smithi036.stdout: auth service ticket ttl = 240 # 4m 2022-09-13T07:23:36.580 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.580 INFO:teuthology.orchestra.run.smithi036.stdout: # don't complain about insecure global_id in the test suite 2022-09-13T07:23:36.581 INFO:teuthology.orchestra.run.smithi036.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-13T07:23:36.581 INFO:teuthology.orchestra.run.smithi036.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-13T07:23:36.581 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.582 INFO:teuthology.orchestra.run.smithi036.stdout: # 1m isn't quite enough 2022-09-13T07:23:36.582 INFO:teuthology.orchestra.run.smithi036.stdout: mon_down_mkfs_grace = 2m 2022-09-13T07:23:36.583 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.583 INFO:teuthology.orchestra.run.smithi036.stdout: mon_warn_on_filestore_osds = false 2022-09-13T07:23:36.583 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:36.584 INFO:teuthology.orchestra.run.smithi036.stdout:[client] 2022-09-13T07:23:36.584 INFO:teuthology.orchestra.run.smithi036.stdout: rgw cache enabled = true 2022-09-13T07:23:36.584 INFO:teuthology.orchestra.run.smithi036.stdout: rgw enable ops log = true 2022-09-13T07:23:36.585 INFO:teuthology.orchestra.run.smithi036.stdout: rgw enable usage log = true 2022-09-13T07:23:36.585 INFO:teuthology.orchestra.run.smithi036.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-13T07:23:36.585 INFO:teuthology.orchestra.run.smithi036.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-13T07:23:36.585 INFO:teuthology.orchestra.run.smithi036.stdout: debug rgw = 20 2022-09-13T07:23:36.586 INFO:teuthology.orchestra.run.smithi036.stdout: debug rgw sync = 20 2022-09-13T07:23:36.586 INFO:teuthology.orchestra.run.smithi036.stdout: rgw crypt require ssl = False 2022-09-13T07:23:36.586 INFO:teuthology.orchestra.run.smithi036.stdout: rgw crypt s3 kms backend = testing 2022-09-13T07:23:36.587 INFO:teuthology.orchestra.run.smithi036.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-13T07:23:36.587 INFO:teuthology.orchestra.run.smithi036.stdout: rgw curl low speed time = 300 2022-09-13T07:23:36.587 INFO:teuthology.orchestra.run.smithi036.stdout: rgw data log num shards = 4 2022-09-13T07:23:36.587 INFO:teuthology.orchestra.run.smithi036.stdout: rgw md log max shards = 4 2022-09-13T07:23:36.587 INFO:teuthology.orchestra.run.smithi036.stdout: rgw sync data inject err probability = 0.1 2022-09-13T07:23:36.588 INFO:teuthology.orchestra.run.smithi036.stdout: rgw sync log trim interval = 0 2022-09-13T07:23:36.588 INFO:teuthology.orchestra.run.smithi036.stdout: rgw sync meta inject err probability = 0.1 2022-09-13T07:23:36.588 INFO:teuthology.orchestra.run.smithi036.stdout: rgw sync obj etag verify = True 2022-09-13T07:23:36.588 INFO:teuthology.orchestra.run.smithi036.stdout: setgroup = ceph 2022-09-13T07:23:36.589 INFO:teuthology.orchestra.run.smithi036.stdout: setuser = ceph 2022-09-13T07:23:36.589 INFO:teuthology.orchestra.run.smithi036.stdout:[mon.a] 2022-09-13T07:23:36.589 INFO:teuthology.orchestra.run.smithi036.stdout:[client.0] 2022-09-13T07:23:36.589 INFO:teuthology.orchestra.run.smithi036.stdout: rgw data notify interval msec = 0 2022-09-13T07:23:36.589 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.0] 2022-09-13T07:23:36.590 INFO:teuthology.orchestra.run.smithi036.stdout: osd_max_omap_entries_per_request = 10 2022-09-13T07:23:36.590 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.1] 2022-09-13T07:23:36.590 INFO:teuthology.orchestra.run.smithi036.stdout: osd_max_omap_entries_per_request = 1000 2022-09-13T07:23:36.591 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.2] 2022-09-13T07:23:36.591 INFO:teuthology.orchestra.run.smithi036.stdout: osd_max_omap_entries_per_request = 10000 2022-09-13T07:23:36.593 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2022-09-13T07:23:36.593 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2022-09-13T07:23:36.593 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-13T07:23:36.593 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2022-09-13T07:23:36.594 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.594 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2022-09-13T07:23:36.594 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.595 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2022-09-13T07:23:36.595 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.595 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2022-09-13T07:23:36.595 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2022-09-13T07:23:36.596 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.596 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2022-09-13T07:23:36.596 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2022-09-13T07:23:36.596 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.597 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-13T07:23:36.597 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2022-09-13T07:23:36.597 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.597 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2022-09-13T07:23:36.597 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-13T07:23:36.598 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.598 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2022-09-13T07:23:36.598 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.598 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2022-09-13T07:23:36.599 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2022-09-13T07:23:36.599 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2022-09-13T07:23:36.599 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2022-09-13T07:23:36.599 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2022-09-13T07:23:36.599 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2022-09-13T07:23:36.600 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2022-09-13T07:23:36.600 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-13T07:23:36.600 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-13T07:23:36.600 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2022-09-13T07:23:36.601 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.601 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-13T07:23:36.601 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.601 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2022-09-13T07:23:36.601 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.602 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2022-09-13T07:23:36.602 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.602 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2022-09-13T07:23:36.602 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2022-09-13T07:23:36.603 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2022-09-13T07:23:36.603 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = 172.21.15.36 2022-09-13T07:23:36.603 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 5cb3badf-0576-4aaa-b80a-741bd397fa0b 2022-09-13T07:23:36.603 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.603 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2022-09-13T07:23:36.604 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2022-09-13T07:23:36.604 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.604 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2022-09-13T07:23:36.604 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2022-09-13T07:23:36.604 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.605 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2022-09-13T07:23:36.605 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2022-09-13T07:23:36.605 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.606 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2022-09-13T07:23:36.606 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2022-09-13T07:23:36.606 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2022-09-13T07:23:36.606 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.607 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2022-09-13T07:23:36.607 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2022-09-13T07:23:36.607 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.608 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2022-09-13T07:23:36.608 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.608 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2022-09-13T07:23:36.608 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.608 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2022-09-13T07:23:36.609 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.609 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2022-09-13T07:23:36.609 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2022-09-13T07:23:36.609 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.610 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2022-09-13T07:23:36.610 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2022-09-13T07:23:36.610 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2022-09-13T07:23:36.611 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2022-09-13T07:23:36.611 INFO:teuthology.orchestra.run.smithi083.stdout: rocksdb delete range threshold = 0 2022-09-13T07:23:36.611 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.612 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2022-09-13T07:23:36.612 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2022-09-13T07:23:36.612 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2022-09-13T07:23:36.613 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2022-09-13T07:23:36.613 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2022-09-13T07:23:36.613 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2022-09-13T07:23:36.614 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2022-09-13T07:23:36.614 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2022-09-13T07:23:36.614 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.615 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2022-09-13T07:23:36.615 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2022-09-13T07:23:36.615 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2022-09-13T07:23:36.616 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2022-09-13T07:23:36.616 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2022-09-13T07:23:36.616 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2022-09-13T07:23:36.617 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2022-09-13T07:23:36.617 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2022-09-13T07:23:36.617 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2022-09-13T07:23:36.618 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2022-09-13T07:23:36.618 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2022-09-13T07:23:36.619 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.619 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-13T07:23:36.619 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2022-09-13T07:23:36.620 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2022-09-13T07:23:36.620 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.620 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2022-09-13T07:23:36.621 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-13T07:23:36.621 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-13T07:23:36.621 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.622 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2022-09-13T07:23:36.622 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2022-09-13T07:23:36.623 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.623 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2022-09-13T07:23:36.623 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:23:36.624 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2022-09-13T07:23:36.624 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2022-09-13T07:23:36.624 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2022-09-13T07:23:36.625 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2022-09-13T07:23:36.625 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-13T07:23:36.625 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-13T07:23:36.626 INFO:teuthology.orchestra.run.smithi083.stdout: debug rgw = 20 2022-09-13T07:23:36.626 INFO:teuthology.orchestra.run.smithi083.stdout: debug rgw sync = 20 2022-09-13T07:23:36.626 INFO:teuthology.orchestra.run.smithi083.stdout: rgw crypt require ssl = False 2022-09-13T07:23:36.627 INFO:teuthology.orchestra.run.smithi083.stdout: rgw crypt s3 kms backend = testing 2022-09-13T07:23:36.627 INFO:teuthology.orchestra.run.smithi083.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-13T07:23:36.627 INFO:teuthology.orchestra.run.smithi083.stdout: rgw curl low speed time = 300 2022-09-13T07:23:36.627 INFO:teuthology.orchestra.run.smithi083.stdout: rgw data log num shards = 4 2022-09-13T07:23:36.627 INFO:teuthology.orchestra.run.smithi083.stdout: rgw md log max shards = 4 2022-09-13T07:23:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: rgw sync data inject err probability = 0.1 2022-09-13T07:23:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: rgw sync log trim interval = 0 2022-09-13T07:23:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: rgw sync meta inject err probability = 0.1 2022-09-13T07:23:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: rgw sync obj etag verify = True 2022-09-13T07:23:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: setgroup = ceph 2022-09-13T07:23:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: setuser = ceph 2022-09-13T07:23:36.629 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2022-09-13T07:23:36.629 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2022-09-13T07:23:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: rgw data notify interval msec = 0 2022-09-13T07:23:36.630 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.0] 2022-09-13T07:23:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: osd_max_omap_entries_per_request = 10 2022-09-13T07:23:36.630 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.1] 2022-09-13T07:23:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: osd_max_omap_entries_per_request = 1000 2022-09-13T07:23:36.631 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.2] 2022-09-13T07:23:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: osd_max_omap_entries_per_request = 10000 2022-09-13T07:23:36.631 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2022-09-13T07:23:36.632 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2022-09-13T07:23:36.691 INFO:tasks.ceph:Copying monmap to all nodes... 2022-09-13T07:23:36.691 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:36.692 DEBUG:teuthology.orchestra.run.smithi036:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2022-09-13T07:23:36.710 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:36.710 DEBUG:teuthology.orchestra.run.smithi036:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2022-09-13T07:23:36.769 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi036.front.sepia.ceph.com 2022-09-13T07:23:36.770 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:36.770 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd of=/etc/ceph/c1.keyring 2022-09-13T07:23:36.771 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-09-13T07:23:36.847 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:36.847 DEBUG:teuthology.orchestra.run.smithi036:> dd of=/home/ubuntu/cephtest/c1.monmap 2022-09-13T07:23:36.905 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi083.front.sepia.ceph.com 2022-09-13T07:23:36.906 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:23:36.907 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/c1.keyring 2022-09-13T07:23:36.908 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-09-13T07:23:36.947 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:23:36.948 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/c1.monmap 2022-09-13T07:23:37.005 INFO:tasks.ceph:Setting up mon nodes... 2022-09-13T07:23:37.006 INFO:tasks.ceph:Setting up mgr nodes... 2022-09-13T07:23:37.006 DEBUG:teuthology.orchestra.run.smithi036:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2022-09-13T07:23:37.060 INFO:teuthology.orchestra.run.smithi036.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2022-09-13T07:23:37.063 INFO:tasks.ceph:Setting up mds nodes... 2022-09-13T07:23:37.063 INFO:tasks.ceph_client:Setting up client nodes... 2022-09-13T07:23:37.064 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2022-09-13T07:23:37.108 INFO:teuthology.orchestra.run.smithi036.stdout:creating /etc/ceph/c1.client.0.keyring 2022-09-13T07:23:37.121 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2022-09-13T07:23:37.205 INFO:teuthology.orchestra.run.smithi036.stdout:creating /etc/ceph/c1.client.1.keyring 2022-09-13T07:23:37.217 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-09-13T07:23:37.218 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi036.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-13T07:23:37.218 DEBUG:teuthology.orchestra.run.smithi036:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2022-09-13T07:23:37.245 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-13T07:23:37.246 INFO:tasks.ceph:role: c1.osd.0 2022-09-13T07:23:37.246 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi036.front.sepia.ceph.com 2022-09-13T07:23:37.247 DEBUG:teuthology.orchestra.run.smithi036:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-09-13T07:23:37.321 INFO:teuthology.orchestra.run.smithi036.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-09-13T07:23:37.322 INFO:teuthology.orchestra.run.smithi036.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-13T07:23:37.322 INFO:teuthology.orchestra.run.smithi036.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-13T07:23:37.323 INFO:teuthology.orchestra.run.smithi036.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-13T07:23:37.323 INFO:teuthology.orchestra.run.smithi036.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-13T07:23:37.323 INFO:teuthology.orchestra.run.smithi036.stdout: = sunit=0 swidth=0 blks 2022-09-13T07:23:37.324 INFO:teuthology.orchestra.run.smithi036.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-13T07:23:37.324 INFO:teuthology.orchestra.run.smithi036.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-13T07:23:37.325 INFO:teuthology.orchestra.run.smithi036.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-13T07:23:37.325 INFO:teuthology.orchestra.run.smithi036.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-13T07:23:37.746 INFO:teuthology.orchestra.run.smithi036.stdout:Discarding blocks...Done. 2022-09-13T07:23:37.748 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi036.front.sepia.ceph.com -o noatime 2022-09-13T07:23:37.749 DEBUG:teuthology.orchestra.run.smithi036:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2022-09-13T07:23:37.917 DEBUG:teuthology.orchestra.run.smithi036:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2022-09-13T07:23:37.961 DEBUG:teuthology.orchestra.run.smithi036:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2022-09-13T07:23:38.038 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-13T07:23:38.039 INFO:tasks.ceph:role: c1.osd.1 2022-09-13T07:23:38.039 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi036.front.sepia.ceph.com 2022-09-13T07:23:38.040 DEBUG:teuthology.orchestra.run.smithi036:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-09-13T07:23:38.118 INFO:teuthology.orchestra.run.smithi036.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-09-13T07:23:38.118 INFO:teuthology.orchestra.run.smithi036.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-13T07:23:38.119 INFO:teuthology.orchestra.run.smithi036.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-13T07:23:38.119 INFO:teuthology.orchestra.run.smithi036.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-13T07:23:38.119 INFO:teuthology.orchestra.run.smithi036.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-13T07:23:38.119 INFO:teuthology.orchestra.run.smithi036.stdout: = sunit=0 swidth=0 blks 2022-09-13T07:23:38.120 INFO:teuthology.orchestra.run.smithi036.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-13T07:23:38.120 INFO:teuthology.orchestra.run.smithi036.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-13T07:23:38.120 INFO:teuthology.orchestra.run.smithi036.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-13T07:23:38.120 INFO:teuthology.orchestra.run.smithi036.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-13T07:23:38.538 INFO:teuthology.orchestra.run.smithi036.stdout:Discarding blocks...Done. 2022-09-13T07:23:38.538 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi036.front.sepia.ceph.com -o noatime 2022-09-13T07:23:38.540 DEBUG:teuthology.orchestra.run.smithi036:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2022-09-13T07:23:38.661 DEBUG:teuthology.orchestra.run.smithi036:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2022-09-13T07:23:38.736 DEBUG:teuthology.orchestra.run.smithi036:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2022-09-13T07:23:38.807 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-13T07:23:38.808 INFO:tasks.ceph:role: c1.osd.2 2022-09-13T07:23:38.808 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi036.front.sepia.ceph.com 2022-09-13T07:23:38.808 DEBUG:teuthology.orchestra.run.smithi036:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-09-13T07:23:38.886 INFO:teuthology.orchestra.run.smithi036.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-09-13T07:23:38.887 INFO:teuthology.orchestra.run.smithi036.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-13T07:23:38.888 INFO:teuthology.orchestra.run.smithi036.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-13T07:23:38.888 INFO:teuthology.orchestra.run.smithi036.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-13T07:23:38.889 INFO:teuthology.orchestra.run.smithi036.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-13T07:23:38.889 INFO:teuthology.orchestra.run.smithi036.stdout: = sunit=0 swidth=0 blks 2022-09-13T07:23:38.890 INFO:teuthology.orchestra.run.smithi036.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-13T07:23:38.890 INFO:teuthology.orchestra.run.smithi036.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-13T07:23:38.891 INFO:teuthology.orchestra.run.smithi036.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-13T07:23:38.891 INFO:teuthology.orchestra.run.smithi036.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-13T07:23:39.330 INFO:teuthology.orchestra.run.smithi036.stdout:Discarding blocks...Done. 2022-09-13T07:23:39.332 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi036.front.sepia.ceph.com -o noatime 2022-09-13T07:23:39.332 DEBUG:teuthology.orchestra.run.smithi036:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2022-09-13T07:23:39.493 DEBUG:teuthology.orchestra.run.smithi036:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2022-09-13T07:23:39.529 DEBUG:teuthology.orchestra.run.smithi036:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2022-09-13T07:23:39.619 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:39.616+0000 7f7aa081d440 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2022-09-13T07:23:39.620 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:39.617+0000 7f7aa081d440 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2022-09-13T07:23:39.621 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:39.617+0000 7f7aa081d440 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-09-13T07:23:39.621 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:39.617+0000 7f7aa081d440 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-09-13T07:23:39.622 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:39.617+0000 7f7aa081d440 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-09-13T07:23:39.627 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:39.624+0000 7f7aa081d440 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2022-09-13T07:23:41.699 DEBUG:teuthology.orchestra.run.smithi036:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2022-09-13T07:23:41.734 DEBUG:teuthology.orchestra.run.smithi036:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2022-09-13T07:23:41.830 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:41.826+0000 7f5bc9fa0440 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2022-09-13T07:23:41.830 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:41.826+0000 7f5bc9fa0440 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2022-09-13T07:23:41.830 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:41.826+0000 7f5bc9fa0440 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-09-13T07:23:41.831 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:41.826+0000 7f5bc9fa0440 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-09-13T07:23:41.831 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:41.826+0000 7f5bc9fa0440 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-09-13T07:23:41.831 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:41.827+0000 7f5bc9fa0440 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2022-09-13T07:23:43.901 DEBUG:teuthology.orchestra.run.smithi036:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2022-09-13T07:23:43.936 DEBUG:teuthology.orchestra.run.smithi036:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2022-09-13T07:23:44.030 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:44.025+0000 7f5c88aef440 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2022-09-13T07:23:44.030 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:44.025+0000 7f5c88aef440 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2022-09-13T07:23:44.031 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:44.025+0000 7f5c88aef440 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-09-13T07:23:44.031 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:44.025+0000 7f5c88aef440 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-09-13T07:23:44.031 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:44.025+0000 7f5c88aef440 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-09-13T07:23:44.032 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:23:44.026+0000 7f5c88aef440 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2022-09-13T07:23:46.103 DEBUG:teuthology.orchestra.run.smithi036:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2022-09-13T07:23:46.137 INFO:tasks.ceph:Reading keys from all nodes... 2022-09-13T07:23:46.139 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:46.140 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2022-09-13T07:23:46.211 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:46.212 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2022-09-13T07:23:46.282 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:46.283 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2022-09-13T07:23:46.353 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:46.353 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2022-09-13T07:23:46.422 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:46.423 DEBUG:teuthology.orchestra.run.smithi036:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2022-09-13T07:23:46.481 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:46.482 DEBUG:teuthology.orchestra.run.smithi036:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2022-09-13T07:23:46.540 INFO:tasks.ceph:Adding keys to all mons... 2022-09-13T07:23:46.540 DEBUG:teuthology.orchestra.run.smithi036:> sudo tee -a /etc/ceph/c1.keyring 2022-09-13T07:23:46.607 INFO:teuthology.orchestra.run.smithi036.stdout:[mgr.x] 2022-09-13T07:23:46.608 INFO:teuthology.orchestra.run.smithi036.stdout: key = AQD5LyBjki6EAxAAl+bgj/j3IZ1b7olzK+MTOw== 2022-09-13T07:23:46.608 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.0] 2022-09-13T07:23:46.608 INFO:teuthology.orchestra.run.smithi036.stdout: key = AQD7LyBjcEPQJBAAu1hS4DHDfy/aNjTF8SNPYw== 2022-09-13T07:23:46.609 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.1] 2022-09-13T07:23:46.609 INFO:teuthology.orchestra.run.smithi036.stdout: key = AQD9LyBjaFVUMRAAsbApc0/oOzGtjXJbpviuIw== 2022-09-13T07:23:46.609 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.2] 2022-09-13T07:23:46.609 INFO:teuthology.orchestra.run.smithi036.stdout: key = AQAAMCBjhweaARAA2rGh8UCmaG6D6chyuYwLXg== 2022-09-13T07:23:46.610 INFO:teuthology.orchestra.run.smithi036.stdout:[client.0] 2022-09-13T07:23:46.610 INFO:teuthology.orchestra.run.smithi036.stdout: key = AQD5LyBj+YpaBhAAidfsVw0U1cq3KsqXYzGEag== 2022-09-13T07:23:46.610 INFO:teuthology.orchestra.run.smithi036.stdout:[client.1] 2022-09-13T07:23:46.610 INFO:teuthology.orchestra.run.smithi036.stdout: key = AQD5LyBjLVMgDBAAyWZJU54HPJQG5dRmqy6Exw== 2022-09-13T07:23:46.612 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-09-13T07:23:46.708 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-13T07:23:46.763 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-13T07:23:46.820 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-13T07:23:46.875 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-13T07:23:46.933 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-13T07:23:46.991 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-09-13T07:23:46.991 DEBUG:teuthology.orchestra.run.smithi036:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2022-09-13T07:23:47.019 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2022-09-13T07:23:47.119 DEBUG:teuthology.orchestra.run.smithi036:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2022-09-13T07:23:47.147 DEBUG:teuthology.orchestra.run.smithi036:> rm -- /home/ubuntu/cephtest/c1.monmap 2022-09-13T07:23:47.207 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2022-09-13T07:23:47.208 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2022-09-13T07:23:47.209 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2022-09-13T07:23:47.249 INFO:tasks.ceph.c1.mon.a:Started 2022-09-13T07:23:47.250 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2022-09-13T07:23:47.251 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2022-09-13T07:23:47.251 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2022-09-13T07:23:47.253 INFO:tasks.ceph.c1.mgr.x:Started 2022-09-13T07:23:47.254 INFO:tasks.ceph:Setting crush tunables to default 2022-09-13T07:23:47.254 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph --cluster c1 osd crush tunables default 2022-09-13T07:23:47.466 INFO:teuthology.orchestra.run.smithi036.stderr:adjusted tunables profile to default 2022-09-13T07:23:47.477 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2022-09-13T07:23:47.478 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:47.478 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2022-09-13T07:23:47.508 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:47.508 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2022-09-13T07:23:47.575 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:23:47.576 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2022-09-13T07:23:47.645 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph --cluster c1 osd new 1b15dcf5-6e6b-4e8a-8c6e-50e2d3bdce2b 0 2022-09-13T07:23:47.650 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:47.647+0000 7f836a673080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-13T07:23:47.816 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:47.813+0000 7f836a673080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-13T07:23:47.881 INFO:teuthology.orchestra.run.smithi036.stdout:0 2022-09-13T07:23:47.893 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph --cluster c1 osd new 1d74416f-f6f4-49ea-815e-b2cd7ebfd4bb 1 2022-09-13T07:23:48.133 INFO:teuthology.orchestra.run.smithi036.stdout:1 2022-09-13T07:23:48.145 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph --cluster c1 osd new b2c4cea1-1b7e-43ef-8535-eaa77e15ee3f 2 2022-09-13T07:23:48.398 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:48.395+0000 7f836a673080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-13T07:23:48.403 INFO:teuthology.orchestra.run.smithi036.stdout:2 2022-09-13T07:23:48.416 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2022-09-13T07:23:48.417 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2022-09-13T07:23:48.420 INFO:tasks.ceph.c1.osd.0:Started 2022-09-13T07:23:48.420 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2022-09-13T07:23:48.421 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2022-09-13T07:23:48.424 INFO:tasks.ceph.c1.osd.1:Started 2022-09-13T07:23:48.425 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2022-09-13T07:23:48.425 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2022-09-13T07:23:48.428 INFO:tasks.ceph.c1.osd.2:Started 2022-09-13T07:23:48.429 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-09-13T07:23:48.652 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:48.653 INFO:teuthology.orchestra.run.smithi036.stdout:{"epoch":5,"fsid":"5cb3badf-0576-4aaa-b80a-741bd397fa0b","created":"2022-09-13T07:23:47.366415+0000","modified":"2022-09-13T07:23:48.399710+0000","last_up_change":"0.000000","last_in_change":"2022-09-13T07:23:48.399710+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"1b15dcf5-6e6b-4e8a-8c6e-50e2d3bdce2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1d74416f-f6f4-49ea-815e-b2cd7ebfd4bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2c4cea1-1b7e-43ef-8535-eaa77e15ee3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-13T07:23:48.665 INFO:tasks.ceph.ceph_manager.c1:[] 2022-09-13T07:23:48.666 INFO:tasks.ceph:Waiting for OSDs to come up 2022-09-13T07:23:48.787 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:48.784+0000 7f836a673080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-13T07:23:48.889 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:48.886+0000 7f836a673080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-13T07:23:48.975 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2022-09-13T07:23:49.028 INFO:tasks.ceph.c1.osd.1.smithi036.stderr:2022-09-13T07:23:49.025+0000 7f1a60dc4440 -1 Falling back to public interface 2022-09-13T07:23:49.035 INFO:tasks.ceph.c1.osd.0.smithi036.stderr:2022-09-13T07:23:49.032+0000 7f65aa740440 -1 Falling back to public interface 2022-09-13T07:23:49.061 INFO:tasks.ceph.c1.osd.2.smithi036.stderr:2022-09-13T07:23:49.057+0000 7fddafbd3440 -1 Falling back to public interface 2022-09-13T07:23:49.110 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:49.103+0000 7f836a673080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-13T07:23:49.174 INFO:teuthology.misc.health.smithi036.stdout: 2022-09-13T07:23:49.175 INFO:teuthology.misc.health.smithi036.stdout:{"epoch":5,"fsid":"5cb3badf-0576-4aaa-b80a-741bd397fa0b","created":"2022-09-13T07:23:47.366415+0000","modified":"2022-09-13T07:23:48.399710+0000","last_up_change":"0.000000","last_in_change":"2022-09-13T07:23:48.399710+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"1b15dcf5-6e6b-4e8a-8c6e-50e2d3bdce2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1d74416f-f6f4-49ea-815e-b2cd7ebfd4bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2c4cea1-1b7e-43ef-8535-eaa77e15ee3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-13T07:23:49.186 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-09-13T07:23:50.089 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:50.086+0000 7f836a673080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-13T07:23:50.169 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:50.166+0000 7f836a673080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-13T07:23:50.260 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:50.257+0000 7f836a673080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-13T07:23:50.342 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:50.339+0000 7f836a673080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-13T07:23:50.643 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:50.640+0000 7f836a673080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-13T07:23:50.718 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:50.715+0000 7f836a673080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-13T07:23:50.793 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:50.789+0000 7f836a673080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-13T07:23:51.446 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:51.443+0000 7f836a673080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-13T07:23:51.591 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:51.587+0000 7f836a673080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-13T07:23:51.743 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:51.740+0000 7f836a673080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-13T07:23:51.746 INFO:tasks.ceph.c1.osd.0.smithi036.stderr:2022-09-13T07:23:51.743+0000 7f65aa740440 -1 osd.0 0 log_to_monitors true 2022-09-13T07:23:51.758 INFO:tasks.ceph.c1.osd.1.smithi036.stderr:2022-09-13T07:23:51.755+0000 7f1a60dc4440 -1 osd.1 0 log_to_monitors true 2022-09-13T07:23:51.789 INFO:tasks.ceph.c1.osd.2.smithi036.stderr:2022-09-13T07:23:51.785+0000 7fddafbd3440 -1 osd.2 0 log_to_monitors true 2022-09-13T07:23:51.824 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:51.820+0000 7f836a673080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-13T07:23:51.961 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:51.958+0000 7f836a673080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-13T07:23:52.056 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:52.053+0000 7f836a673080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-13T07:23:52.310 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:52.306+0000 7f836a673080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-13T07:23:52.591 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:52.587+0000 7f836a673080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-13T07:23:52.675 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:52.672+0000 7f836a673080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-13T07:23:52.877 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:52.874+0000 7f836a673080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-13T07:23:53.065 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:53.061+0000 7f836a673080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-13T07:23:53.947 INFO:tasks.ceph.c1.osd.0.smithi036.stderr:2022-09-13T07:23:53.943+0000 7f659e4ed700 -1 osd.0 0 waiting for initial osdmap 2022-09-13T07:23:53.958 INFO:tasks.ceph.c1.osd.0.smithi036.stderr:2022-09-13T07:23:53.955+0000 7f65968c8700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-13T07:23:54.006 INFO:tasks.ceph.c1.osd.1.smithi036.stderr:2022-09-13T07:23:54.003+0000 7f1a54b71700 -1 osd.1 0 waiting for initial osdmap 2022-09-13T07:23:54.094 INFO:tasks.ceph.c1.osd.1.smithi036.stderr:2022-09-13T07:23:54.091+0000 7f1a4cf4c700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-13T07:23:54.133 INFO:tasks.ceph.c1.mgr.x.smithi036.stderr:2022-09-13T07:23:54.130+0000 7f832942c700 -1 mgr.server handle_report got status from non-daemon mon.a 2022-09-13T07:23:54.413 INFO:tasks.ceph.c1.osd.2.smithi036.stderr:2022-09-13T07:23:54.409+0000 7fdda3980700 -1 osd.2 0 waiting for initial osdmap 2022-09-13T07:23:54.423 INFO:tasks.ceph.c1.osd.2.smithi036.stderr:2022-09-13T07:23:54.420+0000 7fdd9bd5b700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-13T07:23:55.490 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2022-09-13T07:23:55.817 INFO:teuthology.misc.health.smithi036.stdout: 2022-09-13T07:23:55.817 INFO:teuthology.misc.health.smithi036.stdout:{"epoch":8,"fsid":"5cb3badf-0576-4aaa-b80a-741bd397fa0b","created":"2022-09-13T07:23:47.366415+0000","modified":"2022-09-13T07:23:54.945776+0000","last_up_change":"2022-09-13T07:23:54.945776+0000","last_in_change":"2022-09-13T07:23:48.399710+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"1b15dcf5-6e6b-4e8a-8c6e-50e2d3bdce2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6808","nonce":101695},{"type":"v1","addr":"172.21.15.36:6809","nonce":101695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6810","nonce":101695},{"type":"v1","addr":"172.21.15.36:6811","nonce":101695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6814","nonce":101695},{"type":"v1","addr":"172.21.15.36:6815","nonce":101695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6812","nonce":101695},{"type":"v1","addr":"172.21.15.36:6813","nonce":101695}]},"public_addr":"172.21.15.36:6809/101695","cluster_addr":"172.21.15.36:6811/101695","heartbeat_back_addr":"172.21.15.36:6815/101695","heartbeat_front_addr":"172.21.15.36:6813/101695","state":["exists","up"]},{"osd":1,"uuid":"1d74416f-f6f4-49ea-815e-b2cd7ebfd4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6800","nonce":101694},{"type":"v1","addr":"172.21.15.36:6801","nonce":101694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6802","nonce":101694},{"type":"v1","addr":"172.21.15.36:6803","nonce":101694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6806","nonce":101694},{"type":"v1","addr":"172.21.15.36:6807","nonce":101694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6804","nonce":101694},{"type":"v1","addr":"172.21.15.36:6805","nonce":101694}]},"public_addr":"172.21.15.36:6801/101694","cluster_addr":"172.21.15.36:6803/101694","heartbeat_back_addr":"172.21.15.36:6807/101694","heartbeat_front_addr":"172.21.15.36:6805/101694","state":["exists","up"]},{"osd":2,"uuid":"b2c4cea1-1b7e-43ef-8535-eaa77e15ee3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6816","nonce":101724},{"type":"v1","addr":"172.21.15.36:6817","nonce":101724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6818","nonce":101724},{"type":"v1","addr":"172.21.15.36:6819","nonce":101724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6822","nonce":101724},{"type":"v1","addr":"172.21.15.36:6823","nonce":101724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6820","nonce":101724},{"type":"v1","addr":"172.21.15.36:6821","nonce":101724}]},"public_addr":"172.21.15.36:6817/101724","cluster_addr":"172.21.15.36:6819/101724","heartbeat_back_addr":"172.21.15.36:6823/101724","heartbeat_front_addr":"172.21.15.36:6821/101724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-13T07:23:55.830 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-09-13T07:23:55.830 INFO:tasks.ceph:Creating RBD pool 2022-09-13T07:23:55.830 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph --cluster c1 osd pool create rbd 8 2022-09-13T07:23:57.142 INFO:teuthology.orchestra.run.smithi036.stderr:pool 'rbd' created 2022-09-13T07:23:57.158 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2022-09-13T07:23:58.143 INFO:teuthology.orchestra.run.smithi036.stderr:enabled application 'rbd' on pool 'rbd' 2022-09-13T07:23:58.158 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2022-09-13T07:23:58.159 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2022-09-13T07:23:58.159 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2022-09-13T07:23:58.160 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2022-09-13T07:23:58.160 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-09-13T07:23:58.540 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:58.552 INFO:teuthology.orchestra.run.smithi036.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6824","nonce":101279},{"type":"v1","addr":"172.21.15.36:6825","nonce":101279}]},"active_addr":"172.21.15.36:6825/101279","active_change":"2022-09-13T07:23:53.076177+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.36:0","nonce":2898282658}]},{"addrvec":[{"type":"v2","addr":"172.21.15.36:0","nonce":2642993680}]},{"addrvec":[{"type":"v2","addr":"172.21.15.36:0","nonce":2854714622}]}]}} 2022-09-13T07:23:58.558 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2022-09-13T07:23:58.558 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2022-09-13T07:23:58.559 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-09-13T07:23:58.880 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:58.881 INFO:teuthology.orchestra.run.smithi036.stdout:{"epoch":11,"fsid":"5cb3badf-0576-4aaa-b80a-741bd397fa0b","created":"2022-09-13T07:23:47.366415+0000","modified":"2022-09-13T07:23:58.132959+0000","last_up_change":"2022-09-13T07:23:54.945776+0000","last_in_change":"2022-09-13T07:23:48.399710+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-13T07:23:55.185879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-09-13T07:23:56.179172+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"1b15dcf5-6e6b-4e8a-8c6e-50e2d3bdce2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6808","nonce":101695},{"type":"v1","addr":"172.21.15.36:6809","nonce":101695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6810","nonce":101695},{"type":"v1","addr":"172.21.15.36:6811","nonce":101695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6814","nonce":101695},{"type":"v1","addr":"172.21.15.36:6815","nonce":101695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6812","nonce":101695},{"type":"v1","addr":"172.21.15.36:6813","nonce":101695}]},"public_addr":"172.21.15.36:6809/101695","cluster_addr":"172.21.15.36:6811/101695","heartbeat_back_addr":"172.21.15.36:6815/101695","heartbeat_front_addr":"172.21.15.36:6813/101695","state":["exists","up"]},{"osd":1,"uuid":"1d74416f-f6f4-49ea-815e-b2cd7ebfd4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6800","nonce":101694},{"type":"v1","addr":"172.21.15.36:6801","nonce":101694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6802","nonce":101694},{"type":"v1","addr":"172.21.15.36:6803","nonce":101694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6806","nonce":101694},{"type":"v1","addr":"172.21.15.36:6807","nonce":101694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6804","nonce":101694},{"type":"v1","addr":"172.21.15.36:6805","nonce":101694}]},"public_addr":"172.21.15.36:6801/101694","cluster_addr":"172.21.15.36:6803/101694","heartbeat_back_addr":"172.21.15.36:6807/101694","heartbeat_front_addr":"172.21.15.36:6805/101694","state":["exists","up"]},{"osd":2,"uuid":"b2c4cea1-1b7e-43ef-8535-eaa77e15ee3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6816","nonce":101724},{"type":"v1","addr":"172.21.15.36:6817","nonce":101724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6818","nonce":101724},{"type":"v1","addr":"172.21.15.36:6819","nonce":101724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6822","nonce":101724},{"type":"v1","addr":"172.21.15.36:6823","nonce":101724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6820","nonce":101724},{"type":"v1","addr":"172.21.15.36:6821","nonce":101724}]},"public_addr":"172.21.15.36:6817/101724","cluster_addr":"172.21.15.36:6819/101724","heartbeat_back_addr":"172.21.15.36:6823/101724","heartbeat_front_addr":"172.21.15.36:6821/101724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-13T07:23:52.778878+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-13T07:23:52.740972+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-13T07:23:52.769124+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-13T07:23:58.894 INFO:tasks.ceph.ceph_manager.c1:all up! 2022-09-13T07:23:58.894 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-09-13T07:23:59.218 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:23:59.218 INFO:teuthology.orchestra.run.smithi036.stdout:{"epoch":12,"fsid":"5cb3badf-0576-4aaa-b80a-741bd397fa0b","created":"2022-09-13T07:23:47.366415+0000","modified":"2022-09-13T07:23:59.136803+0000","last_up_change":"2022-09-13T07:23:54.945776+0000","last_in_change":"2022-09-13T07:23:48.399710+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-13T07:23:55.185879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-09-13T07:23:56.179172+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"1b15dcf5-6e6b-4e8a-8c6e-50e2d3bdce2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6808","nonce":101695},{"type":"v1","addr":"172.21.15.36:6809","nonce":101695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6810","nonce":101695},{"type":"v1","addr":"172.21.15.36:6811","nonce":101695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6814","nonce":101695},{"type":"v1","addr":"172.21.15.36:6815","nonce":101695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6812","nonce":101695},{"type":"v1","addr":"172.21.15.36:6813","nonce":101695}]},"public_addr":"172.21.15.36:6809/101695","cluster_addr":"172.21.15.36:6811/101695","heartbeat_back_addr":"172.21.15.36:6815/101695","heartbeat_front_addr":"172.21.15.36:6813/101695","state":["exists","up"]},{"osd":1,"uuid":"1d74416f-f6f4-49ea-815e-b2cd7ebfd4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6800","nonce":101694},{"type":"v1","addr":"172.21.15.36:6801","nonce":101694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6802","nonce":101694},{"type":"v1","addr":"172.21.15.36:6803","nonce":101694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6806","nonce":101694},{"type":"v1","addr":"172.21.15.36:6807","nonce":101694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6804","nonce":101694},{"type":"v1","addr":"172.21.15.36:6805","nonce":101694}]},"public_addr":"172.21.15.36:6801/101694","cluster_addr":"172.21.15.36:6803/101694","heartbeat_back_addr":"172.21.15.36:6807/101694","heartbeat_front_addr":"172.21.15.36:6805/101694","state":["exists","up"]},{"osd":2,"uuid":"b2c4cea1-1b7e-43ef-8535-eaa77e15ee3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6816","nonce":101724},{"type":"v1","addr":"172.21.15.36:6817","nonce":101724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6818","nonce":101724},{"type":"v1","addr":"172.21.15.36:6819","nonce":101724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6822","nonce":101724},{"type":"v1","addr":"172.21.15.36:6823","nonce":101724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6820","nonce":101724},{"type":"v1","addr":"172.21.15.36:6821","nonce":101724}]},"public_addr":"172.21.15.36:6817/101724","cluster_addr":"172.21.15.36:6819/101724","heartbeat_back_addr":"172.21.15.36:6823/101724","heartbeat_front_addr":"172.21.15.36:6821/101724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-13T07:23:52.778878+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-13T07:23:52.740972+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-13T07:23:52.769124+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-13T07:23:59.232 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2022-09-13T07:23:59.233 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2022-09-13T07:23:59.233 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2022-09-13T07:23:59.401 INFO:teuthology.orchestra.run.smithi036.stdout:34359738370 2022-09-13T07:23:59.402 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-09-13T07:23:59.406 INFO:teuthology.orchestra.run.smithi036.stdout:34359738371 2022-09-13T07:23:59.406 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-09-13T07:23:59.407 INFO:teuthology.orchestra.run.smithi036.stdout:34359738370 2022-09-13T07:23:59.408 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2022-09-13T07:23:59.733 INFO:teuthology.orchestra.run.smithi036.stdout:34359738369 2022-09-13T07:23:59.747 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.2 2022-09-13T07:23:59.785 INFO:teuthology.orchestra.run.smithi036.stdout:34359738370 2022-09-13T07:23:59.789 INFO:teuthology.orchestra.run.smithi036.stdout:34359738369 2022-09-13T07:23:59.800 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2022-09-13T07:23:59.804 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.1 2022-09-13T07:24:00.749 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-09-13T07:24:00.801 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-09-13T07:24:00.804 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2022-09-13T07:24:01.095 INFO:teuthology.orchestra.run.smithi036.stdout:34359738369 2022-09-13T07:24:01.110 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.2 2022-09-13T07:24:01.144 INFO:teuthology.orchestra.run.smithi036.stdout:34359738370 2022-09-13T07:24:01.159 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2022-09-13T07:24:01.188 INFO:teuthology.orchestra.run.smithi036.stdout:34359738371 2022-09-13T07:24:01.202 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738371 for osd.1 2022-09-13T07:24:01.203 DEBUG:teuthology.parallel:result is None 2022-09-13T07:24:02.111 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-09-13T07:24:02.162 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-09-13T07:24:02.453 INFO:teuthology.orchestra.run.smithi036.stdout:34359738371 2022-09-13T07:24:02.467 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738371 for osd.2 2022-09-13T07:24:02.468 DEBUG:teuthology.parallel:result is None 2022-09-13T07:24:02.494 INFO:teuthology.orchestra.run.smithi036.stdout:34359738372 2022-09-13T07:24:02.509 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738372 for osd.0 2022-09-13T07:24:02.510 DEBUG:teuthology.parallel:result is None 2022-09-13T07:24:02.510 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2022-09-13T07:24:02.511 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2022-09-13T07:24:02.866 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:02.868 INFO:teuthology.orchestra.run.smithi036.stderr:dumped all 2022-09-13T07:24:02.880 INFO:teuthology.orchestra.run.smithi036.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2022-09-13T07:24:01.128084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25404,"kb_used_data":1452,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547396,"statfs":{"total":322122547200,"available":322096533504,"internally_reserved":0,"allocated":1486848,"data_stored":1260256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.386299+0000","last_change":"2022-09-13T07:23:58.143634+0000","last_active":"2022-09-13T07:23:59.386299+0000","last_peered":"2022-09-13T07:23:59.386299+0000","last_clean":"2022-09-13T07:23:59.386299+0000","last_became_active":"2022-09-13T07:23:58.142957+0000","last_became_peered":"2022-09-13T07:23:58.142957+0000","last_unstale":"2022-09-13T07:23:59.386299+0000","last_undegraded":"2022-09-13T07:23:59.386299+0000","last_fullsized":"2022-09-13T07:23:59.386299+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T19:14:14.013139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.386280+0000","last_change":"2022-09-13T07:23:58.144211+0000","last_active":"2022-09-13T07:23:59.386280+0000","last_peered":"2022-09-13T07:23:59.386280+0000","last_clean":"2022-09-13T07:23:59.386280+0000","last_became_active":"2022-09-13T07:23:58.143451+0000","last_became_peered":"2022-09-13T07:23:58.143451+0000","last_unstale":"2022-09-13T07:23:59.386280+0000","last_undegraded":"2022-09-13T07:23:59.386280+0000","last_fullsized":"2022-09-13T07:23:59.386280+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T10:49:10.621688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.386263+0000","last_change":"2022-09-13T07:23:58.144072+0000","last_active":"2022-09-13T07:23:59.386263+0000","last_peered":"2022-09-13T07:23:59.386263+0000","last_clean":"2022-09-13T07:23:59.386263+0000","last_became_active":"2022-09-13T07:23:58.143751+0000","last_became_peered":"2022-09-13T07:23:58.143751+0000","last_unstale":"2022-09-13T07:23:59.386263+0000","last_undegraded":"2022-09-13T07:23:59.386263+0000","last_fullsized":"2022-09-13T07:23:59.386263+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T13:25:53.305474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.140147+0000","last_change":"2022-09-13T07:23:58.142849+0000","last_active":"2022-09-13T07:23:59.140147+0000","last_peered":"2022-09-13T07:23:59.140147+0000","last_clean":"2022-09-13T07:23:59.140147+0000","last_became_active":"2022-09-13T07:23:58.142580+0000","last_became_peered":"2022-09-13T07:23:58.142580+0000","last_unstale":"2022-09-13T07:23:59.140147+0000","last_undegraded":"2022-09-13T07:23:59.140147+0000","last_fullsized":"2022-09-13T07:23:59.140147+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T13:28:47.095559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":138,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.386178+0000","last_change":"2022-09-13T07:23:57.141694+0000","last_active":"2022-09-13T07:23:59.386178+0000","last_peered":"2022-09-13T07:23:59.386178+0000","last_clean":"2022-09-13T07:23:59.386178+0000","last_became_active":"2022-09-13T07:23:57.140672+0000","last_became_peered":"2022-09-13T07:23:57.140672+0000","last_unstale":"2022-09-13T07:23:59.386178+0000","last_undegraded":"2022-09-13T07:23:59.386178+0000","last_fullsized":"2022-09-13T07:23:59.386178+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:56.131519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:56.131519+0000","last_clean_scrub_stamp":"2022-09-13T07:23:56.131519+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T15:40:57.550488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.386212+0000","last_change":"2022-09-13T07:23:58.140701+0000","last_active":"2022-09-13T07:23:59.386212+0000","last_peered":"2022-09-13T07:23:59.386212+0000","last_clean":"2022-09-13T07:23:59.386212+0000","last_became_active":"2022-09-13T07:23:58.140409+0000","last_became_peered":"2022-09-13T07:23:58.140409+0000","last_unstale":"2022-09-13T07:23:59.386212+0000","last_undegraded":"2022-09-13T07:23:59.386212+0000","last_fullsized":"2022-09-13T07:23:59.386212+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T17:15:39.343579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.140147+0000","last_change":"2022-09-13T07:23:58.142883+0000","last_active":"2022-09-13T07:23:59.140147+0000","last_peered":"2022-09-13T07:23:59.140147+0000","last_clean":"2022-09-13T07:23:59.140147+0000","last_became_active":"2022-09-13T07:23:58.142603+0000","last_became_peered":"2022-09-13T07:23:58.142603+0000","last_unstale":"2022-09-13T07:23:59.140147+0000","last_undegraded":"2022-09-13T07:23:59.140147+0000","last_fullsized":"2022-09-13T07:23:59.140147+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T19:21:01.276691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:23:58.144968+0000","last_change":"2022-09-13T07:23:58.143109+0000","last_active":"2022-09-13T07:23:58.144968+0000","last_peered":"2022-09-13T07:23:58.144968+0000","last_clean":"2022-09-13T07:23:58.144968+0000","last_became_active":"2022-09-13T07:23:58.142821+0000","last_became_peered":"2022-09-13T07:23:58.142821+0000","last_unstale":"2022-09-13T07:23:58.144968+0000","last_undegraded":"2022-09-13T07:23:58.144968+0000","last_fullsized":"2022-09-13T07:23:58.144968+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T11:21:16.152504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.386232+0000","last_change":"2022-09-13T07:23:58.144351+0000","last_active":"2022-09-13T07:23:59.386232+0000","last_peered":"2022-09-13T07:23:59.386232+0000","last_clean":"2022-09-13T07:23:59.386232+0000","last_became_active":"2022-09-13T07:23:58.142898+0000","last_became_peered":"2022-09-13T07:23:58.142898+0000","last_unstale":"2022-09-13T07:23:59.386232+0000","last_undegraded":"2022-09-13T07:23:59.386232+0000","last_fullsized":"2022-09-13T07:23:59.386232+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T15:30:49.820680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7664,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849936,"statfs":{"total":107374182400,"available":107366334464,"internally_reserved":0,"allocated":106496,"data_stored":29071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":9120,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848480,"statfs":{"total":107374182400,"available":107364843520,"internally_reserved":0,"allocated":688128,"data_stored":615212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8620,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848980,"statfs":{"total":107374182400,"available":107365355520,"internally_reserved":0,"allocated":692224,"data_stored":615973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-13T07:24:02.881 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2022-09-13T07:24:03.196 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:03.198 INFO:teuthology.orchestra.run.smithi036.stderr:dumped all 2022-09-13T07:24:03.210 INFO:teuthology.orchestra.run.smithi036.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2022-09-13T07:24:03.128331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25404,"kb_used_data":1452,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547396,"statfs":{"total":322122547200,"available":322096533504,"internally_reserved":0,"allocated":1486848,"data_stored":1260256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000247"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.386299+0000","last_change":"2022-09-13T07:23:58.143634+0000","last_active":"2022-09-13T07:23:59.386299+0000","last_peered":"2022-09-13T07:23:59.386299+0000","last_clean":"2022-09-13T07:23:59.386299+0000","last_became_active":"2022-09-13T07:23:58.142957+0000","last_became_peered":"2022-09-13T07:23:58.142957+0000","last_unstale":"2022-09-13T07:23:59.386299+0000","last_undegraded":"2022-09-13T07:23:59.386299+0000","last_fullsized":"2022-09-13T07:23:59.386299+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T19:14:14.013139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.386280+0000","last_change":"2022-09-13T07:23:58.144211+0000","last_active":"2022-09-13T07:23:59.386280+0000","last_peered":"2022-09-13T07:23:59.386280+0000","last_clean":"2022-09-13T07:23:59.386280+0000","last_became_active":"2022-09-13T07:23:58.143451+0000","last_became_peered":"2022-09-13T07:23:58.143451+0000","last_unstale":"2022-09-13T07:23:59.386280+0000","last_undegraded":"2022-09-13T07:23:59.386280+0000","last_fullsized":"2022-09-13T07:23:59.386280+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T10:49:10.621688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.386263+0000","last_change":"2022-09-13T07:23:58.144072+0000","last_active":"2022-09-13T07:23:59.386263+0000","last_peered":"2022-09-13T07:23:59.386263+0000","last_clean":"2022-09-13T07:23:59.386263+0000","last_became_active":"2022-09-13T07:23:58.143751+0000","last_became_peered":"2022-09-13T07:23:58.143751+0000","last_unstale":"2022-09-13T07:23:59.386263+0000","last_undegraded":"2022-09-13T07:23:59.386263+0000","last_fullsized":"2022-09-13T07:23:59.386263+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T13:25:53.305474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.140147+0000","last_change":"2022-09-13T07:23:58.142849+0000","last_active":"2022-09-13T07:23:59.140147+0000","last_peered":"2022-09-13T07:23:59.140147+0000","last_clean":"2022-09-13T07:23:59.140147+0000","last_became_active":"2022-09-13T07:23:58.142580+0000","last_became_peered":"2022-09-13T07:23:58.142580+0000","last_unstale":"2022-09-13T07:23:59.140147+0000","last_undegraded":"2022-09-13T07:23:59.140147+0000","last_fullsized":"2022-09-13T07:23:59.140147+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T13:28:47.095559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":138,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.386178+0000","last_change":"2022-09-13T07:23:57.141694+0000","last_active":"2022-09-13T07:23:59.386178+0000","last_peered":"2022-09-13T07:23:59.386178+0000","last_clean":"2022-09-13T07:23:59.386178+0000","last_became_active":"2022-09-13T07:23:57.140672+0000","last_became_peered":"2022-09-13T07:23:57.140672+0000","last_unstale":"2022-09-13T07:23:59.386178+0000","last_undegraded":"2022-09-13T07:23:59.386178+0000","last_fullsized":"2022-09-13T07:23:59.386178+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:56.131519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:56.131519+0000","last_clean_scrub_stamp":"2022-09-13T07:23:56.131519+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T15:40:57.550488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.386212+0000","last_change":"2022-09-13T07:23:58.140701+0000","last_active":"2022-09-13T07:23:59.386212+0000","last_peered":"2022-09-13T07:23:59.386212+0000","last_clean":"2022-09-13T07:23:59.386212+0000","last_became_active":"2022-09-13T07:23:58.140409+0000","last_became_peered":"2022-09-13T07:23:58.140409+0000","last_unstale":"2022-09-13T07:23:59.386212+0000","last_undegraded":"2022-09-13T07:23:59.386212+0000","last_fullsized":"2022-09-13T07:23:59.386212+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T17:15:39.343579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.140147+0000","last_change":"2022-09-13T07:23:58.142883+0000","last_active":"2022-09-13T07:23:59.140147+0000","last_peered":"2022-09-13T07:23:59.140147+0000","last_clean":"2022-09-13T07:23:59.140147+0000","last_became_active":"2022-09-13T07:23:58.142603+0000","last_became_peered":"2022-09-13T07:23:58.142603+0000","last_unstale":"2022-09-13T07:23:59.140147+0000","last_undegraded":"2022-09-13T07:23:59.140147+0000","last_fullsized":"2022-09-13T07:23:59.140147+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T19:21:01.276691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:23:58.144968+0000","last_change":"2022-09-13T07:23:58.143109+0000","last_active":"2022-09-13T07:23:58.144968+0000","last_peered":"2022-09-13T07:23:58.144968+0000","last_clean":"2022-09-13T07:23:58.144968+0000","last_became_active":"2022-09-13T07:23:58.142821+0000","last_became_peered":"2022-09-13T07:23:58.142821+0000","last_unstale":"2022-09-13T07:23:58.144968+0000","last_undegraded":"2022-09-13T07:23:58.144968+0000","last_fullsized":"2022-09-13T07:23:58.144968+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T11:21:16.152504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:23:59.386232+0000","last_change":"2022-09-13T07:23:58.144351+0000","last_active":"2022-09-13T07:23:59.386232+0000","last_peered":"2022-09-13T07:23:59.386232+0000","last_clean":"2022-09-13T07:23:59.386232+0000","last_became_active":"2022-09-13T07:23:58.142898+0000","last_became_peered":"2022-09-13T07:23:58.142898+0000","last_unstale":"2022-09-13T07:23:59.386232+0000","last_undegraded":"2022-09-13T07:23:59.386232+0000","last_fullsized":"2022-09-13T07:23:59.386232+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:23:57.132145+0000","last_clean_scrub_stamp":"2022-09-13T07:23:57.132145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T15:30:49.820680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7664,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849936,"statfs":{"total":107374182400,"available":107366334464,"internally_reserved":0,"allocated":106496,"data_stored":29071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":9120,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848480,"statfs":{"total":107374182400,"available":107364843520,"internally_reserved":0,"allocated":688128,"data_stored":615212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8620,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848980,"statfs":{"total":107374182400,"available":107365355520,"internally_reserved":0,"allocated":692224,"data_stored":615973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-13T07:24:03.211 INFO:tasks.ceph.ceph_manager.c1:clean! 2022-09-13T07:24:03.211 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2022-09-13T07:24:03.212 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2022-09-13T07:24:03.212 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2022-09-13T07:24:03.576 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:03.577 INFO:teuthology.orchestra.run.smithi036.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-09-13T07:24:03.591 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2022-09-13T07:24:03.592 INFO:teuthology.run_tasks:Running task ceph... 2022-09-13T07:24:03.603 INFO:tasks.ceph:Creating ceph cluster c2... 2022-09-13T07:24:03.603 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-09-13T07:24:03.604 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuvalif-2022-09-13_07:01:40-rgw:multisite-wip-yuval-fix-56572-distro-default-smithi/7030425', 'branch': 'wip-yuval-fix-56572', 'description': 'rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7030425', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuvalif-2022-09-13_07:01:40-rgw:multisite-wip-yuval-fix-56572-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuval-fix-56572'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '840d05418c459144c92fb2a90d74a2ded29610da', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '840d05418c459144c92fb2a90d74a2ded29610da'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuval-fix-56572', 'sha1': '840d05418c459144c92fb2a90d74a2ded29610da'}}, 'owner': 'scheduled_yuvalif@teuthology', 'priority': 50, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '840d05418c459144c92fb2a90d74a2ded29610da', 'sleep_before_teardown': 0, 'suite': 'rgw:multisite', 'suite_branch': 'wip-yuval-fix-56572', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '840d05418c459144c92fb2a90d74a2ded29610da', 'targets': {'smithi036.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwZ10DhqfIPFcUwNSkeR9qLDxnG1HoxeKhmdLj9JcAN5J/ZJ7gkiU4iMSbViuINM2NTvcD2/EL0PGkYz+BhowcJPciculEA/wU67kHrJUdjcxBug8sEgbDHF6t7iR47g0ksrysKN+YpSUseVtgCSrHRlawbpy2GHby45dGTVM8J6XB9KpPP+733LuNmL5/ZsSCi1zu3D/WkIqHeTHWXBBkUlJB+hjOiWhXS20e9hlBWTxv13p/IQ0fK+byN48zP5uxcHlFqapgOfXbI3SBhXlBJgRke/pcipC7+UyukyvFgi0sIsHyoFTgl31Ihw1qZ7Vfiok+f04z5Z01pbbrdvuh56cjrIUxZKunLxoSTnZhDIWNeTwb3Wx5VxyOkmsucjcXgUOXRI8DpIEYwBPpiVI6QfJpQ8pLMYerFKkvxqYibT7ypK66xRruGMRK8IDy5RAM7z3TRrF8ICNPVUiTxPbm5lVhI3zD35JoLewYboMo6mLRe9MBYo2ploAK0cGynf0=', 'smithi083.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKhIgPHWHpUJcIGujf2yUi90hlD6vf5UNtX5WGaN0K+n6B8OUcfmnejuPWM21jMK7T/xfUT+nYNdWaHs/me21DI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '840d05418c459144c92fb2a90d74a2ded29610da', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '840d05418c459144c92fb2a90d74a2ded29610da', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'main', 'teuthology_sha1': 'b58f2c18636eb10faa77ed3614abd00cb85dfc2c', 'timestamp': '2022-09-13_07:01:40', 'tube': 'smithi', 'user': 'yuvalif', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.20382'} 2022-09-13T07:24:03.605 DEBUG:teuthology.orchestra.run.smithi036:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2022-09-13T07:24:03.607 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2022-09-13T07:24:03.627 DEBUG:teuthology.orchestra.run.smithi036:> sudo install -d -m0777 -- /var/run/ceph 2022-09-13T07:24:03.666 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/run/ceph 2022-09-13T07:24:03.696 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:24:03.696 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2022-09-13T07:24:03.755 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-09-13T07:24:03.756 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-09-13T07:24:03.815 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-09-13T07:24:03.815 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-13T07:24:03.816 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 360409 Links: 1 2022-09-13T07:24:03.816 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-13T07:24:03.817 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2022-09-13T07:24:03.818 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2022-09-13 07:23:27.490379115 +0000 2022-09-13T07:24:03.818 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2022-09-13 07:20:02.137302986 +0000 2022-09-13T07:24:03.818 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2022-09-13 07:20:02.137302986 +0000 2022-09-13T07:24:03.818 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2022-09-13T07:24:03.824 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2022-09-13T07:24:03.824 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2022-09-13T07:24:03.824 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000260117 s, 2.0 MB/s 2022-09-13T07:24:03.827 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-09-13T07:24:03.885 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-09-13T07:24:03.885 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-13T07:24:03.886 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 361894 Links: 1 2022-09-13T07:24:03.886 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-13T07:24:03.887 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2022-09-13T07:24:03.887 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2022-09-13 07:23:27.490379115 +0000 2022-09-13T07:24:03.887 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2022-09-13 07:20:02.876284860 +0000 2022-09-13T07:24:03.888 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2022-09-13 07:20:02.876284860 +0000 2022-09-13T07:24:03.888 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2022-09-13T07:24:03.894 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2022-09-13T07:24:03.894 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2022-09-13T07:24:03.894 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000232626 s, 2.2 MB/s 2022-09-13T07:24:03.896 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-09-13T07:24:03.954 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-09-13T07:24:03.955 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-13T07:24:03.955 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 366772 Links: 1 2022-09-13T07:24:03.955 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-13T07:24:03.956 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2022-09-13T07:24:03.956 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2022-09-13 07:23:27.490379115 +0000 2022-09-13T07:24:03.956 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2022-09-13 07:20:03.413271688 +0000 2022-09-13T07:24:03.956 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2022-09-13 07:20:03.413271688 +0000 2022-09-13T07:24:03.957 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2022-09-13T07:24:03.964 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2022-09-13T07:24:03.964 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2022-09-13T07:24:03.964 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000215663 s, 2.4 MB/s 2022-09-13T07:24:03.966 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-09-13T07:24:04.030 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-09-13T07:24:04.037 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-13T07:24:04.037 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 365777 Links: 1 2022-09-13T07:24:04.038 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-13T07:24:04.039 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2022-09-13T07:24:04.039 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2022-09-13 07:23:27.490379115 +0000 2022-09-13T07:24:04.040 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2022-09-13 07:20:03.955258394 +0000 2022-09-13T07:24:04.040 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2022-09-13 07:20:03.955258394 +0000 2022-09-13T07:24:04.040 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2022-09-13T07:24:04.043 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2022-09-13T07:24:04.043 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2022-09-13T07:24:04.044 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000225743 s, 2.3 MB/s 2022-09-13T07:24:04.045 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-13T07:24:04.046 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-13T07:24:04.047 INFO:tasks.ceph:Generating config... 2022-09-13T07:24:04.050 INFO:tasks.ceph:[client] debug rgw = 20 2022-09-13T07:24:04.051 INFO:tasks.ceph:[client] debug rgw sync = 20 2022-09-13T07:24:04.052 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2022-09-13T07:24:04.053 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2022-09-13T07:24:04.053 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-13T07:24:04.054 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2022-09-13T07:24:04.054 INFO:tasks.ceph:[client] rgw data log num shards = 4 2022-09-13T07:24:04.055 INFO:tasks.ceph:[client] rgw md log max shards = 4 2022-09-13T07:24:04.057 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2022-09-13T07:24:04.058 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2022-09-13T07:24:04.058 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2022-09-13T07:24:04.059 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2022-09-13T07:24:04.060 INFO:tasks.ceph:[client] setgroup = ceph 2022-09-13T07:24:04.060 INFO:tasks.ceph:[client] setuser = ceph 2022-09-13T07:24:04.061 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2022-09-13T07:24:04.061 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-09-13T07:24:04.062 INFO:tasks.ceph:[mgr] debug ms = 1 2022-09-13T07:24:04.063 INFO:tasks.ceph:[mon] debug mon = 20 2022-09-13T07:24:04.063 INFO:tasks.ceph:[mon] debug ms = 1 2022-09-13T07:24:04.064 INFO:tasks.ceph:[mon] debug paxos = 20 2022-09-13T07:24:04.065 INFO:tasks.ceph:[osd] debug ms = 1 2022-09-13T07:24:04.066 INFO:tasks.ceph:[osd] debug osd = 20 2022-09-13T07:24:04.066 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2022-09-13T07:24:04.067 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2022-09-13T07:24:04.067 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2022-09-13T07:24:04.068 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2022-09-13T07:24:04.069 INFO:tasks.ceph:Setting up c2.mon.a... 2022-09-13T07:24:04.069 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2022-09-13T07:24:04.135 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/c2.keyring 2022-09-13T07:24:04.138 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2022-09-13T07:24:04.240 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-09-13T07:24:04.322 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.83')] 2022-09-13T07:24:04.323 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.83'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2022-09-13T07:24:04.325 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:24:04.326 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-09-13T07:24:04.385 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.83 --print /home/ubuntu/cephtest/c2.monmap 2022-09-13T07:24:04.469 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:24:04.470 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:24:04.472 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2022-09-13T07:24:04.472 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: generated fsid 236e0d75-a3cd-49bc-af63-ffaa2d54d993 2022-09-13T07:24:04.473 INFO:teuthology.orchestra.run.smithi083.stdout:setting min_mon_release = octopus 2022-09-13T07:24:04.473 INFO:teuthology.orchestra.run.smithi083.stdout:epoch 0 2022-09-13T07:24:04.474 INFO:teuthology.orchestra.run.smithi083.stdout:fsid 236e0d75-a3cd-49bc-af63-ffaa2d54d993 2022-09-13T07:24:04.475 INFO:teuthology.orchestra.run.smithi083.stdout:last_changed 2022-09-13T07:24:04.467760+0000 2022-09-13T07:24:04.475 INFO:teuthology.orchestra.run.smithi083.stdout:created 2022-09-13T07:24:04.467760+0000 2022-09-13T07:24:04.476 INFO:teuthology.orchestra.run.smithi083.stdout:min_mon_release 15 (octopus) 2022-09-13T07:24:04.476 INFO:teuthology.orchestra.run.smithi083.stdout:election_strategy: 1 2022-09-13T07:24:04.477 INFO:teuthology.orchestra.run.smithi083.stdout:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2022-09-13T07:24:04.477 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2022-09-13T07:24:04.480 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-09-13T07:24:04.555 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 236e0d75-a3cd-49bc-af63-ffaa2d54d993... 2022-09-13T07:24:04.556 DEBUG:teuthology.orchestra.run.smithi036:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2022-09-13T07:24:04.564 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2022-09-13T07:24:04.615 INFO:teuthology.orchestra.run.smithi036.stdout:[global] 2022-09-13T07:24:04.616 INFO:teuthology.orchestra.run.smithi036.stdout: chdir = "" 2022-09-13T07:24:04.617 INFO:teuthology.orchestra.run.smithi036.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-13T07:24:04.622 INFO:teuthology.orchestra.run.smithi036.stdout: auth supported = cephx 2022-09-13T07:24:04.622 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.623 INFO:teuthology.orchestra.run.smithi036.stdout: filestore xattr use omap = true 2022-09-13T07:24:04.627 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.630 INFO:teuthology.orchestra.run.smithi036.stdout: mon clock drift allowed = 1.000 2022-09-13T07:24:04.630 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.630 INFO:teuthology.orchestra.run.smithi036.stdout: osd crush chooseleaf type = 0 2022-09-13T07:24:04.630 INFO:teuthology.orchestra.run.smithi036.stdout: auth debug = true 2022-09-13T07:24:04.631 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.631 INFO:teuthology.orchestra.run.smithi036.stdout: ms die on old message = true 2022-09-13T07:24:04.631 INFO:teuthology.orchestra.run.smithi036.stdout: ms die on bug = true 2022-09-13T07:24:04.631 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.632 INFO:teuthology.orchestra.run.smithi036.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-13T07:24:04.632 INFO:teuthology.orchestra.run.smithi036.stdout: mon pg warn max object skew = 0 2022-09-13T07:24:04.632 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.632 INFO:teuthology.orchestra.run.smithi036.stdout: # disable pg_autoscaler by default for new pools 2022-09-13T07:24:04.632 INFO:teuthology.orchestra.run.smithi036.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-13T07:24:04.633 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.633 INFO:teuthology.orchestra.run.smithi036.stdout: osd pool default size = 2 2022-09-13T07:24:04.633 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.633 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd allow primary affinity = true 2022-09-13T07:24:04.634 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd allow pg remap = true 2022-09-13T07:24:04.634 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on legacy crush tunables = false 2022-09-13T07:24:04.634 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on crush straw calc version zero = false 2022-09-13T07:24:04.634 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on no sortbitwise = false 2022-09-13T07:24:04.634 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on osd down out interval zero = false 2022-09-13T07:24:04.635 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on too few osds = false 2022-09-13T07:24:04.635 INFO:teuthology.orchestra.run.smithi036.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-13T07:24:04.635 INFO:teuthology.orchestra.run.smithi036.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-13T07:24:04.635 INFO:teuthology.orchestra.run.smithi036.stdout: mon_allow_pool_size_one = true 2022-09-13T07:24:04.636 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.636 INFO:teuthology.orchestra.run.smithi036.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-13T07:24:04.636 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.636 INFO:teuthology.orchestra.run.smithi036.stdout: osd default data pool replay window = 5 2022-09-13T07:24:04.636 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.637 INFO:teuthology.orchestra.run.smithi036.stdout: mon allow pool delete = true 2022-09-13T07:24:04.637 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.637 INFO:teuthology.orchestra.run.smithi036.stdout: mon cluster log file level = debug 2022-09-13T07:24:04.637 INFO:teuthology.orchestra.run.smithi036.stdout: debug asserts on shutdown = true 2022-09-13T07:24:04.637 INFO:teuthology.orchestra.run.smithi036.stdout: mon health detail to clog = false 2022-09-13T07:24:04.638 INFO:teuthology.orchestra.run.smithi036.stdout: mon host = 172.21.15.83 2022-09-13T07:24:04.638 INFO:teuthology.orchestra.run.smithi036.stdout: fsid = 236e0d75-a3cd-49bc-af63-ffaa2d54d993 2022-09-13T07:24:04.638 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.638 INFO:teuthology.orchestra.run.smithi036.stdout:[osd] 2022-09-13T07:24:04.639 INFO:teuthology.orchestra.run.smithi036.stdout: osd journal size = 100 2022-09-13T07:24:04.639 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.639 INFO:teuthology.orchestra.run.smithi036.stdout: osd scrub load threshold = 5.0 2022-09-13T07:24:04.639 INFO:teuthology.orchestra.run.smithi036.stdout: osd scrub max interval = 600 2022-09-13T07:24:04.639 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.640 INFO:teuthology.orchestra.run.smithi036.stdout: osd recover clone overlap = true 2022-09-13T07:24:04.640 INFO:teuthology.orchestra.run.smithi036.stdout: osd recovery max chunk = 1048576 2022-09-13T07:24:04.640 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.640 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug shutdown = true 2022-09-13T07:24:04.641 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug op order = true 2022-09-13T07:24:04.641 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug verify stray on activate = true 2022-09-13T07:24:04.641 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.641 INFO:teuthology.orchestra.run.smithi036.stdout: osd open classes on start = true 2022-09-13T07:24:04.641 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug pg log writeout = true 2022-09-13T07:24:04.642 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.642 INFO:teuthology.orchestra.run.smithi036.stdout: osd deep scrub update digest min age = 30 2022-09-13T07:24:04.642 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.642 INFO:teuthology.orchestra.run.smithi036.stdout: osd map max advance = 10 2022-09-13T07:24:04.642 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.643 INFO:teuthology.orchestra.run.smithi036.stdout: journal zero on create = true 2022-09-13T07:24:04.643 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.643 INFO:teuthology.orchestra.run.smithi036.stdout: filestore ondisk finisher threads = 3 2022-09-13T07:24:04.643 INFO:teuthology.orchestra.run.smithi036.stdout: filestore apply finisher threads = 3 2022-09-13T07:24:04.643 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.644 INFO:teuthology.orchestra.run.smithi036.stdout: bdev debug aio = true 2022-09-13T07:24:04.644 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug misdirected ops = true 2022-09-13T07:24:04.644 INFO:teuthology.orchestra.run.smithi036.stdout: debug ms = 1 2022-09-13T07:24:04.644 INFO:teuthology.orchestra.run.smithi036.stdout: debug osd = 20 2022-09-13T07:24:04.645 INFO:teuthology.orchestra.run.smithi036.stdout: rocksdb delete range threshold = 0 2022-09-13T07:24:04.645 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.645 INFO:teuthology.orchestra.run.smithi036.stdout:[mgr] 2022-09-13T07:24:04.645 INFO:teuthology.orchestra.run.smithi036.stdout: debug ms = 1 2022-09-13T07:24:04.646 INFO:teuthology.orchestra.run.smithi036.stdout: debug mgr = 20 2022-09-13T07:24:04.646 INFO:teuthology.orchestra.run.smithi036.stdout: debug mon = 20 2022-09-13T07:24:04.646 INFO:teuthology.orchestra.run.smithi036.stdout: debug auth = 20 2022-09-13T07:24:04.646 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min pgs per osd = 4 2022-09-13T07:24:04.646 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min bytes per osd = 10 2022-09-13T07:24:04.647 INFO:teuthology.orchestra.run.smithi036.stdout: mgr/telemetry/nag = false 2022-09-13T07:24:04.647 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.647 INFO:teuthology.orchestra.run.smithi036.stdout:[mon] 2022-09-13T07:24:04.647 INFO:teuthology.orchestra.run.smithi036.stdout: debug ms = 1 2022-09-13T07:24:04.647 INFO:teuthology.orchestra.run.smithi036.stdout: debug mon = 20 2022-09-13T07:24:04.648 INFO:teuthology.orchestra.run.smithi036.stdout: debug paxos = 20 2022-09-13T07:24:04.648 INFO:teuthology.orchestra.run.smithi036.stdout: debug auth = 20 2022-09-13T07:24:04.648 INFO:teuthology.orchestra.run.smithi036.stdout: mon data avail warn = 5 2022-09-13T07:24:04.648 INFO:teuthology.orchestra.run.smithi036.stdout: mon mgr mkfs grace = 240 2022-09-13T07:24:04.649 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min pgs per osd = 4 2022-09-13T07:24:04.649 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd reporter subtree level = osd 2022-09-13T07:24:04.649 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd prime pg temp = true 2022-09-13T07:24:04.649 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min bytes per osd = 10 2022-09-13T07:24:04.649 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.650 INFO:teuthology.orchestra.run.smithi036.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-13T07:24:04.650 INFO:teuthology.orchestra.run.smithi036.stdout: auth mon ticket ttl = 660 # 11m 2022-09-13T07:24:04.650 INFO:teuthology.orchestra.run.smithi036.stdout: auth service ticket ttl = 240 # 4m 2022-09-13T07:24:04.650 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.651 INFO:teuthology.orchestra.run.smithi036.stdout: # don't complain about insecure global_id in the test suite 2022-09-13T07:24:04.651 INFO:teuthology.orchestra.run.smithi036.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-13T07:24:04.652 INFO:teuthology.orchestra.run.smithi036.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-13T07:24:04.652 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.652 INFO:teuthology.orchestra.run.smithi036.stdout: # 1m isn't quite enough 2022-09-13T07:24:04.653 INFO:teuthology.orchestra.run.smithi036.stdout: mon_down_mkfs_grace = 2m 2022-09-13T07:24:04.653 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.654 INFO:teuthology.orchestra.run.smithi036.stdout: mon_warn_on_filestore_osds = false 2022-09-13T07:24:04.654 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:24:04.654 INFO:teuthology.orchestra.run.smithi036.stdout:[client] 2022-09-13T07:24:04.655 INFO:teuthology.orchestra.run.smithi036.stdout: rgw cache enabled = true 2022-09-13T07:24:04.655 INFO:teuthology.orchestra.run.smithi036.stdout: rgw enable ops log = true 2022-09-13T07:24:04.656 INFO:teuthology.orchestra.run.smithi036.stdout: rgw enable usage log = true 2022-09-13T07:24:04.656 INFO:teuthology.orchestra.run.smithi036.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-13T07:24:04.657 INFO:teuthology.orchestra.run.smithi036.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-13T07:24:04.657 INFO:teuthology.orchestra.run.smithi036.stdout: debug rgw = 20 2022-09-13T07:24:04.657 INFO:teuthology.orchestra.run.smithi036.stdout: debug rgw sync = 20 2022-09-13T07:24:04.658 INFO:teuthology.orchestra.run.smithi036.stdout: rgw crypt require ssl = False 2022-09-13T07:24:04.658 INFO:teuthology.orchestra.run.smithi036.stdout: rgw crypt s3 kms backend = testing 2022-09-13T07:24:04.658 INFO:teuthology.orchestra.run.smithi036.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-13T07:24:04.659 INFO:teuthology.orchestra.run.smithi036.stdout: rgw curl low speed time = 300 2022-09-13T07:24:04.659 INFO:teuthology.orchestra.run.smithi036.stdout: rgw data log num shards = 4 2022-09-13T07:24:04.660 INFO:teuthology.orchestra.run.smithi036.stdout: rgw md log max shards = 4 2022-09-13T07:24:04.660 INFO:teuthology.orchestra.run.smithi036.stdout: rgw sync data inject err probability = 0.1 2022-09-13T07:24:04.660 INFO:teuthology.orchestra.run.smithi036.stdout: rgw sync log trim interval = 0 2022-09-13T07:24:04.661 INFO:teuthology.orchestra.run.smithi036.stdout: rgw sync meta inject err probability = 0.1 2022-09-13T07:24:04.661 INFO:teuthology.orchestra.run.smithi036.stdout: rgw sync obj etag verify = True 2022-09-13T07:24:04.661 INFO:teuthology.orchestra.run.smithi036.stdout: setgroup = ceph 2022-09-13T07:24:04.662 INFO:teuthology.orchestra.run.smithi036.stdout: setuser = ceph 2022-09-13T07:24:04.662 INFO:teuthology.orchestra.run.smithi036.stdout:[mon.a] 2022-09-13T07:24:04.663 INFO:teuthology.orchestra.run.smithi036.stdout:[client.0] 2022-09-13T07:24:04.663 INFO:teuthology.orchestra.run.smithi036.stdout: rgw data notify interval msec = 0 2022-09-13T07:24:04.663 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.0] 2022-09-13T07:24:04.664 INFO:teuthology.orchestra.run.smithi036.stdout: osd_max_omap_entries_per_request = 10 2022-09-13T07:24:04.664 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.1] 2022-09-13T07:24:04.664 INFO:teuthology.orchestra.run.smithi036.stdout: osd_max_omap_entries_per_request = 1000 2022-09-13T07:24:04.665 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.2] 2022-09-13T07:24:04.665 INFO:teuthology.orchestra.run.smithi036.stdout: osd_max_omap_entries_per_request = 10000 2022-09-13T07:24:04.668 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2022-09-13T07:24:04.668 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2022-09-13T07:24:04.669 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-13T07:24:04.669 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2022-09-13T07:24:04.669 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.670 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2022-09-13T07:24:04.670 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.671 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2022-09-13T07:24:04.671 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.671 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2022-09-13T07:24:04.672 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2022-09-13T07:24:04.672 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.672 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2022-09-13T07:24:04.673 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2022-09-13T07:24:04.673 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.673 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-13T07:24:04.674 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2022-09-13T07:24:04.674 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.675 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2022-09-13T07:24:04.675 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-13T07:24:04.675 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.676 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2022-09-13T07:24:04.676 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.677 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2022-09-13T07:24:04.677 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2022-09-13T07:24:04.677 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2022-09-13T07:24:04.678 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2022-09-13T07:24:04.678 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2022-09-13T07:24:04.678 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2022-09-13T07:24:04.679 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2022-09-13T07:24:04.679 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-13T07:24:04.680 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-13T07:24:04.680 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2022-09-13T07:24:04.680 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.681 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-13T07:24:04.681 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.682 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2022-09-13T07:24:04.682 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.682 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2022-09-13T07:24:04.683 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.683 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2022-09-13T07:24:04.683 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2022-09-13T07:24:04.684 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2022-09-13T07:24:04.684 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = 172.21.15.83 2022-09-13T07:24:04.685 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 236e0d75-a3cd-49bc-af63-ffaa2d54d993 2022-09-13T07:24:04.685 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.685 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2022-09-13T07:24:04.686 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2022-09-13T07:24:04.686 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.687 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2022-09-13T07:24:04.687 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2022-09-13T07:24:04.687 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.688 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2022-09-13T07:24:04.688 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2022-09-13T07:24:04.688 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.689 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2022-09-13T07:24:04.689 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2022-09-13T07:24:04.690 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2022-09-13T07:24:04.690 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.690 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2022-09-13T07:24:04.691 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2022-09-13T07:24:04.691 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.691 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2022-09-13T07:24:04.692 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.692 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2022-09-13T07:24:04.692 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.693 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2022-09-13T07:24:04.693 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.693 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2022-09-13T07:24:04.693 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2022-09-13T07:24:04.694 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.694 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2022-09-13T07:24:04.694 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2022-09-13T07:24:04.694 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2022-09-13T07:24:04.695 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2022-09-13T07:24:04.695 INFO:teuthology.orchestra.run.smithi083.stdout: rocksdb delete range threshold = 0 2022-09-13T07:24:04.695 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.695 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2022-09-13T07:24:04.696 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2022-09-13T07:24:04.696 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2022-09-13T07:24:04.696 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2022-09-13T07:24:04.697 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2022-09-13T07:24:04.697 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2022-09-13T07:24:04.697 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2022-09-13T07:24:04.698 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2022-09-13T07:24:04.698 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.698 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2022-09-13T07:24:04.698 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2022-09-13T07:24:04.699 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2022-09-13T07:24:04.699 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2022-09-13T07:24:04.699 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2022-09-13T07:24:04.700 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2022-09-13T07:24:04.700 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2022-09-13T07:24:04.700 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2022-09-13T07:24:04.700 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2022-09-13T07:24:04.701 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2022-09-13T07:24:04.701 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2022-09-13T07:24:04.701 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.701 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-13T07:24:04.701 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2022-09-13T07:24:04.702 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2022-09-13T07:24:04.702 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.702 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2022-09-13T07:24:04.702 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-13T07:24:04.703 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-13T07:24:04.703 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.703 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2022-09-13T07:24:04.703 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2022-09-13T07:24:04.704 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.704 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2022-09-13T07:24:04.704 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:04.704 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2022-09-13T07:24:04.705 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2022-09-13T07:24:04.705 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2022-09-13T07:24:04.705 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2022-09-13T07:24:04.705 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-13T07:24:04.706 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-13T07:24:04.706 INFO:teuthology.orchestra.run.smithi083.stdout: debug rgw = 20 2022-09-13T07:24:04.706 INFO:teuthology.orchestra.run.smithi083.stdout: debug rgw sync = 20 2022-09-13T07:24:04.706 INFO:teuthology.orchestra.run.smithi083.stdout: rgw crypt require ssl = False 2022-09-13T07:24:04.707 INFO:teuthology.orchestra.run.smithi083.stdout: rgw crypt s3 kms backend = testing 2022-09-13T07:24:04.707 INFO:teuthology.orchestra.run.smithi083.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-13T07:24:04.707 INFO:teuthology.orchestra.run.smithi083.stdout: rgw curl low speed time = 300 2022-09-13T07:24:04.708 INFO:teuthology.orchestra.run.smithi083.stdout: rgw data log num shards = 4 2022-09-13T07:24:04.708 INFO:teuthology.orchestra.run.smithi083.stdout: rgw md log max shards = 4 2022-09-13T07:24:04.708 INFO:teuthology.orchestra.run.smithi083.stdout: rgw sync data inject err probability = 0.1 2022-09-13T07:24:04.709 INFO:teuthology.orchestra.run.smithi083.stdout: rgw sync log trim interval = 0 2022-09-13T07:24:04.709 INFO:teuthology.orchestra.run.smithi083.stdout: rgw sync meta inject err probability = 0.1 2022-09-13T07:24:04.709 INFO:teuthology.orchestra.run.smithi083.stdout: rgw sync obj etag verify = True 2022-09-13T07:24:04.709 INFO:teuthology.orchestra.run.smithi083.stdout: setgroup = ceph 2022-09-13T07:24:04.710 INFO:teuthology.orchestra.run.smithi083.stdout: setuser = ceph 2022-09-13T07:24:04.710 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2022-09-13T07:24:04.710 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2022-09-13T07:24:04.710 INFO:teuthology.orchestra.run.smithi083.stdout: rgw data notify interval msec = 0 2022-09-13T07:24:04.711 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.0] 2022-09-13T07:24:04.711 INFO:teuthology.orchestra.run.smithi083.stdout: osd_max_omap_entries_per_request = 10 2022-09-13T07:24:04.711 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.1] 2022-09-13T07:24:04.711 INFO:teuthology.orchestra.run.smithi083.stdout: osd_max_omap_entries_per_request = 1000 2022-09-13T07:24:04.711 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.2] 2022-09-13T07:24:04.712 INFO:teuthology.orchestra.run.smithi083.stdout: osd_max_omap_entries_per_request = 10000 2022-09-13T07:24:04.712 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2022-09-13T07:24:04.713 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2022-09-13T07:24:04.771 INFO:tasks.ceph:Copying monmap to all nodes... 2022-09-13T07:24:04.771 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:24:04.772 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2022-09-13T07:24:04.792 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:24:04.793 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2022-09-13T07:24:04.858 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi036.front.sepia.ceph.com 2022-09-13T07:24:04.861 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:24:04.864 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd of=/etc/ceph/c2.keyring 2022-09-13T07:24:04.867 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-09-13T07:24:04.916 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2022-09-13T07:24:04.917 DEBUG:teuthology.orchestra.run.smithi036:> dd of=/home/ubuntu/cephtest/c2.monmap 2022-09-13T07:24:04.980 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi083.front.sepia.ceph.com 2022-09-13T07:24:04.980 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:24:04.981 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/c2.keyring 2022-09-13T07:24:04.982 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-09-13T07:24:05.018 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:24:05.018 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/c2.monmap 2022-09-13T07:24:05.076 INFO:tasks.ceph:Setting up mon nodes... 2022-09-13T07:24:05.076 INFO:tasks.ceph:Setting up mgr nodes... 2022-09-13T07:24:05.077 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2022-09-13T07:24:05.168 INFO:teuthology.orchestra.run.smithi083.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2022-09-13T07:24:05.171 INFO:tasks.ceph:Setting up mds nodes... 2022-09-13T07:24:05.171 INFO:tasks.ceph_client:Setting up client nodes... 2022-09-13T07:24:05.172 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2022-09-13T07:24:05.215 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/c2.client.0.keyring 2022-09-13T07:24:05.227 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2022-09-13T07:24:05.310 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/c2.client.1.keyring 2022-09-13T07:24:05.322 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-09-13T07:24:05.322 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi036.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-13T07:24:05.323 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2022-09-13T07:24:05.388 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-13T07:24:05.389 INFO:tasks.ceph:role: c2.osd.0 2022-09-13T07:24:05.389 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com 2022-09-13T07:24:05.390 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-09-13T07:24:05.463 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-09-13T07:24:05.463 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-13T07:24:05.464 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-13T07:24:05.464 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-13T07:24:05.464 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-13T07:24:05.465 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2022-09-13T07:24:05.465 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-13T07:24:05.465 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-13T07:24:05.465 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-13T07:24:05.465 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-13T07:24:05.886 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2022-09-13T07:24:05.888 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2022-09-13T07:24:05.888 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2022-09-13T07:24:06.060 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2022-09-13T07:24:06.093 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2022-09-13T07:24:06.160 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-13T07:24:06.161 INFO:tasks.ceph:role: c2.osd.1 2022-09-13T07:24:06.161 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com 2022-09-13T07:24:06.162 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-09-13T07:24:06.236 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-09-13T07:24:06.236 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-13T07:24:06.236 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-13T07:24:06.237 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-13T07:24:06.237 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-13T07:24:06.237 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2022-09-13T07:24:06.238 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-13T07:24:06.238 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-13T07:24:06.238 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-13T07:24:06.239 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-13T07:24:06.663 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2022-09-13T07:24:06.665 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2022-09-13T07:24:06.666 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2022-09-13T07:24:06.769 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2022-09-13T07:24:06.801 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2022-09-13T07:24:06.868 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-13T07:24:06.869 INFO:tasks.ceph:role: c2.osd.2 2022-09-13T07:24:06.869 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com 2022-09-13T07:24:06.869 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-09-13T07:24:06.943 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-09-13T07:24:06.944 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-13T07:24:06.944 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-13T07:24:06.944 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-13T07:24:06.945 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-13T07:24:06.945 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2022-09-13T07:24:06.945 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-13T07:24:06.945 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-13T07:24:06.945 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-13T07:24:06.946 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-13T07:24:07.382 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2022-09-13T07:24:07.384 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2022-09-13T07:24:07.384 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2022-09-13T07:24:07.536 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2022-09-13T07:24:07.570 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2022-09-13T07:24:07.657 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:07.654+0000 7efea1c61440 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2022-09-13T07:24:07.659 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:07.656+0000 7efea1c61440 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2022-09-13T07:24:07.660 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:07.656+0000 7efea1c61440 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-09-13T07:24:07.660 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:07.656+0000 7efea1c61440 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-09-13T07:24:07.660 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:07.656+0000 7efea1c61440 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-09-13T07:24:07.661 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:07.658+0000 7efea1c61440 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2022-09-13T07:24:09.731 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2022-09-13T07:24:09.760 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2022-09-13T07:24:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:09.843+0000 7f21e6f6e440 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2022-09-13T07:24:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:09.843+0000 7f21e6f6e440 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2022-09-13T07:24:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:09.843+0000 7f21e6f6e440 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-09-13T07:24:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:09.843+0000 7f21e6f6e440 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-09-13T07:24:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:09.843+0000 7f21e6f6e440 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-09-13T07:24:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:09.844+0000 7f21e6f6e440 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2022-09-13T07:24:11.914 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2022-09-13T07:24:11.943 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2022-09-13T07:24:12.029 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:12.026+0000 7fdadb27b440 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2022-09-13T07:24:12.030 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:12.026+0000 7fdadb27b440 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2022-09-13T07:24:12.030 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:12.026+0000 7fdadb27b440 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-09-13T07:24:12.030 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:12.026+0000 7fdadb27b440 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-09-13T07:24:12.031 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:12.026+0000 7fdadb27b440 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-09-13T07:24:12.031 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:12.027+0000 7fdadb27b440 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2022-09-13T07:24:14.096 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2022-09-13T07:24:14.124 INFO:tasks.ceph:Reading keys from all nodes... 2022-09-13T07:24:14.125 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:24:14.125 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2022-09-13T07:24:14.192 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:24:14.192 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2022-09-13T07:24:14.259 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:24:14.259 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2022-09-13T07:24:14.326 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:24:14.326 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2022-09-13T07:24:14.393 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:24:14.394 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2022-09-13T07:24:14.451 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:24:14.452 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2022-09-13T07:24:14.516 INFO:tasks.ceph:Adding keys to all mons... 2022-09-13T07:24:14.517 DEBUG:teuthology.orchestra.run.smithi083:> sudo tee -a /etc/ceph/c2.keyring 2022-09-13T07:24:14.590 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr.x] 2022-09-13T07:24:14.590 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAVMCBjdIvzCRAAqXmvS1NKRUNwQN+AV8YTzg== 2022-09-13T07:24:14.591 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.0] 2022-09-13T07:24:14.591 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAXMCBjHXARJxAAP5CpSsFgz2jnD1neB0uAUw== 2022-09-13T07:24:14.592 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.1] 2022-09-13T07:24:14.592 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAZMCBjqENYMhAAJIpQXKLrhQd0gC2pdYuZlw== 2022-09-13T07:24:14.592 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.2] 2022-09-13T07:24:14.593 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAcMCBjQZ+jARAA0xbJor9XOWUAv4GZqcFopA== 2022-09-13T07:24:14.593 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2022-09-13T07:24:14.593 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAVMCBjcr20DBAAcv+bmDu3Iip8/q6fl6V+AQ== 2022-09-13T07:24:14.593 INFO:teuthology.orchestra.run.smithi083.stdout:[client.1] 2022-09-13T07:24:14.594 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAVMCBjPsJfEhAAaDb7I0ENmCt5rxWbdxcipA== 2022-09-13T07:24:14.595 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-09-13T07:24:14.696 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-13T07:24:14.749 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-13T07:24:14.804 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-13T07:24:14.879 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-13T07:24:14.986 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-13T07:24:15.044 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-09-13T07:24:15.045 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2022-09-13T07:24:15.073 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2022-09-13T07:24:15.173 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2022-09-13T07:24:15.205 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/c2.monmap 2022-09-13T07:24:15.270 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2022-09-13T07:24:15.271 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2022-09-13T07:24:15.271 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2022-09-13T07:24:15.316 INFO:tasks.ceph.c2.mon.a:Started 2022-09-13T07:24:15.317 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2022-09-13T07:24:15.318 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2022-09-13T07:24:15.318 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2022-09-13T07:24:15.321 INFO:tasks.ceph.c2.mgr.x:Started 2022-09-13T07:24:15.322 INFO:tasks.ceph:Setting crush tunables to default 2022-09-13T07:24:15.322 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster c2 osd crush tunables default 2022-09-13T07:24:15.528 INFO:teuthology.orchestra.run.smithi083.stderr:adjusted tunables profile to default 2022-09-13T07:24:15.539 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2022-09-13T07:24:15.540 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:24:15.541 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2022-09-13T07:24:15.568 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:24:15.569 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2022-09-13T07:24:15.636 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2022-09-13T07:24:15.636 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2022-09-13T07:24:15.702 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster c2 osd new 1feb045f-cd41-4d40-a903-66fb5a9fbbfd 0 2022-09-13T07:24:15.708 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:15.706+0000 7fe7799f6080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-13T07:24:15.860 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:15.857+0000 7fe7799f6080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-13T07:24:15.919 INFO:teuthology.orchestra.run.smithi083.stdout:0 2022-09-13T07:24:15.931 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster c2 osd new 2cdece5d-f209-429d-881d-222fc370ee58 1 2022-09-13T07:24:16.142 INFO:teuthology.orchestra.run.smithi083.stdout:1 2022-09-13T07:24:16.155 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster c2 osd new 993fa405-7fa9-4643-b681-fac7d5414f51 2 2022-09-13T07:24:16.391 INFO:teuthology.orchestra.run.smithi083.stdout:2 2022-09-13T07:24:16.396 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:16.393+0000 7fe7799f6080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-13T07:24:16.404 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2022-09-13T07:24:16.404 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2022-09-13T07:24:16.407 INFO:tasks.ceph.c2.osd.0:Started 2022-09-13T07:24:16.408 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2022-09-13T07:24:16.408 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2022-09-13T07:24:16.411 INFO:tasks.ceph.c2.osd.1:Started 2022-09-13T07:24:16.412 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2022-09-13T07:24:16.413 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2022-09-13T07:24:16.416 INFO:tasks.ceph.c2.osd.2:Started 2022-09-13T07:24:16.417 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-09-13T07:24:16.682 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:16.683 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":5,"fsid":"236e0d75-a3cd-49bc-af63-ffaa2d54d993","created":"2022-09-13T07:24:15.430464+0000","modified":"2022-09-13T07:24:16.387717+0000","last_up_change":"0.000000","last_in_change":"2022-09-13T07:24:16.387717+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"1feb045f-cd41-4d40-a903-66fb5a9fbbfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2cdece5d-f209-429d-881d-222fc370ee58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"993fa405-7fa9-4643-b681-fac7d5414f51","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-13T07:24:16.694 INFO:tasks.ceph.ceph_manager.c2:[] 2022-09-13T07:24:16.695 INFO:tasks.ceph:Waiting for OSDs to come up 2022-09-13T07:24:16.820 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:16.817+0000 7fe7799f6080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-13T07:24:16.917 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:16.915+0000 7fe7799f6080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-13T07:24:16.998 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2022-09-13T07:24:17.013 INFO:tasks.ceph.c2.osd.1.smithi083.stderr:2022-09-13T07:24:17.011+0000 7fe2fa567440 -1 Falling back to public interface 2022-09-13T07:24:17.016 INFO:tasks.ceph.c2.osd.0.smithi083.stderr:2022-09-13T07:24:17.014+0000 7f4471cc9440 -1 Falling back to public interface 2022-09-13T07:24:17.025 INFO:tasks.ceph.c2.osd.2.smithi083.stderr:2022-09-13T07:24:17.023+0000 7f4ee0123440 -1 Falling back to public interface 2022-09-13T07:24:17.125 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:17.123+0000 7fe7799f6080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-13T07:24:17.187 INFO:teuthology.misc.health.smithi083.stdout: 2022-09-13T07:24:17.188 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":5,"fsid":"236e0d75-a3cd-49bc-af63-ffaa2d54d993","created":"2022-09-13T07:24:15.430464+0000","modified":"2022-09-13T07:24:16.387717+0000","last_up_change":"0.000000","last_in_change":"2022-09-13T07:24:16.387717+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"1feb045f-cd41-4d40-a903-66fb5a9fbbfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2cdece5d-f209-429d-881d-222fc370ee58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"993fa405-7fa9-4643-b681-fac7d5414f51","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-13T07:24:17.199 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-09-13T07:24:18.061 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:18.058+0000 7fe7799f6080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-13T07:24:18.139 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:18.136+0000 7fe7799f6080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-13T07:24:18.241 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:18.239+0000 7fe7799f6080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-13T07:24:18.319 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:18.317+0000 7fe7799f6080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-13T07:24:18.608 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:18.605+0000 7fe7799f6080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-13T07:24:18.680 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:18.678+0000 7fe7799f6080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-13T07:24:18.752 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:18.749+0000 7fe7799f6080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-13T07:24:19.373 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:19.370+0000 7fe7799f6080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-13T07:24:19.513 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:19.510+0000 7fe7799f6080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-13T07:24:19.657 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:19.654+0000 7fe7799f6080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-13T07:24:19.729 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:19.727+0000 7fe7799f6080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-13T07:24:19.771 INFO:tasks.ceph.c2.osd.0.smithi083.stderr:2022-09-13T07:24:19.768+0000 7f4471cc9440 -1 osd.0 0 log_to_monitors true 2022-09-13T07:24:19.776 INFO:tasks.ceph.c2.osd.1.smithi083.stderr:2022-09-13T07:24:19.773+0000 7fe2fa567440 -1 osd.1 0 log_to_monitors true 2022-09-13T07:24:19.785 INFO:tasks.ceph.c2.osd.2.smithi083.stderr:2022-09-13T07:24:19.783+0000 7f4ee0123440 -1 osd.2 0 log_to_monitors true 2022-09-13T07:24:19.873 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:19.870+0000 7fe7799f6080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-13T07:24:19.967 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:19.965+0000 7fe7799f6080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-13T07:24:20.209 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:20.206+0000 7fe7799f6080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-13T07:24:20.473 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:20.471+0000 7fe7799f6080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-13T07:24:20.554 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:20.551+0000 7fe7799f6080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-13T07:24:20.733 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:20.731+0000 7fe7799f6080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-13T07:24:20.900 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:20.897+0000 7fe7799f6080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-13T07:24:21.932 INFO:tasks.ceph.c2.osd.2.smithi083.stderr:2022-09-13T07:24:21.930+0000 7f4ed3ed0700 -1 osd.2 0 waiting for initial osdmap 2022-09-13T07:24:21.960 INFO:tasks.ceph.c2.osd.2.smithi083.stderr:2022-09-13T07:24:21.958+0000 7f4ecc2ab700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-13T07:24:21.977 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T07:24:21.975+0000 7fe7387af700 -1 mgr.server handle_report got status from non-daemon mon.a 2022-09-13T07:24:22.037 INFO:tasks.ceph.c2.osd.0.smithi083.stderr:2022-09-13T07:24:22.035+0000 7f4465a76700 -1 osd.0 0 waiting for initial osdmap 2022-09-13T07:24:22.137 INFO:tasks.ceph.c2.osd.0.smithi083.stderr:2022-09-13T07:24:22.135+0000 7f445de51700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-13T07:24:22.420 INFO:tasks.ceph.c2.osd.1.smithi083.stderr:2022-09-13T07:24:22.418+0000 7fe2ee314700 -1 osd.1 0 waiting for initial osdmap 2022-09-13T07:24:22.437 INFO:tasks.ceph.c2.osd.1.smithi083.stderr:2022-09-13T07:24:22.431+0000 7fe2e66ef700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-13T07:24:23.503 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2022-09-13T07:24:23.805 INFO:teuthology.misc.health.smithi083.stdout: 2022-09-13T07:24:23.805 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":8,"fsid":"236e0d75-a3cd-49bc-af63-ffaa2d54d993","created":"2022-09-13T07:24:15.430464+0000","modified":"2022-09-13T07:24:22.969262+0000","last_up_change":"2022-09-13T07:24:22.969262+0000","last_in_change":"2022-09-13T07:24:16.387717+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"1feb045f-cd41-4d40-a903-66fb5a9fbbfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":101800},{"type":"v1","addr":"172.21.15.83:6809","nonce":101800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":101800},{"type":"v1","addr":"172.21.15.83:6811","nonce":101800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":101800},{"type":"v1","addr":"172.21.15.83:6815","nonce":101800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":101800},{"type":"v1","addr":"172.21.15.83:6813","nonce":101800}]},"public_addr":"172.21.15.83:6809/101800","cluster_addr":"172.21.15.83:6811/101800","heartbeat_back_addr":"172.21.15.83:6815/101800","heartbeat_front_addr":"172.21.15.83:6813/101800","state":["exists","up"]},{"osd":1,"uuid":"2cdece5d-f209-429d-881d-222fc370ee58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":101801},{"type":"v1","addr":"172.21.15.83:6801","nonce":101801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":101801},{"type":"v1","addr":"172.21.15.83:6803","nonce":101801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":101801},{"type":"v1","addr":"172.21.15.83:6807","nonce":101801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":101801},{"type":"v1","addr":"172.21.15.83:6805","nonce":101801}]},"public_addr":"172.21.15.83:6801/101801","cluster_addr":"172.21.15.83:6803/101801","heartbeat_back_addr":"172.21.15.83:6807/101801","heartbeat_front_addr":"172.21.15.83:6805/101801","state":["exists","up"]},{"osd":2,"uuid":"993fa405-7fa9-4643-b681-fac7d5414f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":101802},{"type":"v1","addr":"172.21.15.83:6817","nonce":101802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":101802},{"type":"v1","addr":"172.21.15.83:6819","nonce":101802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":101802},{"type":"v1","addr":"172.21.15.83:6823","nonce":101802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":101802},{"type":"v1","addr":"172.21.15.83:6821","nonce":101802}]},"public_addr":"172.21.15.83:6817/101802","cluster_addr":"172.21.15.83:6819/101802","heartbeat_back_addr":"172.21.15.83:6823/101802","heartbeat_front_addr":"172.21.15.83:6821/101802","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-13T07:24:23.818 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-09-13T07:24:23.819 INFO:tasks.ceph:Creating RBD pool 2022-09-13T07:24:23.819 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster c2 osd pool create rbd 8 2022-09-13T07:24:24.985 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'rbd' created 2022-09-13T07:24:25.000 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2022-09-13T07:24:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:enabled application 'rbd' on pool 'rbd' 2022-09-13T07:24:26.001 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2022-09-13T07:24:26.002 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2022-09-13T07:24:26.002 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2022-09-13T07:24:26.003 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2022-09-13T07:24:26.003 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-09-13T07:24:26.361 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:26.373 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":101385},{"type":"v1","addr":"172.21.15.83:6825","nonce":101385}]},"active_addr":"172.21.15.83:6825/101385","active_change":"2022-09-13T07:24:20.910023+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":3954120100}]},{"addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":177531800}]},{"addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":3231380059}]}]}} 2022-09-13T07:24:26.376 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2022-09-13T07:24:26.377 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2022-09-13T07:24:26.377 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-09-13T07:24:26.683 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:26.683 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":11,"fsid":"236e0d75-a3cd-49bc-af63-ffaa2d54d993","created":"2022-09-13T07:24:15.430464+0000","modified":"2022-09-13T07:24:25.977875+0000","last_up_change":"2022-09-13T07:24:22.969262+0000","last_in_change":"2022-09-13T07:24:16.387717+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-13T07:24:23.008156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-09-13T07:24:24.142375+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"1feb045f-cd41-4d40-a903-66fb5a9fbbfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":101800},{"type":"v1","addr":"172.21.15.83:6809","nonce":101800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":101800},{"type":"v1","addr":"172.21.15.83:6811","nonce":101800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":101800},{"type":"v1","addr":"172.21.15.83:6815","nonce":101800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":101800},{"type":"v1","addr":"172.21.15.83:6813","nonce":101800}]},"public_addr":"172.21.15.83:6809/101800","cluster_addr":"172.21.15.83:6811/101800","heartbeat_back_addr":"172.21.15.83:6815/101800","heartbeat_front_addr":"172.21.15.83:6813/101800","state":["exists","up"]},{"osd":1,"uuid":"2cdece5d-f209-429d-881d-222fc370ee58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":101801},{"type":"v1","addr":"172.21.15.83:6801","nonce":101801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":101801},{"type":"v1","addr":"172.21.15.83:6803","nonce":101801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":101801},{"type":"v1","addr":"172.21.15.83:6807","nonce":101801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":101801},{"type":"v1","addr":"172.21.15.83:6805","nonce":101801}]},"public_addr":"172.21.15.83:6801/101801","cluster_addr":"172.21.15.83:6803/101801","heartbeat_back_addr":"172.21.15.83:6807/101801","heartbeat_front_addr":"172.21.15.83:6805/101801","state":["exists","up"]},{"osd":2,"uuid":"993fa405-7fa9-4643-b681-fac7d5414f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":101802},{"type":"v1","addr":"172.21.15.83:6817","nonce":101802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":101802},{"type":"v1","addr":"172.21.15.83:6819","nonce":101802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":101802},{"type":"v1","addr":"172.21.15.83:6823","nonce":101802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":101802},{"type":"v1","addr":"172.21.15.83:6821","nonce":101802}]},"public_addr":"172.21.15.83:6817/101802","cluster_addr":"172.21.15.83:6819/101802","heartbeat_back_addr":"172.21.15.83:6823/101802","heartbeat_front_addr":"172.21.15.83:6821/101802","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-13T07:24:20.733169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-13T07:24:20.770908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-13T07:24:20.759305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-13T07:24:26.696 INFO:tasks.ceph.ceph_manager.c2:all up! 2022-09-13T07:24:26.697 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-09-13T07:24:27.005 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:27.006 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":12,"fsid":"236e0d75-a3cd-49bc-af63-ffaa2d54d993","created":"2022-09-13T07:24:15.430464+0000","modified":"2022-09-13T07:24:26.980616+0000","last_up_change":"2022-09-13T07:24:22.969262+0000","last_in_change":"2022-09-13T07:24:16.387717+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-13T07:24:23.008156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-09-13T07:24:24.142375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"1feb045f-cd41-4d40-a903-66fb5a9fbbfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":101800},{"type":"v1","addr":"172.21.15.83:6809","nonce":101800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":101800},{"type":"v1","addr":"172.21.15.83:6811","nonce":101800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":101800},{"type":"v1","addr":"172.21.15.83:6815","nonce":101800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":101800},{"type":"v1","addr":"172.21.15.83:6813","nonce":101800}]},"public_addr":"172.21.15.83:6809/101800","cluster_addr":"172.21.15.83:6811/101800","heartbeat_back_addr":"172.21.15.83:6815/101800","heartbeat_front_addr":"172.21.15.83:6813/101800","state":["exists","up"]},{"osd":1,"uuid":"2cdece5d-f209-429d-881d-222fc370ee58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":101801},{"type":"v1","addr":"172.21.15.83:6801","nonce":101801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":101801},{"type":"v1","addr":"172.21.15.83:6803","nonce":101801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":101801},{"type":"v1","addr":"172.21.15.83:6807","nonce":101801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":101801},{"type":"v1","addr":"172.21.15.83:6805","nonce":101801}]},"public_addr":"172.21.15.83:6801/101801","cluster_addr":"172.21.15.83:6803/101801","heartbeat_back_addr":"172.21.15.83:6807/101801","heartbeat_front_addr":"172.21.15.83:6805/101801","state":["exists","up"]},{"osd":2,"uuid":"993fa405-7fa9-4643-b681-fac7d5414f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":101802},{"type":"v1","addr":"172.21.15.83:6817","nonce":101802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":101802},{"type":"v1","addr":"172.21.15.83:6819","nonce":101802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":101802},{"type":"v1","addr":"172.21.15.83:6823","nonce":101802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":101802},{"type":"v1","addr":"172.21.15.83:6821","nonce":101802}]},"public_addr":"172.21.15.83:6817/101802","cluster_addr":"172.21.15.83:6819/101802","heartbeat_back_addr":"172.21.15.83:6823/101802","heartbeat_front_addr":"172.21.15.83:6821/101802","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-13T07:24:20.733169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-13T07:24:20.770908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-13T07:24:20.759305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-13T07:24:27.020 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2022-09-13T07:24:27.020 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2022-09-13T07:24:27.021 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2022-09-13T07:24:27.184 INFO:teuthology.orchestra.run.smithi083.stdout:34359738370 2022-09-13T07:24:27.184 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2022-09-13T07:24:27.206 INFO:teuthology.orchestra.run.smithi083.stdout:34359738370 2022-09-13T07:24:27.207 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-09-13T07:24:27.208 INFO:teuthology.orchestra.run.smithi083.stdout:34359738370 2022-09-13T07:24:27.209 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-09-13T07:24:27.525 INFO:teuthology.orchestra.run.smithi083.stdout:34359738369 2022-09-13T07:24:27.539 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.2 2022-09-13T07:24:27.571 INFO:teuthology.orchestra.run.smithi083.stdout:34359738369 2022-09-13T07:24:27.584 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.0 2022-09-13T07:24:27.585 INFO:teuthology.orchestra.run.smithi083.stdout:34359738369 2022-09-13T07:24:27.597 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.1 2022-09-13T07:24:28.540 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2022-09-13T07:24:28.586 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-09-13T07:24:28.599 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-09-13T07:24:28.870 INFO:teuthology.orchestra.run.smithi083.stdout:34359738369 2022-09-13T07:24:28.884 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.2 2022-09-13T07:24:28.936 INFO:teuthology.orchestra.run.smithi083.stdout:34359738369 2022-09-13T07:24:28.954 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.0 2022-09-13T07:24:28.984 INFO:teuthology.orchestra.run.smithi083.stdout:34359738369 2022-09-13T07:24:28.998 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.1 2022-09-13T07:24:29.886 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2022-09-13T07:24:29.955 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-09-13T07:24:30.000 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-09-13T07:24:30.235 INFO:teuthology.orchestra.run.smithi083.stdout:34359738370 2022-09-13T07:24:30.250 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.2 2022-09-13T07:24:30.250 DEBUG:teuthology.parallel:result is None 2022-09-13T07:24:30.344 INFO:teuthology.orchestra.run.smithi083.stdout:34359738370 2022-09-13T07:24:30.359 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.0 2022-09-13T07:24:30.360 DEBUG:teuthology.parallel:result is None 2022-09-13T07:24:30.374 INFO:teuthology.orchestra.run.smithi083.stdout:34359738370 2022-09-13T07:24:30.387 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.1 2022-09-13T07:24:30.388 DEBUG:teuthology.parallel:result is None 2022-09-13T07:24:30.388 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2022-09-13T07:24:30.389 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2022-09-13T07:24:30.729 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:30.731 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2022-09-13T07:24:30.742 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2022-09-13T07:24:28.969950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25024,"kb_used_data":1448,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547776,"statfs":{"total":322122547200,"available":322096922624,"internally_reserved":0,"allocated":1482752,"data_stored":1257551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:24:25.987811+0000","last_change":"2022-09-13T07:24:25.986607+0000","last_active":"2022-09-13T07:24:25.987811+0000","last_peered":"2022-09-13T07:24:25.987811+0000","last_clean":"2022-09-13T07:24:25.987811+0000","last_became_active":"2022-09-13T07:24:25.985618+0000","last_became_peered":"2022-09-13T07:24:25.985618+0000","last_unstale":"2022-09-13T07:24:25.987811+0000","last_undegraded":"2022-09-13T07:24:25.987811+0000","last_fullsized":"2022-09-13T07:24:25.987811+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T11:08:30.610480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:24:25.988034+0000","last_change":"2022-09-13T07:24:25.985973+0000","last_active":"2022-09-13T07:24:25.988034+0000","last_peered":"2022-09-13T07:24:25.988034+0000","last_clean":"2022-09-13T07:24:25.988034+0000","last_became_active":"2022-09-13T07:24:25.985655+0000","last_became_peered":"2022-09-13T07:24:25.985655+0000","last_unstale":"2022-09-13T07:24:25.988034+0000","last_undegraded":"2022-09-13T07:24:25.988034+0000","last_fullsized":"2022-09-13T07:24:25.988034+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T17:48:18.185370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:24:25.987791+0000","last_change":"2022-09-13T07:24:25.986331+0000","last_active":"2022-09-13T07:24:25.987791+0000","last_peered":"2022-09-13T07:24:25.987791+0000","last_clean":"2022-09-13T07:24:25.987791+0000","last_became_active":"2022-09-13T07:24:25.985707+0000","last_became_peered":"2022-09-13T07:24:25.985707+0000","last_unstale":"2022-09-13T07:24:25.987791+0000","last_undegraded":"2022-09-13T07:24:25.987791+0000","last_fullsized":"2022-09-13T07:24:25.987791+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T17:50:21.664000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-13T07:24:26.983701+0000","last_change":"2022-09-13T07:24:25.987584+0000","last_active":"2022-09-13T07:24:26.983701+0000","last_peered":"2022-09-13T07:24:26.983701+0000","last_clean":"2022-09-13T07:24:26.983701+0000","last_became_active":"2022-09-13T07:24:25.987332+0000","last_became_peered":"2022-09-13T07:24:25.987332+0000","last_unstale":"2022-09-13T07:24:26.983701+0000","last_undegraded":"2022-09-13T07:24:26.983701+0000","last_fullsized":"2022-09-13T07:24:26.983701+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T10:03:12.854641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:24:25.981765+0000","last_change":"2022-09-13T07:24:24.987668+0000","last_active":"2022-09-13T07:24:25.981765+0000","last_peered":"2022-09-13T07:24:25.981765+0000","last_clean":"2022-09-13T07:24:25.981765+0000","last_became_active":"2022-09-13T07:24:24.986879+0000","last_became_peered":"2022-09-13T07:24:24.986879+0000","last_unstale":"2022-09-13T07:24:25.981765+0000","last_undegraded":"2022-09-13T07:24:25.981765+0000","last_fullsized":"2022-09-13T07:24:25.981765+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:23.970161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:23.970161+0000","last_clean_scrub_stamp":"2022-09-13T07:24:23.970161+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T09:46:31.295508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:24:25.987846+0000","last_change":"2022-09-13T07:24:25.986130+0000","last_active":"2022-09-13T07:24:25.987846+0000","last_peered":"2022-09-13T07:24:25.987846+0000","last_clean":"2022-09-13T07:24:25.987846+0000","last_became_active":"2022-09-13T07:24:25.985877+0000","last_became_peered":"2022-09-13T07:24:25.985877+0000","last_unstale":"2022-09-13T07:24:25.987846+0000","last_undegraded":"2022-09-13T07:24:25.987846+0000","last_fullsized":"2022-09-13T07:24:25.987846+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T12:53:56.187740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-13T07:24:26.983963+0000","last_change":"2022-09-13T07:24:25.986914+0000","last_active":"2022-09-13T07:24:26.983963+0000","last_peered":"2022-09-13T07:24:26.983963+0000","last_clean":"2022-09-13T07:24:26.983963+0000","last_became_active":"2022-09-13T07:24:25.986667+0000","last_became_peered":"2022-09-13T07:24:25.986667+0000","last_unstale":"2022-09-13T07:24:26.983963+0000","last_undegraded":"2022-09-13T07:24:26.983963+0000","last_fullsized":"2022-09-13T07:24:26.983963+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T16:09:12.063204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:24:25.987896+0000","last_change":"2022-09-13T07:24:25.987311+0000","last_active":"2022-09-13T07:24:25.987896+0000","last_peered":"2022-09-13T07:24:25.987896+0000","last_clean":"2022-09-13T07:24:25.987896+0000","last_became_active":"2022-09-13T07:24:25.987001+0000","last_became_peered":"2022-09-13T07:24:25.987001+0000","last_unstale":"2022-09-13T07:24:25.987896+0000","last_undegraded":"2022-09-13T07:24:25.987896+0000","last_fullsized":"2022-09-13T07:24:25.987896+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T17:48:33.225683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:24:25.987755+0000","last_change":"2022-09-13T07:24:25.986212+0000","last_active":"2022-09-13T07:24:25.987755+0000","last_peered":"2022-09-13T07:24:25.987755+0000","last_clean":"2022-09-13T07:24:25.987755+0000","last_became_active":"2022-09-13T07:24:25.985952+0000","last_became_peered":"2022-09-13T07:24:25.985952+0000","last_unstale":"2022-09-13T07:24:25.987755+0000","last_undegraded":"2022-09-13T07:24:25.987755+0000","last_fullsized":"2022-09-13T07:24:25.987755+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T13:38:32.119495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7656,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849944,"statfs":{"total":107374182400,"available":107366342656,"internally_reserved":0,"allocated":98304,"data_stored":25605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8620,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848980,"statfs":{"total":107374182400,"available":107365355520,"internally_reserved":0,"allocated":692224,"data_stored":615973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8748,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848852,"statfs":{"total":107374182400,"available":107365224448,"internally_reserved":0,"allocated":692224,"data_stored":615973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.222}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-13T07:24:30.744 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2022-09-13T07:24:31.048 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:31.050 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2022-09-13T07:24:31.062 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2022-09-13T07:24:30.970296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25040,"kb_used_data":1464,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547760,"statfs":{"total":322122547200,"available":322096906240,"internally_reserved":0,"allocated":1499136,"data_stored":1264483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000346"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:24:25.987811+0000","last_change":"2022-09-13T07:24:25.986607+0000","last_active":"2022-09-13T07:24:25.987811+0000","last_peered":"2022-09-13T07:24:25.987811+0000","last_clean":"2022-09-13T07:24:25.987811+0000","last_became_active":"2022-09-13T07:24:25.985618+0000","last_became_peered":"2022-09-13T07:24:25.985618+0000","last_unstale":"2022-09-13T07:24:25.987811+0000","last_undegraded":"2022-09-13T07:24:25.987811+0000","last_fullsized":"2022-09-13T07:24:25.987811+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T11:08:30.610480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:24:25.988034+0000","last_change":"2022-09-13T07:24:25.985973+0000","last_active":"2022-09-13T07:24:25.988034+0000","last_peered":"2022-09-13T07:24:25.988034+0000","last_clean":"2022-09-13T07:24:25.988034+0000","last_became_active":"2022-09-13T07:24:25.985655+0000","last_became_peered":"2022-09-13T07:24:25.985655+0000","last_unstale":"2022-09-13T07:24:25.988034+0000","last_undegraded":"2022-09-13T07:24:25.988034+0000","last_fullsized":"2022-09-13T07:24:25.988034+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T17:48:18.185370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:24:25.987791+0000","last_change":"2022-09-13T07:24:25.986331+0000","last_active":"2022-09-13T07:24:25.987791+0000","last_peered":"2022-09-13T07:24:25.987791+0000","last_clean":"2022-09-13T07:24:25.987791+0000","last_became_active":"2022-09-13T07:24:25.985707+0000","last_became_peered":"2022-09-13T07:24:25.985707+0000","last_unstale":"2022-09-13T07:24:25.987791+0000","last_undegraded":"2022-09-13T07:24:25.987791+0000","last_fullsized":"2022-09-13T07:24:25.987791+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T17:50:21.664000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-13T07:24:26.983701+0000","last_change":"2022-09-13T07:24:25.987584+0000","last_active":"2022-09-13T07:24:26.983701+0000","last_peered":"2022-09-13T07:24:26.983701+0000","last_clean":"2022-09-13T07:24:26.983701+0000","last_became_active":"2022-09-13T07:24:25.987332+0000","last_became_peered":"2022-09-13T07:24:25.987332+0000","last_unstale":"2022-09-13T07:24:26.983701+0000","last_undegraded":"2022-09-13T07:24:26.983701+0000","last_fullsized":"2022-09-13T07:24:26.983701+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T10:03:12.854641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:24:25.981765+0000","last_change":"2022-09-13T07:24:24.987668+0000","last_active":"2022-09-13T07:24:25.981765+0000","last_peered":"2022-09-13T07:24:25.981765+0000","last_clean":"2022-09-13T07:24:25.981765+0000","last_became_active":"2022-09-13T07:24:24.986879+0000","last_became_peered":"2022-09-13T07:24:24.986879+0000","last_unstale":"2022-09-13T07:24:25.981765+0000","last_undegraded":"2022-09-13T07:24:25.981765+0000","last_fullsized":"2022-09-13T07:24:25.981765+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:23.970161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:23.970161+0000","last_clean_scrub_stamp":"2022-09-13T07:24:23.970161+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T09:46:31.295508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:24:25.987846+0000","last_change":"2022-09-13T07:24:25.986130+0000","last_active":"2022-09-13T07:24:25.987846+0000","last_peered":"2022-09-13T07:24:25.987846+0000","last_clean":"2022-09-13T07:24:25.987846+0000","last_became_active":"2022-09-13T07:24:25.985877+0000","last_became_peered":"2022-09-13T07:24:25.985877+0000","last_unstale":"2022-09-13T07:24:25.987846+0000","last_undegraded":"2022-09-13T07:24:25.987846+0000","last_fullsized":"2022-09-13T07:24:25.987846+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T12:53:56.187740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-13T07:24:26.983963+0000","last_change":"2022-09-13T07:24:25.986914+0000","last_active":"2022-09-13T07:24:26.983963+0000","last_peered":"2022-09-13T07:24:26.983963+0000","last_clean":"2022-09-13T07:24:26.983963+0000","last_became_active":"2022-09-13T07:24:25.986667+0000","last_became_peered":"2022-09-13T07:24:25.986667+0000","last_unstale":"2022-09-13T07:24:26.983963+0000","last_undegraded":"2022-09-13T07:24:26.983963+0000","last_fullsized":"2022-09-13T07:24:26.983963+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T16:09:12.063204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:24:27.191521+0000","last_change":"2022-09-13T07:24:25.987311+0000","last_active":"2022-09-13T07:24:27.191521+0000","last_peered":"2022-09-13T07:24:27.191521+0000","last_clean":"2022-09-13T07:24:27.191521+0000","last_became_active":"2022-09-13T07:24:25.987001+0000","last_became_peered":"2022-09-13T07:24:25.987001+0000","last_unstale":"2022-09-13T07:24:27.191521+0000","last_undegraded":"2022-09-13T07:24:27.191521+0000","last_fullsized":"2022-09-13T07:24:27.191521+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T17:48:33.225683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-13T07:24:25.987755+0000","last_change":"2022-09-13T07:24:25.986212+0000","last_active":"2022-09-13T07:24:25.987755+0000","last_peered":"2022-09-13T07:24:25.987755+0000","last_clean":"2022-09-13T07:24:25.987755+0000","last_became_active":"2022-09-13T07:24:25.985952+0000","last_became_peered":"2022-09-13T07:24:25.985952+0000","last_unstale":"2022-09-13T07:24:25.987755+0000","last_undegraded":"2022-09-13T07:24:25.987755+0000","last_fullsized":"2022-09-13T07:24:25.987755+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-13T07:24:24.975779+0000","last_clean_scrub_stamp":"2022-09-13T07:24:24.975779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T13:38:32.119495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7664,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849936,"statfs":{"total":107374182400,"available":107366334464,"internally_reserved":0,"allocated":106496,"data_stored":29071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8628,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848972,"statfs":{"total":107374182400,"available":107365347328,"internally_reserved":0,"allocated":700416,"data_stored":619439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8748,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848852,"statfs":{"total":107374182400,"available":107365224448,"internally_reserved":0,"allocated":692224,"data_stored":615973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.222}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-13T07:24:31.064 INFO:tasks.ceph.ceph_manager.c2:clean! 2022-09-13T07:24:31.064 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2022-09-13T07:24:31.065 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2022-09-13T07:24:31.066 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2022-09-13T07:24:31.407 INFO:teuthology.orchestra.run.smithi083.stdout: 2022-09-13T07:24:31.408 INFO:teuthology.orchestra.run.smithi083.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-09-13T07:24:31.421 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2022-09-13T07:24:31.422 INFO:teuthology.run_tasks:Running task rgw... 2022-09-13T07:24:31.438 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2022-09-13T07:24:31.439 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2022-09-13T07:24:31.439 INFO:tasks.rgw:Creating data pools 2022-09-13T07:24:31.440 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2022-09-13T07:24:31.440 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2022-09-13T07:24:32.392 INFO:teuthology.orchestra.run.smithi036.stderr:pool 'default.rgw.buckets.data' created 2022-09-13T07:24:32.407 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2022-09-13T07:24:33.401 INFO:teuthology.orchestra.run.smithi036.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-13T07:24:33.422 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2022-09-13T07:24:34.408 INFO:teuthology.orchestra.run.smithi036.stderr:pool 'default.rgw.buckets.index' created 2022-09-13T07:24:34.419 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2022-09-13T07:24:35.426 INFO:teuthology.orchestra.run.smithi036.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-13T07:24:35.441 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2022-09-13T07:24:35.442 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2022-09-13T07:24:35.772 INFO:teuthology.orchestra.run.smithi036.stderr:pool 'default.rgw.buckets.data' already exists 2022-09-13T07:24:35.783 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2022-09-13T07:24:36.410 INFO:teuthology.orchestra.run.smithi036.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-13T07:24:36.421 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2022-09-13T07:24:36.745 INFO:teuthology.orchestra.run.smithi036.stderr:pool 'default.rgw.buckets.index' already exists 2022-09-13T07:24:36.756 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2022-09-13T07:24:37.411 INFO:teuthology.orchestra.run.smithi036.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-13T07:24:37.422 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2022-09-13T07:24:37.423 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2022-09-13T07:24:38.428 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'default.rgw.buckets.data' created 2022-09-13T07:24:38.440 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2022-09-13T07:24:39.434 INFO:teuthology.orchestra.run.smithi083.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-13T07:24:39.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2022-09-13T07:24:40.451 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'default.rgw.buckets.index' created 2022-09-13T07:24:40.467 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2022-09-13T07:24:41.462 INFO:teuthology.orchestra.run.smithi083.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-13T07:24:41.485 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2022-09-13T07:24:41.485 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2022-09-13T07:24:41.795 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'default.rgw.buckets.data' already exists 2022-09-13T07:24:41.806 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2022-09-13T07:24:42.455 INFO:teuthology.orchestra.run.smithi083.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-13T07:24:42.466 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2022-09-13T07:24:42.775 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'default.rgw.buckets.index' already exists 2022-09-13T07:24:42.786 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2022-09-13T07:24:43.457 INFO:teuthology.orchestra.run.smithi083.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-13T07:24:43.469 DEBUG:tasks.rgw:Pools created 2022-09-13T07:24:43.469 INFO:tasks.rgw:Configuring compression type = random 2022-09-13T07:24:43.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2022-09-13T07:24:43.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2022-09-13T07:24:43.472 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2022-09-13T07:24:43.551 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:24:43.551 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:24:43.558 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:43.554+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:45.438 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.434+0000 7fd8f8fa9600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:45.438 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.434+0000 7fd8f8fa9600 20 realm 2022-09-13T07:24:45.439 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.434+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:45.439 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.435+0000 7fd8f8fa9600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:45.440 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.435+0000 7fd8f8fa9600 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-13T07:24:45.440 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.435+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:45.441 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.435+0000 7fd8f8fa9600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:45.441 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.435+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:45.442 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.436+0000 7fd8f8fa9600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:45.442 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.436+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:45.442 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.436+0000 7fd8f8fa9600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:45.442 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.436+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:45.443 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.437+0000 7fd8f8fa9600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:45.443 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.437+0000 7fd8f8fa9600 10 Creating default zonegroup 2022-09-13T07:24:45.443 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.437+0000 7fd8f8fa9600 10 couldn't find old data placement pools config, setting up new ones for the zone 2022-09-13T07:24:45.451 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.447+0000 7fd8f8fa9600 10 WARNING: store->list_zones() returned r=-2 2022-09-13T07:24:45.451 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.447+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:45.451 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.448+0000 7fd8f8fa9600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:45.453 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.450+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:45.454 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.450+0000 7fd8f8fa9600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:45.454 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.450+0000 7fd8f8fa9600 10 could not read realm id: (2) No such file or directory 2022-09-13T07:24:45.454 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.450+0000 7fd8f8fa9600 10 WARNING: failed to set zone as default, r=-22 2022-09-13T07:24:45.455 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.450+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:45.455 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.451+0000 7fd8f8fa9600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:45.458 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.455+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:45.459 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.456+0000 7fd8f8fa9600 20 rados_obj.operate() r=0 bl.length=873 2022-09-13T07:24:45.460 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.456+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:45.460 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.456+0000 7fd8f8fa9600 20 rados_obj.operate() r=0 bl.length=398 2022-09-13T07:24:45.460 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.456+0000 7fd8f8fa9600 10 Using default name default 2022-09-13T07:24:45.461 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.456+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:45.461 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.457+0000 7fd8f8fa9600 20 rados_obj.operate() r=0 bl.length=46 2022-09-13T07:24:45.461 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.457+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:45.462 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.457+0000 7fd8f8fa9600 20 rados_obj.operate() r=0 bl.length=873 2022-09-13T07:24:45.462 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.457+0000 7fd8f8fa9600 20 zone default found 2022-09-13T07:24:45.463 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.457+0000 7fd8f8fa9600 4 Realm: () 2022-09-13T07:24:45.463 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.457+0000 7fd8f8fa9600 4 ZoneGroup: default (6ccbf938-a605-4e5e-9e3f-0b3c486c3422) 2022-09-13T07:24:45.463 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.457+0000 7fd8f8fa9600 4 Zone: default (049575f2-6d5c-49c3-872c-56ad0f34ff5a) 2022-09-13T07:24:45.463 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.457+0000 7fd8f8fa9600 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-13T07:24:45.463 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.457+0000 7fd8f8fa9600 20 zonegroup default 2022-09-13T07:24:45.464 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.457+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:45.464 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.457+0000 7fd8f8fa9600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:45.464 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:45.457+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:47.455 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:47.452+0000 7fd8f8fa9600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:47.455 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:47.452+0000 7fd8f8fa9600 20 rados->read ofs=0 len=0 2022-09-13T07:24:47.456 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:47.453+0000 7fd8f8fa9600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:47.457 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:47.453+0000 7fd8f8fa9600 20 started sync module instance, tier type = 2022-09-13T07:24:47.457 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:47.453+0000 7fd8f8fa9600 20 started zone id=049575f2-6d5c-49c3-872c-56ad0f34ff5a (name=default) with tier type = 2022-09-13T07:24:49.469 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.466+0000 7fd8f8fa9600 20 add_watcher() i=0 2022-09-13T07:24:49.470 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.466+0000 7fd8f8fa9600 20 add_watcher() i=1 2022-09-13T07:24:49.470 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.466+0000 7fd8f8fa9600 20 add_watcher() i=2 2022-09-13T07:24:49.470 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.466+0000 7fd8f8fa9600 20 add_watcher() i=3 2022-09-13T07:24:49.471 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.466+0000 7fd8f8fa9600 20 add_watcher() i=4 2022-09-13T07:24:49.471 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.466+0000 7fd8f8fa9600 20 add_watcher() i=5 2022-09-13T07:24:49.471 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.466+0000 7fd8f8fa9600 20 add_watcher() i=6 2022-09-13T07:24:49.473 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.467+0000 7fd8f8fa9600 20 add_watcher() i=7 2022-09-13T07:24:49.473 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.467+0000 7fd8f8fa9600 2 all 8 watchers are set, enabling cache 2022-09-13T07:24:49.473 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.467+0000 7fd8f8fa9600 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2022-09-13T07:24:49.474 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.467+0000 7fd8f8fa9600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2022-09-13T07:24:49.474 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.467+0000 7fd8f8fa9600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-13T07:24:49.474 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.467+0000 7fd8f8fa9600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2022-09-13T07:24:49.474 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.467+0000 7fd8f8fa9600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2022-09-13T07:24:49.475 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.467+0000 7fd8f8fa9600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-13T07:24:49.475 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.468+0000 7fd8f8fa9600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2022-09-13T07:24:49.475 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.468+0000 7fd8f8fa9600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2022-09-13T07:24:49.475 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.468+0000 7fd8f8fa9600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-13T07:24:49.476 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.468+0000 7fd8f8fa9600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2022-09-13T07:24:49.476 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.468+0000 7fd8f8fa9600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2022-09-13T07:24:49.476 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.468+0000 7fd8f8fa9600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-13T07:24:49.476 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.468+0000 7fd8f8fa9600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2022-09-13T07:24:49.477 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.468+0000 7fd8f8fa9600 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2022-09-13T07:24:49.477 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.469+0000 7fd8f8fa9600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-13T07:24:49.477 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.474+0000 7fd8e172a700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-13T07:24:49.478 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.474+0000 7fd8f8fa9600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-13T07:24:49.478 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:49.474+0000 7fd8f8fa9600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-13T07:24:52.449 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.447+0000 7fd8f8fa9600 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2022-09-13T07:24:52.450 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.447+0000 7fd8f8fa9600 5 note: GC not initialized 2022-09-13T07:24:52.451 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.449+0000 7fd8f8fa9600 20 init_complete bucket index max shards: 11 2022-09-13T07:24:52.451 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.449+0000 7fd8f8fa9600 10 rgw notify: Started notification manager with: 1 workers 2022-09-13T07:24:52.452 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.449+0000 7fd8dcf21700 20 reqs_thread_entry: start 2022-09-13T07:24:52.452 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.449+0000 7fd8daf1d700 20 rgw notify: INFO: next queues processing will happen at: Tue Sep 13 07:25:22 2022 2022-09-13T07:24:52.452 INFO:teuthology.orchestra.run.smithi036.stderr: 2022-09-13T07:24:52.453 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.449+0000 7fd8da71c700 20 reqs_thread_entry: start 2022-09-13T07:24:52.473 INFO:teuthology.orchestra.run.smithi036.stdout:[] 2022-09-13T07:24:52.473 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.470+0000 7fd8f8fa9600 20 remove_watcher() i=0 2022-09-13T07:24:52.473 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.470+0000 7fd8f8fa9600 2 removed watcher, disabling cache 2022-09-13T07:24:52.474 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.471+0000 7fd8f8fa9600 20 remove_watcher() i=1 2022-09-13T07:24:52.476 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.474+0000 7fd8f8fa9600 20 remove_watcher() i=2 2022-09-13T07:24:52.479 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.476+0000 7fd8f8fa9600 20 remove_watcher() i=3 2022-09-13T07:24:52.481 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.479+0000 7fd8f8fa9600 20 remove_watcher() i=4 2022-09-13T07:24:52.482 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.480+0000 7fd8f8fa9600 20 remove_watcher() i=5 2022-09-13T07:24:52.484 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.482+0000 7fd8f8fa9600 20 remove_watcher() i=6 2022-09-13T07:24:52.486 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.484+0000 7fd8f8fa9600 20 remove_watcher() i=7 2022-09-13T07:24:52.487 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.485+0000 7fd8e172a700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-13T07:24:52.492 DEBUG:tasks.util.rgw: json result: [] 2022-09-13T07:24:52.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-13T07:24:52.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-13T07:24:52.494 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-13T07:24:52.549 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:24:52.550 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:24:52.556 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.553+0000 7fd290826600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-13T07:24:52.556 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.553+0000 7fd290826600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-13T07:24:52.560 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.557+0000 7fd2757a0700 20 reqs_thread_entry: start 2022-09-13T07:24:52.560 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.557+0000 7fd290826600 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-13T07:24:52.561 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.557+0000 7fd290826600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.562 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.558+0000 7fd290826600 20 rados_obj.operate() r=0 bl.length=46 2022-09-13T07:24:52.563 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.558+0000 7fd290826600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.563 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.558+0000 7fd290826600 20 rados_obj.operate() r=0 bl.length=873 2022-09-13T07:24:52.570 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.567+0000 7fd290826600 20 RGWRados::pool_iterate: got zonegroup_info.6ccbf938-a605-4e5e-9e3f-0b3c486c3422 2022-09-13T07:24:52.570 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.567+0000 7fd290826600 20 RGWRados::pool_iterate: got zone_info.049575f2-6d5c-49c3-872c-56ad0f34ff5a 2022-09-13T07:24:52.571 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.567+0000 7fd290826600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-13T07:24:52.571 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.567+0000 7fd290826600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-13T07:24:52.571 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.567+0000 7fd290826600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.572 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.567+0000 7fd290826600 20 rados_obj.operate() r=0 bl.length=873 2022-09-13T07:24:52.572 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.567+0000 7fd290826600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.572 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.567+0000 7fd290826600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:52.573 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.567+0000 7fd290826600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.573 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.568+0000 7fd290826600 20 rados_obj.operate() r=0 bl.length=46 2022-09-13T07:24:52.574 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.568+0000 7fd290826600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.574 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.568+0000 7fd290826600 20 rados_obj.operate() r=0 bl.length=398 2022-09-13T07:24:52.574 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:24:52.575 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "049575f2-6d5c-49c3-872c-56ad0f34ff5a", 2022-09-13T07:24:52.575 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "default", 2022-09-13T07:24:52.575 INFO:teuthology.orchestra.run.smithi036.stdout: "domain_root": "default.rgw.meta:root", 2022-09-13T07:24:52.576 INFO:teuthology.orchestra.run.smithi036.stdout: "control_pool": "default.rgw.control", 2022-09-13T07:24:52.576 INFO:teuthology.orchestra.run.smithi036.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-13T07:24:52.576 INFO:teuthology.orchestra.run.smithi036.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-13T07:24:52.576 INFO:teuthology.orchestra.run.smithi036.stdout: "log_pool": "default.rgw.log", 2022-09-13T07:24:52.577 INFO:teuthology.orchestra.run.smithi036.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-13T07:24:52.577 INFO:teuthology.orchestra.run.smithi036.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-13T07:24:52.577 INFO:teuthology.orchestra.run.smithi036.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-13T07:24:52.577 INFO:teuthology.orchestra.run.smithi036.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-13T07:24:52.577 INFO:teuthology.orchestra.run.smithi036.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-13T07:24:52.578 INFO:teuthology.orchestra.run.smithi036.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-13T07:24:52.578 INFO:teuthology.orchestra.run.smithi036.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-13T07:24:52.578 INFO:teuthology.orchestra.run.smithi036.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-13T07:24:52.578 INFO:teuthology.orchestra.run.smithi036.stdout: "otp_pool": "default.rgw.otp", 2022-09-13T07:24:52.579 INFO:teuthology.orchestra.run.smithi036.stdout: "system_key": { 2022-09-13T07:24:52.579 INFO:teuthology.orchestra.run.smithi036.stdout: "access_key": "", 2022-09-13T07:24:52.579 INFO:teuthology.orchestra.run.smithi036.stdout: "secret_key": "" 2022-09-13T07:24:52.579 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:24:52.580 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_pools": [ 2022-09-13T07:24:52.580 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:24:52.580 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "default-placement", 2022-09-13T07:24:52.580 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:24:52.581 INFO:teuthology.orchestra.run.smithi036.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-13T07:24:52.581 INFO:teuthology.orchestra.run.smithi036.stdout: "storage_classes": { 2022-09-13T07:24:52.581 INFO:teuthology.orchestra.run.smithi036.stdout: "STANDARD": { 2022-09-13T07:24:52.581 INFO:teuthology.orchestra.run.smithi036.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-13T07:24:52.582 INFO:teuthology.orchestra.run.smithi036.stdout: "compression_type": "random" 2022-09-13T07:24:52.582 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:24:52.582 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:24:52.582 INFO:teuthology.orchestra.run.smithi036.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-13T07:24:52.583 INFO:teuthology.orchestra.run.smithi036.stdout: "index_type": 0 2022-09-13T07:24:52.583 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:24:52.583 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:24:52.583 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:24:52.584 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "", 2022-09-13T07:24:52.584 INFO:teuthology.orchestra.run.smithi036.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-13T07:24:52.584 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:24:52.585 DEBUG:tasks.util.rgw: json result: {'id': '049575f2-6d5c-49c3-872c-56ad0f34ff5a', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-13T07:24:52.586 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2022-09-13T07:24:52.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2022-09-13T07:24:52.587 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2022-09-13T07:24:52.633 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:24:52.633 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:24:52.638 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.636+0000 7f102b9f3600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.640 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.637+0000 7f102b9f3600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:52.640 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.637+0000 7f102b9f3600 20 realm 2022-09-13T07:24:52.640 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.637+0000 7f102b9f3600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.641 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.637+0000 7f102b9f3600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:52.641 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.637+0000 7f102b9f3600 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-13T07:24:52.641 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.637+0000 7f102b9f3600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.641 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.638+0000 7f102b9f3600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:52.642 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.638+0000 7f102b9f3600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.642 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.638+0000 7f102b9f3600 20 rados_obj.operate() r=0 bl.length=46 2022-09-13T07:24:52.642 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.638+0000 7f102b9f3600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.643 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.638+0000 7f102b9f3600 20 rados_obj.operate() r=0 bl.length=889 2022-09-13T07:24:52.644 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.638+0000 7f102b9f3600 20 searching for the correct realm 2022-09-13T07:24:52.651 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 20 RGWRados::pool_iterate: got zonegroup_info.6ccbf938-a605-4e5e-9e3f-0b3c486c3422 2022-09-13T07:24:52.651 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 20 RGWRados::pool_iterate: got zone_info.049575f2-6d5c-49c3-872c-56ad0f34ff5a 2022-09-13T07:24:52.652 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-13T07:24:52.652 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-13T07:24:52.652 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.653 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:52.653 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.653 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 20 rados_obj.operate() r=0 bl.length=46 2022-09-13T07:24:52.653 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.653 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 20 rados_obj.operate() r=0 bl.length=398 2022-09-13T07:24:52.654 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 20 zone default found 2022-09-13T07:24:52.654 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 4 Realm: () 2022-09-13T07:24:52.654 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 4 ZoneGroup: default (6ccbf938-a605-4e5e-9e3f-0b3c486c3422) 2022-09-13T07:24:52.654 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 4 Zone: default (049575f2-6d5c-49c3-872c-56ad0f34ff5a) 2022-09-13T07:24:52.655 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-13T07:24:52.655 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 20 zonegroup default 2022-09-13T07:24:52.655 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.646+0000 7f102b9f3600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.655 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.647+0000 7f102b9f3600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:52.656 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.647+0000 7f102b9f3600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.656 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.647+0000 7f102b9f3600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:52.656 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.647+0000 7f102b9f3600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.657 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.647+0000 7f102b9f3600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:52.657 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.647+0000 7f102b9f3600 20 started sync module instance, tier type = 2022-09-13T07:24:52.657 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.647+0000 7f102b9f3600 20 started zone id=049575f2-6d5c-49c3-872c-56ad0f34ff5a (name=default) with tier type = 2022-09-13T07:24:52.660 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.657+0000 7f102b9f3600 20 add_watcher() i=0 2022-09-13T07:24:52.660 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.657+0000 7f102b9f3600 20 add_watcher() i=1 2022-09-13T07:24:52.660 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.657+0000 7f102b9f3600 20 add_watcher() i=2 2022-09-13T07:24:52.660 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.657+0000 7f102b9f3600 20 add_watcher() i=3 2022-09-13T07:24:52.661 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.657+0000 7f102b9f3600 20 add_watcher() i=4 2022-09-13T07:24:52.661 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.657+0000 7f102b9f3600 20 add_watcher() i=5 2022-09-13T07:24:52.661 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.657+0000 7f102b9f3600 20 add_watcher() i=6 2022-09-13T07:24:52.662 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.658+0000 7f102b9f3600 20 add_watcher() i=7 2022-09-13T07:24:52.663 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.658+0000 7f102b9f3600 2 all 8 watchers are set, enabling cache 2022-09-13T07:24:52.663 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.659+0000 7f1014174700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-13T07:24:52.663 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.659+0000 7f102b9f3600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-13T07:24:52.663 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.659+0000 7f102b9f3600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-13T07:24:52.665 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.663+0000 7f102b9f3600 5 note: GC not initialized 2022-09-13T07:24:52.667 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.664+0000 7f100f96b700 20 reqs_thread_entry: start 2022-09-13T07:24:52.667 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.664+0000 7f102b9f3600 20 init_complete bucket index max shards: 11 2022-09-13T07:24:52.667 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.664+0000 7f102b9f3600 10 rgw notify: Started notification manager with: 1 workers 2022-09-13T07:24:52.668 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.664+0000 7f100d967700 20 rgw notify: INFO: next queues processing will happen at: Tue Sep 13 07:25:23 2022 2022-09-13T07:24:52.668 INFO:teuthology.orchestra.run.smithi036.stderr: 2022-09-13T07:24:52.668 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.664+0000 7f100d145700 20 reqs_thread_entry: start 2022-09-13T07:24:52.674 INFO:teuthology.orchestra.run.smithi036.stdout:[] 2022-09-13T07:24:52.675 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.672+0000 7f102b9f3600 20 remove_watcher() i=0 2022-09-13T07:24:52.675 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.672+0000 7f102b9f3600 2 removed watcher, disabling cache 2022-09-13T07:24:52.675 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.673+0000 7f102b9f3600 20 remove_watcher() i=1 2022-09-13T07:24:52.676 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.674+0000 7f102b9f3600 20 remove_watcher() i=2 2022-09-13T07:24:52.678 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.674+0000 7f102b9f3600 20 remove_watcher() i=3 2022-09-13T07:24:52.678 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.675+0000 7f102b9f3600 20 remove_watcher() i=4 2022-09-13T07:24:52.679 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.676+0000 7f102b9f3600 20 remove_watcher() i=5 2022-09-13T07:24:52.680 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.677+0000 7f102b9f3600 20 remove_watcher() i=6 2022-09-13T07:24:52.680 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.678+0000 7f102b9f3600 20 remove_watcher() i=7 2022-09-13T07:24:52.681 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.678+0000 7f1014174700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-13T07:24:52.686 DEBUG:tasks.util.rgw: json result: [] 2022-09-13T07:24:52.687 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-13T07:24:52.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-13T07:24:52.688 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-13T07:24:52.746 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:24:52.747 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:24:52.747 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.743+0000 7f6d99bc4600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-13T07:24:52.747 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.743+0000 7f6d99bc4600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-13T07:24:52.749 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.747+0000 7f6d7eb3e700 20 reqs_thread_entry: start 2022-09-13T07:24:52.750 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.747+0000 7f6d99bc4600 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-13T07:24:52.750 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.747+0000 7f6d99bc4600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.751 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.748+0000 7f6d99bc4600 20 rados_obj.operate() r=0 bl.length=46 2022-09-13T07:24:52.751 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.748+0000 7f6d99bc4600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.751 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.748+0000 7f6d99bc4600 20 rados_obj.operate() r=0 bl.length=889 2022-09-13T07:24:52.759 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.756+0000 7f6d99bc4600 20 RGWRados::pool_iterate: got zonegroup_info.6ccbf938-a605-4e5e-9e3f-0b3c486c3422 2022-09-13T07:24:52.759 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.756+0000 7f6d99bc4600 20 RGWRados::pool_iterate: got zone_info.049575f2-6d5c-49c3-872c-56ad0f34ff5a 2022-09-13T07:24:52.760 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.756+0000 7f6d99bc4600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-13T07:24:52.760 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.756+0000 7f6d99bc4600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-13T07:24:52.761 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.756+0000 7f6d99bc4600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.764 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.756+0000 7f6d99bc4600 20 rados_obj.operate() r=0 bl.length=889 2022-09-13T07:24:52.764 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.756+0000 7f6d99bc4600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.765 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.756+0000 7f6d99bc4600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:52.765 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.756+0000 7f6d99bc4600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.766 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.756+0000 7f6d99bc4600 20 rados_obj.operate() r=0 bl.length=46 2022-09-13T07:24:52.766 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.756+0000 7f6d99bc4600 20 rados->read ofs=0 len=0 2022-09-13T07:24:52.766 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:24:52.757+0000 7f6d99bc4600 20 rados_obj.operate() r=0 bl.length=398 2022-09-13T07:24:52.768 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:24:52.769 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "049575f2-6d5c-49c3-872c-56ad0f34ff5a", 2022-09-13T07:24:52.769 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "default", 2022-09-13T07:24:52.770 INFO:teuthology.orchestra.run.smithi036.stdout: "domain_root": "default.rgw.meta:root", 2022-09-13T07:24:52.770 INFO:teuthology.orchestra.run.smithi036.stdout: "control_pool": "default.rgw.control", 2022-09-13T07:24:52.771 INFO:teuthology.orchestra.run.smithi036.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-13T07:24:52.771 INFO:teuthology.orchestra.run.smithi036.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-13T07:24:52.772 INFO:teuthology.orchestra.run.smithi036.stdout: "log_pool": "default.rgw.log", 2022-09-13T07:24:52.772 INFO:teuthology.orchestra.run.smithi036.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-13T07:24:52.772 INFO:teuthology.orchestra.run.smithi036.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-13T07:24:52.773 INFO:teuthology.orchestra.run.smithi036.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-13T07:24:52.773 INFO:teuthology.orchestra.run.smithi036.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-13T07:24:52.773 INFO:teuthology.orchestra.run.smithi036.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-13T07:24:52.773 INFO:teuthology.orchestra.run.smithi036.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-13T07:24:52.774 INFO:teuthology.orchestra.run.smithi036.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-13T07:24:52.774 INFO:teuthology.orchestra.run.smithi036.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-13T07:24:52.774 INFO:teuthology.orchestra.run.smithi036.stdout: "otp_pool": "default.rgw.otp", 2022-09-13T07:24:52.774 INFO:teuthology.orchestra.run.smithi036.stdout: "system_key": { 2022-09-13T07:24:52.775 INFO:teuthology.orchestra.run.smithi036.stdout: "access_key": "", 2022-09-13T07:24:52.775 INFO:teuthology.orchestra.run.smithi036.stdout: "secret_key": "" 2022-09-13T07:24:52.775 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:24:52.775 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_pools": [ 2022-09-13T07:24:52.775 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:24:52.776 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "default-placement", 2022-09-13T07:24:52.776 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:24:52.776 INFO:teuthology.orchestra.run.smithi036.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-13T07:24:52.777 INFO:teuthology.orchestra.run.smithi036.stdout: "storage_classes": { 2022-09-13T07:24:52.777 INFO:teuthology.orchestra.run.smithi036.stdout: "STANDARD": { 2022-09-13T07:24:52.778 INFO:teuthology.orchestra.run.smithi036.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-13T07:24:52.778 INFO:teuthology.orchestra.run.smithi036.stdout: "compression_type": "random" 2022-09-13T07:24:52.778 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:24:52.779 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:24:52.779 INFO:teuthology.orchestra.run.smithi036.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-13T07:24:52.779 INFO:teuthology.orchestra.run.smithi036.stdout: "index_type": 0 2022-09-13T07:24:52.779 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:24:52.779 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:24:52.780 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:24:52.780 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "", 2022-09-13T07:24:52.780 INFO:teuthology.orchestra.run.smithi036.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-13T07:24:52.780 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:24:52.781 DEBUG:tasks.util.rgw: json result: {'id': '049575f2-6d5c-49c3-872c-56ad0f34ff5a', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-13T07:24:52.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2022-09-13T07:24:52.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2022-09-13T07:24:52.782 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2022-09-13T07:24:52.857 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:24:52.858 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:24:52.864 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:52.860+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:54.484 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.481+0000 7f2fe681f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:54.484 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.481+0000 7f2fe681f600 20 realm 2022-09-13T07:24:54.484 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.481+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:54.485 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.481+0000 7f2fe681f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:54.485 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.481+0000 7f2fe681f600 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-13T07:24:54.485 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.481+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:54.485 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.482+0000 7f2fe681f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:54.486 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.482+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:54.487 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.484+0000 7f2fe681f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:54.487 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.485+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:54.487 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.485+0000 7f2fe681f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:54.488 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.485+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:54.488 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.485+0000 7f2fe681f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:54.489 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.485+0000 7f2fe681f600 10 Creating default zonegroup 2022-09-13T07:24:54.489 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.486+0000 7f2fe681f600 10 couldn't find old data placement pools config, setting up new ones for the zone 2022-09-13T07:24:54.498 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.495+0000 7f2fe681f600 10 WARNING: store->list_zones() returned r=-2 2022-09-13T07:24:54.498 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.495+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:54.498 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.495+0000 7f2fe681f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:54.501 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.499+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:54.502 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.499+0000 7f2fe681f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:54.502 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.499+0000 7f2fe681f600 10 could not read realm id: (2) No such file or directory 2022-09-13T07:24:54.502 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.499+0000 7f2fe681f600 10 WARNING: failed to set zone as default, r=-22 2022-09-13T07:24:54.502 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.499+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:54.503 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.499+0000 7f2fe681f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.504+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:54.508 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.505+0000 7f2fe681f600 20 rados_obj.operate() r=0 bl.length=873 2022-09-13T07:24:54.508 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.505+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.505+0000 7f2fe681f600 20 rados_obj.operate() r=0 bl.length=398 2022-09-13T07:24:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.505+0000 7f2fe681f600 10 Using default name default 2022-09-13T07:24:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.505+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.505+0000 7f2fe681f600 20 rados_obj.operate() r=0 bl.length=46 2022-09-13T07:24:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.505+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.506+0000 7f2fe681f600 20 rados_obj.operate() r=0 bl.length=873 2022-09-13T07:24:54.511 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.506+0000 7f2fe681f600 20 zone default found 2022-09-13T07:24:54.511 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.506+0000 7f2fe681f600 4 Realm: () 2022-09-13T07:24:54.512 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.506+0000 7f2fe681f600 4 ZoneGroup: default (0819700f-3880-4794-a461-f88f21918b1d) 2022-09-13T07:24:54.512 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.506+0000 7f2fe681f600 4 Zone: default (f69f74ab-2a4c-4150-a156-3249f5c9b455) 2022-09-13T07:24:54.512 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.506+0000 7f2fe681f600 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-13T07:24:54.513 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.506+0000 7f2fe681f600 20 zonegroup default 2022-09-13T07:24:54.513 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.506+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:54.513 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.506+0000 7f2fe681f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:54.506+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:56.494 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:56.492+0000 7f2fe681f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:56.495 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:56.492+0000 7f2fe681f600 20 rados->read ofs=0 len=0 2022-09-13T07:24:56.495 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:56.492+0000 7f2fe681f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:24:56.495 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:56.492+0000 7f2fe681f600 20 started sync module instance, tier type = 2022-09-13T07:24:56.495 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:56.492+0000 7f2fe681f600 20 started zone id=f69f74ab-2a4c-4150-a156-3249f5c9b455 (name=default) with tier type = 2022-09-13T07:24:58.523 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.520+0000 7f2fe681f600 20 add_watcher() i=0 2022-09-13T07:24:58.523 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.520+0000 7f2fe681f600 20 add_watcher() i=1 2022-09-13T07:24:58.523 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.520+0000 7f2fe681f600 20 add_watcher() i=2 2022-09-13T07:24:58.524 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.520+0000 7f2fe681f600 20 add_watcher() i=3 2022-09-13T07:24:58.524 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.520+0000 7f2fe681f600 20 add_watcher() i=4 2022-09-13T07:24:58.524 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.520+0000 7f2fe681f600 20 add_watcher() i=5 2022-09-13T07:24:58.524 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.520+0000 7f2fe681f600 20 add_watcher() i=6 2022-09-13T07:24:58.526 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.521+0000 7f2fe681f600 20 add_watcher() i=7 2022-09-13T07:24:58.526 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.521+0000 7f2fe681f600 2 all 8 watchers are set, enabling cache 2022-09-13T07:24:58.527 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.522+0000 7f2fe681f600 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2022-09-13T07:24:58.527 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.522+0000 7f2fe681f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2022-09-13T07:24:58.527 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.522+0000 7f2fe681f600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-13T07:24:58.527 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.522+0000 7f2fe681f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2022-09-13T07:24:58.528 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.522+0000 7f2fe681f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2022-09-13T07:24:58.528 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.522+0000 7f2fe681f600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-13T07:24:58.528 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.522+0000 7f2fe681f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2022-09-13T07:24:58.528 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.522+0000 7f2fe681f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2022-09-13T07:24:58.529 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.522+0000 7f2fe681f600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-13T07:24:58.529 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.522+0000 7f2fe681f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2022-09-13T07:24:58.529 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.522+0000 7f2fe681f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2022-09-13T07:24:58.529 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.522+0000 7f2fe681f600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-13T07:24:58.530 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.522+0000 7f2fe681f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2022-09-13T07:24:58.530 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.522+0000 7f2fe681f600 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2022-09-13T07:24:58.530 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.524+0000 7f2fe681f600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-13T07:24:58.532 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.530+0000 7f2fcefa0700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-13T07:24:58.533 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.530+0000 7f2fe681f600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-13T07:24:58.533 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:24:58.530+0000 7f2fe681f600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-13T07:25:01.484 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.482+0000 7f2fe681f600 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2022-09-13T07:25:01.484 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.482+0000 7f2fe681f600 5 note: GC not initialized 2022-09-13T07:25:01.486 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.483+0000 7f2fe681f600 20 init_complete bucket index max shards: 11 2022-09-13T07:25:01.486 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.483+0000 7f2fe681f600 10 rgw notify: Started notification manager with: 1 workers 2022-09-13T07:25:01.487 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.484+0000 7f2fc8793700 20 rgw notify: INFO: next queues processing will happen at: Tue Sep 13 07:25:31 2022 2022-09-13T07:25:01.488 INFO:teuthology.orchestra.run.smithi083.stderr: 2022-09-13T07:25:01.488 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.484+0000 7f2fc7f92700 20 reqs_thread_entry: start 2022-09-13T07:25:01.488 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.484+0000 7f2fca797700 20 reqs_thread_entry: start 2022-09-13T07:25:01.506 INFO:teuthology.orchestra.run.smithi083.stdout:[] 2022-09-13T07:25:01.507 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.504+0000 7f2fe681f600 20 remove_watcher() i=0 2022-09-13T07:25:01.507 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.504+0000 7f2fe681f600 2 removed watcher, disabling cache 2022-09-13T07:25:01.508 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.505+0000 7f2fe681f600 20 remove_watcher() i=1 2022-09-13T07:25:01.509 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.507+0000 7f2fe681f600 20 remove_watcher() i=2 2022-09-13T07:25:01.511 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.509+0000 7f2fe681f600 20 remove_watcher() i=3 2022-09-13T07:25:01.512 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.510+0000 7f2fe681f600 20 remove_watcher() i=4 2022-09-13T07:25:01.514 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.512+0000 7f2fe681f600 20 remove_watcher() i=5 2022-09-13T07:25:01.516 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.514+0000 7f2fe681f600 20 remove_watcher() i=6 2022-09-13T07:25:01.517 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.515+0000 7f2fe681f600 20 remove_watcher() i=7 2022-09-13T07:25:01.518 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.516+0000 7f2fcefa0700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-13T07:25:01.523 DEBUG:tasks.util.rgw: json result: [] 2022-09-13T07:25:01.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-13T07:25:01.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-13T07:25:01.524 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-13T07:25:01.577 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:01.578 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:01.582 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.580+0000 7fba68d00600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-13T07:25:01.582 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.580+0000 7fba68d00600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-13T07:25:01.586 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.584+0000 7fba4dc7a700 20 reqs_thread_entry: start 2022-09-13T07:25:01.586 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.584+0000 7fba68d00600 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-13T07:25:01.586 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.584+0000 7fba68d00600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.587 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.584+0000 7fba68d00600 20 rados_obj.operate() r=0 bl.length=46 2022-09-13T07:25:01.587 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.584+0000 7fba68d00600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.588 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.585+0000 7fba68d00600 20 rados_obj.operate() r=0 bl.length=873 2022-09-13T07:25:01.596 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.592+0000 7fba68d00600 20 RGWRados::pool_iterate: got zone_info.f69f74ab-2a4c-4150-a156-3249f5c9b455 2022-09-13T07:25:01.596 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.592+0000 7fba68d00600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-13T07:25:01.596 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.592+0000 7fba68d00600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-13T07:25:01.596 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.592+0000 7fba68d00600 20 RGWRados::pool_iterate: got zonegroup_info.0819700f-3880-4794-a461-f88f21918b1d 2022-09-13T07:25:01.597 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.592+0000 7fba68d00600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.597 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.593+0000 7fba68d00600 20 rados_obj.operate() r=0 bl.length=873 2022-09-13T07:25:01.597 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.593+0000 7fba68d00600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.597 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.593+0000 7fba68d00600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:25:01.597 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.593+0000 7fba68d00600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.598 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.593+0000 7fba68d00600 20 rados_obj.operate() r=0 bl.length=46 2022-09-13T07:25:01.598 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.593+0000 7fba68d00600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.598 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.593+0000 7fba68d00600 20 rados_obj.operate() r=0 bl.length=398 2022-09-13T07:25:01.599 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:25:01.599 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "f69f74ab-2a4c-4150-a156-3249f5c9b455", 2022-09-13T07:25:01.600 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "default", 2022-09-13T07:25:01.600 INFO:teuthology.orchestra.run.smithi083.stdout: "domain_root": "default.rgw.meta:root", 2022-09-13T07:25:01.600 INFO:teuthology.orchestra.run.smithi083.stdout: "control_pool": "default.rgw.control", 2022-09-13T07:25:01.600 INFO:teuthology.orchestra.run.smithi083.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-13T07:25:01.600 INFO:teuthology.orchestra.run.smithi083.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-13T07:25:01.601 INFO:teuthology.orchestra.run.smithi083.stdout: "log_pool": "default.rgw.log", 2022-09-13T07:25:01.601 INFO:teuthology.orchestra.run.smithi083.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-13T07:25:01.601 INFO:teuthology.orchestra.run.smithi083.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-13T07:25:01.601 INFO:teuthology.orchestra.run.smithi083.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-13T07:25:01.602 INFO:teuthology.orchestra.run.smithi083.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-13T07:25:01.602 INFO:teuthology.orchestra.run.smithi083.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-13T07:25:01.602 INFO:teuthology.orchestra.run.smithi083.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-13T07:25:01.602 INFO:teuthology.orchestra.run.smithi083.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-13T07:25:01.602 INFO:teuthology.orchestra.run.smithi083.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-13T07:25:01.603 INFO:teuthology.orchestra.run.smithi083.stdout: "otp_pool": "default.rgw.otp", 2022-09-13T07:25:01.603 INFO:teuthology.orchestra.run.smithi083.stdout: "system_key": { 2022-09-13T07:25:01.603 INFO:teuthology.orchestra.run.smithi083.stdout: "access_key": "", 2022-09-13T07:25:01.603 INFO:teuthology.orchestra.run.smithi083.stdout: "secret_key": "" 2022-09-13T07:25:01.604 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:01.604 INFO:teuthology.orchestra.run.smithi083.stdout: "placement_pools": [ 2022-09-13T07:25:01.604 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:01.604 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "default-placement", 2022-09-13T07:25:01.605 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:25:01.605 INFO:teuthology.orchestra.run.smithi083.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-13T07:25:01.605 INFO:teuthology.orchestra.run.smithi083.stdout: "storage_classes": { 2022-09-13T07:25:01.606 INFO:teuthology.orchestra.run.smithi083.stdout: "STANDARD": { 2022-09-13T07:25:01.606 INFO:teuthology.orchestra.run.smithi083.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-13T07:25:01.607 INFO:teuthology.orchestra.run.smithi083.stdout: "compression_type": "random" 2022-09-13T07:25:01.607 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:01.607 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:01.608 INFO:teuthology.orchestra.run.smithi083.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-13T07:25:01.608 INFO:teuthology.orchestra.run.smithi083.stdout: "index_type": 0 2022-09-13T07:25:01.608 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:01.609 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:01.609 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T07:25:01.610 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_id": "", 2022-09-13T07:25:01.610 INFO:teuthology.orchestra.run.smithi083.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-13T07:25:01.610 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:25:01.612 DEBUG:tasks.util.rgw: json result: {'id': 'f69f74ab-2a4c-4150-a156-3249f5c9b455', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-13T07:25:01.613 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2022-09-13T07:25:01.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2022-09-13T07:25:01.615 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2022-09-13T07:25:01.659 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:01.660 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:01.665 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.662+0000 7fb8a3276600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.667 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.663+0000 7fb8a3276600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:25:01.667 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.663+0000 7fb8a3276600 20 realm 2022-09-13T07:25:01.667 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.663+0000 7fb8a3276600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.667 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.663+0000 7fb8a3276600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:25:01.668 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.663+0000 7fb8a3276600 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-13T07:25:01.668 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.663+0000 7fb8a3276600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.668 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.664+0000 7fb8a3276600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:25:01.668 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.664+0000 7fb8a3276600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.669 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.664+0000 7fb8a3276600 20 rados_obj.operate() r=0 bl.length=46 2022-09-13T07:25:01.669 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.664+0000 7fb8a3276600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.669 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.664+0000 7fb8a3276600 20 rados_obj.operate() r=0 bl.length=889 2022-09-13T07:25:01.670 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.664+0000 7fb8a3276600 20 searching for the correct realm 2022-09-13T07:25:01.676 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.671+0000 7fb8a3276600 20 RGWRados::pool_iterate: got zone_info.f69f74ab-2a4c-4150-a156-3249f5c9b455 2022-09-13T07:25:01.677 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.671+0000 7fb8a3276600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-13T07:25:01.677 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.671+0000 7fb8a3276600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-13T07:25:01.677 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.671+0000 7fb8a3276600 20 RGWRados::pool_iterate: got zonegroup_info.0819700f-3880-4794-a461-f88f21918b1d 2022-09-13T07:25:01.677 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.671+0000 7fb8a3276600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.678 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.671+0000 7fb8a3276600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:25:01.678 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.671+0000 7fb8a3276600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.678 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.672+0000 7fb8a3276600 20 rados_obj.operate() r=0 bl.length=46 2022-09-13T07:25:01.678 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.672+0000 7fb8a3276600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.679 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.672+0000 7fb8a3276600 20 rados_obj.operate() r=0 bl.length=398 2022-09-13T07:25:01.679 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.672+0000 7fb8a3276600 20 zone default found 2022-09-13T07:25:01.679 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.672+0000 7fb8a3276600 4 Realm: () 2022-09-13T07:25:01.679 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.672+0000 7fb8a3276600 4 ZoneGroup: default (0819700f-3880-4794-a461-f88f21918b1d) 2022-09-13T07:25:01.680 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.672+0000 7fb8a3276600 4 Zone: default (f69f74ab-2a4c-4150-a156-3249f5c9b455) 2022-09-13T07:25:01.680 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.672+0000 7fb8a3276600 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-13T07:25:01.680 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.672+0000 7fb8a3276600 20 zonegroup default 2022-09-13T07:25:01.680 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.672+0000 7fb8a3276600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.681 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.672+0000 7fb8a3276600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:25:01.681 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.672+0000 7fb8a3276600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.681 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.672+0000 7fb8a3276600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:25:01.681 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.672+0000 7fb8a3276600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.682 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.672+0000 7fb8a3276600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:25:01.682 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.673+0000 7fb8a3276600 20 started sync module instance, tier type = 2022-09-13T07:25:01.682 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.673+0000 7fb8a3276600 20 started zone id=f69f74ab-2a4c-4150-a156-3249f5c9b455 (name=default) with tier type = 2022-09-13T07:25:01.686 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.683+0000 7fb8a3276600 20 add_watcher() i=0 2022-09-13T07:25:01.687 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.683+0000 7fb8a3276600 20 add_watcher() i=1 2022-09-13T07:25:01.687 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.683+0000 7fb8a3276600 20 add_watcher() i=2 2022-09-13T07:25:01.687 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.683+0000 7fb8a3276600 20 add_watcher() i=3 2022-09-13T07:25:01.688 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.683+0000 7fb8a3276600 20 add_watcher() i=4 2022-09-13T07:25:01.688 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.683+0000 7fb8a3276600 20 add_watcher() i=5 2022-09-13T07:25:01.689 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.683+0000 7fb8a3276600 20 add_watcher() i=6 2022-09-13T07:25:01.689 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.684+0000 7fb8a3276600 20 add_watcher() i=7 2022-09-13T07:25:01.690 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.684+0000 7fb8a3276600 2 all 8 watchers are set, enabling cache 2022-09-13T07:25:01.690 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.686+0000 7fb88b9f7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-13T07:25:01.691 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.686+0000 7fb8a3276600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-13T07:25:01.691 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.686+0000 7fb8a3276600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-13T07:25:01.692 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.689+0000 7fb8a3276600 5 note: GC not initialized 2022-09-13T07:25:01.693 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.690+0000 7fb8871ee700 20 reqs_thread_entry: start 2022-09-13T07:25:01.693 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.690+0000 7fb8a3276600 20 init_complete bucket index max shards: 11 2022-09-13T07:25:01.693 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.690+0000 7fb8a3276600 10 rgw notify: Started notification manager with: 1 workers 2022-09-13T07:25:01.693 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.690+0000 7fb8851ea700 20 rgw notify: INFO: next queues processing will happen at: Tue Sep 13 07:25:32 2022 2022-09-13T07:25:01.694 INFO:teuthology.orchestra.run.smithi083.stderr: 2022-09-13T07:25:01.694 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.690+0000 7fb8849c8700 20 reqs_thread_entry: start 2022-09-13T07:25:01.700 INFO:teuthology.orchestra.run.smithi083.stdout:[] 2022-09-13T07:25:01.702 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.698+0000 7fb8a3276600 20 remove_watcher() i=0 2022-09-13T07:25:01.702 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.698+0000 7fb8a3276600 2 removed watcher, disabling cache 2022-09-13T07:25:01.702 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.699+0000 7fb8a3276600 20 remove_watcher() i=1 2022-09-13T07:25:01.703 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.699+0000 7fb8a3276600 20 remove_watcher() i=2 2022-09-13T07:25:01.703 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.700+0000 7fb8a3276600 20 remove_watcher() i=3 2022-09-13T07:25:01.704 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.701+0000 7fb8a3276600 20 remove_watcher() i=4 2022-09-13T07:25:01.704 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.702+0000 7fb8a3276600 20 remove_watcher() i=5 2022-09-13T07:25:01.705 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.703+0000 7fb8a3276600 20 remove_watcher() i=6 2022-09-13T07:25:01.707 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.705+0000 7fb8a3276600 20 remove_watcher() i=7 2022-09-13T07:25:01.707 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.705+0000 7fb88b9f7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-13T07:25:01.713 DEBUG:tasks.util.rgw: json result: [] 2022-09-13T07:25:01.714 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-13T07:25:01.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-13T07:25:01.716 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-13T07:25:01.766 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:01.766 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:01.771 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.768+0000 7f71e9b56600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-13T07:25:01.771 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.768+0000 7f71e9b56600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-13T07:25:01.774 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.772+0000 7f71cead0700 20 reqs_thread_entry: start 2022-09-13T07:25:01.775 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.772+0000 7f71e9b56600 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-13T07:25:01.775 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.772+0000 7f71e9b56600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.776 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.773+0000 7f71e9b56600 20 rados_obj.operate() r=0 bl.length=46 2022-09-13T07:25:01.776 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.773+0000 7f71e9b56600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.777 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.773+0000 7f71e9b56600 20 rados_obj.operate() r=0 bl.length=889 2022-09-13T07:25:01.783 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.780+0000 7f71e9b56600 20 RGWRados::pool_iterate: got zone_info.f69f74ab-2a4c-4150-a156-3249f5c9b455 2022-09-13T07:25:01.783 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.780+0000 7f71e9b56600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-13T07:25:01.783 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.780+0000 7f71e9b56600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-13T07:25:01.784 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.780+0000 7f71e9b56600 20 RGWRados::pool_iterate: got zonegroup_info.0819700f-3880-4794-a461-f88f21918b1d 2022-09-13T07:25:01.784 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.780+0000 7f71e9b56600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.784 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.780+0000 7f71e9b56600 20 rados_obj.operate() r=0 bl.length=889 2022-09-13T07:25:01.784 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.780+0000 7f71e9b56600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.784 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.780+0000 7f71e9b56600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-13T07:25:01.785 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.780+0000 7f71e9b56600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.785 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.780+0000 7f71e9b56600 20 rados_obj.operate() r=0 bl.length=46 2022-09-13T07:25:01.785 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.780+0000 7f71e9b56600 20 rados->read ofs=0 len=0 2022-09-13T07:25:01.786 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:01.780+0000 7f71e9b56600 20 rados_obj.operate() r=0 bl.length=398 2022-09-13T07:25:01.786 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:25:01.786 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "f69f74ab-2a4c-4150-a156-3249f5c9b455", 2022-09-13T07:25:01.787 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "default", 2022-09-13T07:25:01.787 INFO:teuthology.orchestra.run.smithi083.stdout: "domain_root": "default.rgw.meta:root", 2022-09-13T07:25:01.787 INFO:teuthology.orchestra.run.smithi083.stdout: "control_pool": "default.rgw.control", 2022-09-13T07:25:01.787 INFO:teuthology.orchestra.run.smithi083.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-13T07:25:01.788 INFO:teuthology.orchestra.run.smithi083.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-13T07:25:01.788 INFO:teuthology.orchestra.run.smithi083.stdout: "log_pool": "default.rgw.log", 2022-09-13T07:25:01.788 INFO:teuthology.orchestra.run.smithi083.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-13T07:25:01.788 INFO:teuthology.orchestra.run.smithi083.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-13T07:25:01.789 INFO:teuthology.orchestra.run.smithi083.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-13T07:25:01.789 INFO:teuthology.orchestra.run.smithi083.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-13T07:25:01.789 INFO:teuthology.orchestra.run.smithi083.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-13T07:25:01.790 INFO:teuthology.orchestra.run.smithi083.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-13T07:25:01.790 INFO:teuthology.orchestra.run.smithi083.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-13T07:25:01.790 INFO:teuthology.orchestra.run.smithi083.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-13T07:25:01.790 INFO:teuthology.orchestra.run.smithi083.stdout: "otp_pool": "default.rgw.otp", 2022-09-13T07:25:01.791 INFO:teuthology.orchestra.run.smithi083.stdout: "system_key": { 2022-09-13T07:25:01.791 INFO:teuthology.orchestra.run.smithi083.stdout: "access_key": "", 2022-09-13T07:25:01.791 INFO:teuthology.orchestra.run.smithi083.stdout: "secret_key": "" 2022-09-13T07:25:01.791 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:01.791 INFO:teuthology.orchestra.run.smithi083.stdout: "placement_pools": [ 2022-09-13T07:25:01.792 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:01.792 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "default-placement", 2022-09-13T07:25:01.792 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:25:01.792 INFO:teuthology.orchestra.run.smithi083.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-13T07:25:01.793 INFO:teuthology.orchestra.run.smithi083.stdout: "storage_classes": { 2022-09-13T07:25:01.793 INFO:teuthology.orchestra.run.smithi083.stdout: "STANDARD": { 2022-09-13T07:25:01.793 INFO:teuthology.orchestra.run.smithi083.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-13T07:25:01.793 INFO:teuthology.orchestra.run.smithi083.stdout: "compression_type": "random" 2022-09-13T07:25:01.794 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:01.794 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:01.794 INFO:teuthology.orchestra.run.smithi083.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-13T07:25:01.794 INFO:teuthology.orchestra.run.smithi083.stdout: "index_type": 0 2022-09-13T07:25:01.794 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:01.795 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:01.795 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T07:25:01.795 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_id": "", 2022-09-13T07:25:01.795 INFO:teuthology.orchestra.run.smithi083.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-13T07:25:01.796 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:25:01.797 DEBUG:tasks.util.rgw: json result: {'id': 'f69f74ab-2a4c-4150-a156-3249f5c9b455', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-13T07:25:01.797 INFO:tasks.rgw:Starting rgw... 2022-09-13T07:25:01.798 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2022-09-13T07:25:01.798 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-13T07:25:01.799 INFO:tasks.rgw.c1.client.0:Restarting daemon 2022-09-13T07:25:01.799 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2022-09-13T07:25:01.802 INFO:tasks.rgw.c1.client.0:Started 2022-09-13T07:25:01.802 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2022-09-13T07:25:01.803 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-13T07:25:01.803 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-09-13T07:25:01.804 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-09-13T07:25:01.806 INFO:tasks.rgw.c1.client.1:Started 2022-09-13T07:25:01.806 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2022-09-13T07:25:01.807 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-13T07:25:01.807 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-09-13T07:25:01.808 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-09-13T07:25:01.810 INFO:tasks.rgw.c2.client.0:Started 2022-09-13T07:25:01.810 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2022-09-13T07:25:01.811 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-13T07:25:01.811 INFO:tasks.rgw.c2.client.1:Restarting daemon 2022-09-13T07:25:01.811 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2022-09-13T07:25:01.814 INFO:tasks.rgw.c2.client.1:Started 2022-09-13T07:25:01.814 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi036.front.sepia.ceph.com:8000/ 2022-09-13T07:25:01.815 DEBUG:teuthology.orchestra.run.smithi036:> curl http://smithi036.front.sepia.ceph.com:8000/ 2022-09-13T07:25:01.890 INFO:teuthology.orchestra.run.smithi036.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-13T07:25:01.890 INFO:teuthology.orchestra.run.smithi036.stderr: Dload Upload Total Spent Left Speed 2022-09-13T07:25:01.893 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-13T07:25:01.894 INFO:teuthology.orchestra.run.smithi036.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi036.front.sepia.ceph.com port 8000: Connection refused 2022-09-13T07:25:02.895 DEBUG:teuthology.orchestra.run.smithi036:> curl http://smithi036.front.sepia.ceph.com:8000/ 2022-09-13T07:25:02.917 INFO:teuthology.orchestra.run.smithi036.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-13T07:25:02.920 INFO:teuthology.orchestra.run.smithi036.stderr: Dload Upload Total Spent Left Speed 2022-09-13T07:25:02.921 INFO:teuthology.orchestra.run.smithi036.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-13T07:25:02.922 INFO:teuthology.orchestra.run.smithi036.stdout:anonymous 2022-09-13T07:25:02.922 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi036.front.sepia.ceph.com:8001/ 2022-09-13T07:25:02.922 DEBUG:teuthology.orchestra.run.smithi036:> curl http://smithi036.front.sepia.ceph.com:8001/ 2022-09-13T07:25:02.987 INFO:teuthology.orchestra.run.smithi036.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-13T07:25:02.987 INFO:teuthology.orchestra.run.smithi036.stderr: Dload Upload Total Spent Left Speed 2022-09-13T07:25:02.987 INFO:teuthology.orchestra.run.smithi036.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-13T07:25:02.989 INFO:teuthology.orchestra.run.smithi036.stdout:anonymous 2022-09-13T07:25:02.989 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi083.front.sepia.ceph.com:8000/ 2022-09-13T07:25:02.989 DEBUG:teuthology.orchestra.run.smithi083:> curl http://smithi083.front.sepia.ceph.com:8000/ 2022-09-13T07:25:03.029 INFO:teuthology.orchestra.run.smithi083.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-13T07:25:03.029 INFO:teuthology.orchestra.run.smithi083.stderr: Dload Upload Total Spent Left Speed 2022-09-13T07:25:03.030 INFO:teuthology.orchestra.run.smithi083.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-13T07:25:03.032 INFO:teuthology.orchestra.run.smithi083.stdout:anonymous 2022-09-13T07:25:03.032 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi083.front.sepia.ceph.com:8001/ 2022-09-13T07:25:03.033 DEBUG:teuthology.orchestra.run.smithi083:> curl http://smithi083.front.sepia.ceph.com:8001/ 2022-09-13T07:25:03.094 INFO:teuthology.orchestra.run.smithi083.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-13T07:25:03.094 INFO:teuthology.orchestra.run.smithi083.stderr: Dload Upload Total Spent Left Speed 2022-09-13T07:25:03.095 INFO:teuthology.orchestra.run.smithi083.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-13T07:25:03.096 INFO:teuthology.orchestra.run.smithi083.stdout:anonymous 2022-09-13T07:25:03.097 INFO:teuthology.run_tasks:Running task rgw-multisite... 2022-09-13T07:25:03.232 INFO:tasks.rgw_multisite:creating realm.. 2022-09-13T07:25:03.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:03.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:03.234 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T07:25:03.282 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:03.283 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:03.311 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:25:03.312 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T07:25:03.312 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-realm", 2022-09-13T07:25:03.312 INFO:teuthology.orchestra.run.smithi036.stdout: "current_period": "0f1e9404-34a5-47ad-8196-42a4e6e2ff9c", 2022-09-13T07:25:03.312 INFO:teuthology.orchestra.run.smithi036.stdout: "epoch": 1 2022-09-13T07:25:03.313 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:25:03.314 INFO:tasks.rgw_multisite:creating master zonegroup.. 2022-09-13T07:25:03.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:03.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:03.315 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T07:25:03.364 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:03.365 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:03.376 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:25:03.373+0000 7f515893a600 -1 old format 2022-09-13T07:25:03.381 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:25:03.381 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "test-zonegroup", 2022-09-13T07:25:03.382 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zonegroup", 2022-09-13T07:25:03.382 INFO:teuthology.orchestra.run.smithi036.stdout: "api_name": "", 2022-09-13T07:25:03.382 INFO:teuthology.orchestra.run.smithi036.stdout: "is_master": true, 2022-09-13T07:25:03.382 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [ 2022-09-13T07:25:03.383 INFO:teuthology.orchestra.run.smithi036.stdout: "http://smithi036.front.sepia.ceph.com:8000" 2022-09-13T07:25:03.383 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:03.383 INFO:teuthology.orchestra.run.smithi036.stdout: "hostnames": [], 2022-09-13T07:25:03.384 INFO:teuthology.orchestra.run.smithi036.stdout: "hostnames_s3website": [], 2022-09-13T07:25:03.384 INFO:teuthology.orchestra.run.smithi036.stdout: "master_zone": "", 2022-09-13T07:25:03.384 INFO:teuthology.orchestra.run.smithi036.stdout: "zones": [], 2022-09-13T07:25:03.384 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_targets": [], 2022-09-13T07:25:03.385 INFO:teuthology.orchestra.run.smithi036.stdout: "default_placement": "", 2022-09-13T07:25:03.385 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T07:25:03.385 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_policy": { 2022-09-13T07:25:03.385 INFO:teuthology.orchestra.run.smithi036.stdout: "groups": [] 2022-09-13T07:25:03.385 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:03.386 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled_features": [ 2022-09-13T07:25:03.386 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T07:25:03.386 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:25:03.386 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:25:03.388 INFO:tasks.rgw_multisite:creating master zone.. 2022-09-13T07:25:03.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'G693TB5XEJ3206QY', '--secret', 'QS5zddodwd26OlSDjoxox3W0BzgFaGzV', '--endpoints', 'http://smithi036.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:03.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'G693TB5XEJ3206QY', '--secret', 'QS5zddodwd26OlSDjoxox3W0BzgFaGzV', '--endpoints', 'http://smithi036.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:03.389 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key G693TB5XEJ3206QY --secret QS5zddodwd26OlSDjoxox3W0BzgFaGzV --endpoints http://smithi036.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T07:25:03.486 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:03.486 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:03.513 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:25:03.514 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T07:25:03.514 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zone1", 2022-09-13T07:25:03.514 INFO:teuthology.orchestra.run.smithi036.stdout: "domain_root": "test-zone1.rgw.meta:root", 2022-09-13T07:25:03.515 INFO:teuthology.orchestra.run.smithi036.stdout: "control_pool": "test-zone1.rgw.control", 2022-09-13T07:25:03.515 INFO:teuthology.orchestra.run.smithi036.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2022-09-13T07:25:03.515 INFO:teuthology.orchestra.run.smithi036.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2022-09-13T07:25:03.515 INFO:teuthology.orchestra.run.smithi036.stdout: "log_pool": "test-zone1.rgw.log", 2022-09-13T07:25:03.516 INFO:teuthology.orchestra.run.smithi036.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2022-09-13T07:25:03.516 INFO:teuthology.orchestra.run.smithi036.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2022-09-13T07:25:03.516 INFO:teuthology.orchestra.run.smithi036.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2022-09-13T07:25:03.516 INFO:teuthology.orchestra.run.smithi036.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2022-09-13T07:25:03.517 INFO:teuthology.orchestra.run.smithi036.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2022-09-13T07:25:03.517 INFO:teuthology.orchestra.run.smithi036.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2022-09-13T07:25:03.517 INFO:teuthology.orchestra.run.smithi036.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2022-09-13T07:25:03.517 INFO:teuthology.orchestra.run.smithi036.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2022-09-13T07:25:03.517 INFO:teuthology.orchestra.run.smithi036.stdout: "otp_pool": "test-zone1.rgw.otp", 2022-09-13T07:25:03.518 INFO:teuthology.orchestra.run.smithi036.stdout: "system_key": { 2022-09-13T07:25:03.518 INFO:teuthology.orchestra.run.smithi036.stdout: "access_key": "G693TB5XEJ3206QY", 2022-09-13T07:25:03.518 INFO:teuthology.orchestra.run.smithi036.stdout: "secret_key": "QS5zddodwd26OlSDjoxox3W0BzgFaGzV" 2022-09-13T07:25:03.518 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:03.519 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_pools": [ 2022-09-13T07:25:03.519 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:03.519 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "default-placement", 2022-09-13T07:25:03.519 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:25:03.520 INFO:teuthology.orchestra.run.smithi036.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2022-09-13T07:25:03.520 INFO:teuthology.orchestra.run.smithi036.stdout: "storage_classes": { 2022-09-13T07:25:03.520 INFO:teuthology.orchestra.run.smithi036.stdout: "STANDARD": { 2022-09-13T07:25:03.520 INFO:teuthology.orchestra.run.smithi036.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2022-09-13T07:25:03.521 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:03.521 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:03.521 INFO:teuthology.orchestra.run.smithi036.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2022-09-13T07:25:03.521 INFO:teuthology.orchestra.run.smithi036.stdout: "index_type": 0 2022-09-13T07:25:03.521 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:03.522 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:03.522 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:03.522 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T07:25:03.523 INFO:teuthology.orchestra.run.smithi036.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2022-09-13T07:25:03.523 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:25:03.524 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2022-09-13T07:25:04.486 INFO:teuthology.orchestra.run.smithi036.stderr:pool 'test-zone1.rgw.buckets.data' created 2022-09-13T07:25:04.500 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2022-09-13T07:25:05.500 INFO:teuthology.orchestra.run.smithi036.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2022-09-13T07:25:05.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:05.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:05.513 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T07:25:05.567 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:05.568 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:05.591 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:25:05.592 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T07:25:05.592 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zone1", 2022-09-13T07:25:05.592 INFO:teuthology.orchestra.run.smithi036.stdout: "domain_root": "test-zone1.rgw.meta:root", 2022-09-13T07:25:05.593 INFO:teuthology.orchestra.run.smithi036.stdout: "control_pool": "test-zone1.rgw.control", 2022-09-13T07:25:05.593 INFO:teuthology.orchestra.run.smithi036.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2022-09-13T07:25:05.593 INFO:teuthology.orchestra.run.smithi036.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2022-09-13T07:25:05.593 INFO:teuthology.orchestra.run.smithi036.stdout: "log_pool": "test-zone1.rgw.log", 2022-09-13T07:25:05.593 INFO:teuthology.orchestra.run.smithi036.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2022-09-13T07:25:05.594 INFO:teuthology.orchestra.run.smithi036.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2022-09-13T07:25:05.594 INFO:teuthology.orchestra.run.smithi036.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2022-09-13T07:25:05.594 INFO:teuthology.orchestra.run.smithi036.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2022-09-13T07:25:05.594 INFO:teuthology.orchestra.run.smithi036.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2022-09-13T07:25:05.595 INFO:teuthology.orchestra.run.smithi036.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2022-09-13T07:25:05.595 INFO:teuthology.orchestra.run.smithi036.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2022-09-13T07:25:05.595 INFO:teuthology.orchestra.run.smithi036.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2022-09-13T07:25:05.595 INFO:teuthology.orchestra.run.smithi036.stdout: "otp_pool": "test-zone1.rgw.otp", 2022-09-13T07:25:05.596 INFO:teuthology.orchestra.run.smithi036.stdout: "system_key": { 2022-09-13T07:25:05.596 INFO:teuthology.orchestra.run.smithi036.stdout: "access_key": "G693TB5XEJ3206QY", 2022-09-13T07:25:05.596 INFO:teuthology.orchestra.run.smithi036.stdout: "secret_key": "QS5zddodwd26OlSDjoxox3W0BzgFaGzV" 2022-09-13T07:25:05.596 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:05.596 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_pools": [ 2022-09-13T07:25:05.597 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:05.597 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "default-placement", 2022-09-13T07:25:05.597 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:25:05.597 INFO:teuthology.orchestra.run.smithi036.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2022-09-13T07:25:05.599 INFO:teuthology.orchestra.run.smithi036.stdout: "storage_classes": { 2022-09-13T07:25:05.599 INFO:teuthology.orchestra.run.smithi036.stdout: "STANDARD": { 2022-09-13T07:25:05.600 INFO:teuthology.orchestra.run.smithi036.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2022-09-13T07:25:05.600 INFO:teuthology.orchestra.run.smithi036.stdout: "compression_type": "random" 2022-09-13T07:25:05.600 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:05.600 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:05.601 INFO:teuthology.orchestra.run.smithi036.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2022-09-13T07:25:05.601 INFO:teuthology.orchestra.run.smithi036.stdout: "index_type": 0 2022-09-13T07:25:05.601 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:05.601 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:05.602 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:05.602 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "", 2022-09-13T07:25:05.602 INFO:teuthology.orchestra.run.smithi036.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2022-09-13T07:25:05.602 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:25:05.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:05.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:05.605 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T07:25:05.654 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:05.654 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:05.663 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:25:05.661+0000 7f3d390d0600 0 period (0f1e9404-34a5-47ad-8196-42a4e6e2ff9c does not have zone 7103daa4-1b45-47a1-bb49-5d9877f6d137 configured 2022-09-13T07:25:12.570 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:25:12.571 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:25:12.571 INFO:teuthology.orchestra.run.smithi036.stdout: "epoch": 1, 2022-09-13T07:25:12.571 INFO:teuthology.orchestra.run.smithi036.stdout: "predecessor_uuid": "0f1e9404-34a5-47ad-8196-42a4e6e2ff9c", 2022-09-13T07:25:12.572 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": [], 2022-09-13T07:25:12.572 INFO:teuthology.orchestra.run.smithi036.stdout: "period_map": { 2022-09-13T07:25:12.572 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:25:12.572 INFO:teuthology.orchestra.run.smithi036.stdout: "zonegroups": [ 2022-09-13T07:25:12.573 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:12.573 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "test-zonegroup", 2022-09-13T07:25:12.573 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zonegroup", 2022-09-13T07:25:12.573 INFO:teuthology.orchestra.run.smithi036.stdout: "api_name": "", 2022-09-13T07:25:12.573 INFO:teuthology.orchestra.run.smithi036.stdout: "is_master": true, 2022-09-13T07:25:12.574 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [ 2022-09-13T07:25:12.574 INFO:teuthology.orchestra.run.smithi036.stdout: "http://smithi036.front.sepia.ceph.com:8000" 2022-09-13T07:25:12.574 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:12.574 INFO:teuthology.orchestra.run.smithi036.stdout: "hostnames": [], 2022-09-13T07:25:12.575 INFO:teuthology.orchestra.run.smithi036.stdout: "hostnames_s3website": [], 2022-09-13T07:25:12.575 INFO:teuthology.orchestra.run.smithi036.stdout: "master_zone": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T07:25:12.575 INFO:teuthology.orchestra.run.smithi036.stdout: "zones": [ 2022-09-13T07:25:12.575 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:12.576 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T07:25:12.576 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zone1", 2022-09-13T07:25:12.576 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [ 2022-09-13T07:25:12.576 INFO:teuthology.orchestra.run.smithi036.stdout: "http://smithi036.front.sepia.ceph.com:8000" 2022-09-13T07:25:12.577 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:12.577 INFO:teuthology.orchestra.run.smithi036.stdout: "log_meta": false, 2022-09-13T07:25:12.577 INFO:teuthology.orchestra.run.smithi036.stdout: "log_data": false, 2022-09-13T07:25:12.578 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_index_max_shards": 11, 2022-09-13T07:25:12.578 INFO:teuthology.orchestra.run.smithi036.stdout: "read_only": false, 2022-09-13T07:25:12.578 INFO:teuthology.orchestra.run.smithi036.stdout: "tier_type": "", 2022-09-13T07:25:12.578 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from_all": true, 2022-09-13T07:25:12.578 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from": [], 2022-09-13T07:25:12.579 INFO:teuthology.orchestra.run.smithi036.stdout: "redirect_zone": "", 2022-09-13T07:25:12.579 INFO:teuthology.orchestra.run.smithi036.stdout: "supported_features": [ 2022-09-13T07:25:12.579 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T07:25:12.579 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:25:12.580 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:12.580 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:12.580 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_targets": [ 2022-09-13T07:25:12.580 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:12.580 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "default-placement", 2022-09-13T07:25:12.581 INFO:teuthology.orchestra.run.smithi036.stdout: "tags": [], 2022-09-13T07:25:12.581 INFO:teuthology.orchestra.run.smithi036.stdout: "storage_classes": [ 2022-09-13T07:25:12.581 INFO:teuthology.orchestra.run.smithi036.stdout: "STANDARD" 2022-09-13T07:25:12.581 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:25:12.582 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:12.582 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:12.582 INFO:teuthology.orchestra.run.smithi036.stdout: "default_placement": "default-placement", 2022-09-13T07:25:12.582 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T07:25:12.583 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_policy": { 2022-09-13T07:25:12.583 INFO:teuthology.orchestra.run.smithi036.stdout: "groups": [] 2022-09-13T07:25:12.583 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:12.583 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled_features": [ 2022-09-13T07:25:12.583 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T07:25:12.584 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:25:12.584 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:12.584 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:12.584 INFO:teuthology.orchestra.run.smithi036.stdout: "short_zone_ids": [ 2022-09-13T07:25:12.585 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:12.585 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T07:25:12.585 INFO:teuthology.orchestra.run.smithi036.stdout: "val": 2837352077 2022-09-13T07:25:12.585 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:12.586 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:25:12.586 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:12.586 INFO:teuthology.orchestra.run.smithi036.stdout: "master_zonegroup": "test-zonegroup", 2022-09-13T07:25:12.586 INFO:teuthology.orchestra.run.smithi036.stdout: "master_zone": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T07:25:12.587 INFO:teuthology.orchestra.run.smithi036.stdout: "period_config": { 2022-09-13T07:25:12.587 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_quota": { 2022-09-13T07:25:12.587 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false, 2022-09-13T07:25:12.587 INFO:teuthology.orchestra.run.smithi036.stdout: "check_on_raw": false, 2022-09-13T07:25:12.588 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size": -1, 2022-09-13T07:25:12.588 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size_kb": 0, 2022-09-13T07:25:12.588 INFO:teuthology.orchestra.run.smithi036.stdout: "max_objects": -1 2022-09-13T07:25:12.588 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:12.589 INFO:teuthology.orchestra.run.smithi036.stdout: "user_quota": { 2022-09-13T07:25:12.589 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false, 2022-09-13T07:25:12.589 INFO:teuthology.orchestra.run.smithi036.stdout: "check_on_raw": false, 2022-09-13T07:25:12.589 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size": -1, 2022-09-13T07:25:12.589 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size_kb": 0, 2022-09-13T07:25:12.590 INFO:teuthology.orchestra.run.smithi036.stdout: "max_objects": -1 2022-09-13T07:25:12.590 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:12.590 INFO:teuthology.orchestra.run.smithi036.stdout: "user_ratelimit": { 2022-09-13T07:25:12.590 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_ops": 0, 2022-09-13T07:25:12.591 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_ops": 0, 2022-09-13T07:25:12.591 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_bytes": 0, 2022-09-13T07:25:12.591 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_bytes": 0, 2022-09-13T07:25:12.591 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false 2022-09-13T07:25:12.591 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:12.592 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_ratelimit": { 2022-09-13T07:25:12.592 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_ops": 0, 2022-09-13T07:25:12.592 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_ops": 0, 2022-09-13T07:25:12.592 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_bytes": 0, 2022-09-13T07:25:12.593 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_bytes": 0, 2022-09-13T07:25:12.593 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false 2022-09-13T07:25:12.593 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:12.593 INFO:teuthology.orchestra.run.smithi036.stdout: "anonymous_ratelimit": { 2022-09-13T07:25:12.593 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_ops": 0, 2022-09-13T07:25:12.594 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_ops": 0, 2022-09-13T07:25:12.594 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_bytes": 0, 2022-09-13T07:25:12.594 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_bytes": 0, 2022-09-13T07:25:12.594 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false 2022-09-13T07:25:12.595 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:12.595 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:12.595 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T07:25:12.595 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_name": "test-realm", 2022-09-13T07:25:12.595 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:25:12.596 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:25:12.597 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2022-09-13T07:25:12.598 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T07:25:12.598 INFO:tasks.rgw.c1.client.0.smithi036.stdout:2022-09-13T07:25:12.596+0000 7fbf669d5700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 107586) UID: 0 2022-09-13T07:25:12.599 INFO:tasks.rgw.c1.client.0.smithi036.stdout:2022-09-13T07:25:12.596+0000 7fbf6ac40600 -1 shutting down 2022-09-13T07:25:16.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2022-09-13T07:25:18.700 INFO:tasks.rgw.c1.client.0:Stopped 2022-09-13T07:25:18.700 INFO:tasks.rgw.c1.client.0:Restarting daemon 2022-09-13T07:25:18.701 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2022-09-13T07:25:18.704 INFO:tasks.rgw.c1.client.0:Started 2022-09-13T07:25:18.704 DEBUG:teuthology.orchestra.run.smithi036:> curl http://smithi036.front.sepia.ceph.com:8000 2022-09-13T07:25:18.733 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-13T07:25:18.733 INFO:teuthology.orchestra.run.smithi036.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-13T07:25:18.734 INFO:teuthology.orchestra.run.smithi036.stderr: Dload Upload Total Spent Left Speed 2022-09-13T07:25:18.734 INFO:teuthology.orchestra.run.smithi036.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi036.front.sepia.ceph.com port 8000: Connection refused 2022-09-13T07:25:19.736 DEBUG:teuthology.orchestra.run.smithi036:> curl http://smithi036.front.sepia.ceph.com:8000 2022-09-13T07:25:19.763 INFO:teuthology.orchestra.run.smithi036.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-13T07:25:19.764 INFO:teuthology.orchestra.run.smithi036.stderr: Dload Upload Total Spent Left Speed 2022-09-13T07:25:19.764 INFO:teuthology.orchestra.run.smithi036.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-13T07:25:19.765 INFO:teuthology.orchestra.run.smithi036.stdout:anonymous 2022-09-13T07:25:19.766 INFO:tasks.rgw_multisite:creating admin user.. 2022-09-13T07:25:19.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'G693TB5XEJ3206QY', '--secret', 'QS5zddodwd26OlSDjoxox3W0BzgFaGzV', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:19.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'G693TB5XEJ3206QY', '--secret', 'QS5zddodwd26OlSDjoxox3W0BzgFaGzV', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:19.767 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key G693TB5XEJ3206QY --secret QS5zddodwd26OlSDjoxox3W0BzgFaGzV --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T07:25:19.862 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:19.863 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:19.909 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:25:19.910 INFO:teuthology.orchestra.run.smithi036.stdout: "user_id": "realm-admin", 2022-09-13T07:25:19.910 INFO:teuthology.orchestra.run.smithi036.stdout: "display_name": "Realm Admin", 2022-09-13T07:25:19.910 INFO:teuthology.orchestra.run.smithi036.stdout: "email": "", 2022-09-13T07:25:19.910 INFO:teuthology.orchestra.run.smithi036.stdout: "suspended": 0, 2022-09-13T07:25:19.911 INFO:teuthology.orchestra.run.smithi036.stdout: "max_buckets": 1000, 2022-09-13T07:25:19.911 INFO:teuthology.orchestra.run.smithi036.stdout: "subusers": [], 2022-09-13T07:25:19.911 INFO:teuthology.orchestra.run.smithi036.stdout: "keys": [ 2022-09-13T07:25:19.911 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:19.912 INFO:teuthology.orchestra.run.smithi036.stdout: "user": "realm-admin", 2022-09-13T07:25:19.912 INFO:teuthology.orchestra.run.smithi036.stdout: "access_key": "G693TB5XEJ3206QY", 2022-09-13T07:25:19.912 INFO:teuthology.orchestra.run.smithi036.stdout: "secret_key": "QS5zddodwd26OlSDjoxox3W0BzgFaGzV" 2022-09-13T07:25:19.912 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:19.913 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:19.913 INFO:teuthology.orchestra.run.smithi036.stdout: "swift_keys": [], 2022-09-13T07:25:19.913 INFO:teuthology.orchestra.run.smithi036.stdout: "caps": [], 2022-09-13T07:25:19.913 INFO:teuthology.orchestra.run.smithi036.stdout: "op_mask": "read, write, delete", 2022-09-13T07:25:19.914 INFO:teuthology.orchestra.run.smithi036.stdout: "system": true, 2022-09-13T07:25:19.914 INFO:teuthology.orchestra.run.smithi036.stdout: "default_placement": "", 2022-09-13T07:25:19.914 INFO:teuthology.orchestra.run.smithi036.stdout: "default_storage_class": "", 2022-09-13T07:25:19.914 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_tags": [], 2022-09-13T07:25:19.915 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_quota": { 2022-09-13T07:25:19.915 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false, 2022-09-13T07:25:19.915 INFO:teuthology.orchestra.run.smithi036.stdout: "check_on_raw": false, 2022-09-13T07:25:19.915 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size": -1, 2022-09-13T07:25:19.916 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size_kb": 0, 2022-09-13T07:25:19.916 INFO:teuthology.orchestra.run.smithi036.stdout: "max_objects": -1 2022-09-13T07:25:19.916 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:19.916 INFO:teuthology.orchestra.run.smithi036.stdout: "user_quota": { 2022-09-13T07:25:19.916 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false, 2022-09-13T07:25:19.917 INFO:teuthology.orchestra.run.smithi036.stdout: "check_on_raw": false, 2022-09-13T07:25:19.917 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size": -1, 2022-09-13T07:25:19.917 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size_kb": 0, 2022-09-13T07:25:19.917 INFO:teuthology.orchestra.run.smithi036.stdout: "max_objects": -1 2022-09-13T07:25:19.918 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:19.918 INFO:teuthology.orchestra.run.smithi036.stdout: "temp_url_keys": [], 2022-09-13T07:25:19.918 INFO:teuthology.orchestra.run.smithi036.stdout: "type": "rgw", 2022-09-13T07:25:19.918 INFO:teuthology.orchestra.run.smithi036.stdout: "mfa_ids": [] 2022-09-13T07:25:19.919 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:25:19.919 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:25:19.928 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2022-09-13T07:25:19.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi036.front.sepia.ceph.com:8000', '--access-key', 'G693TB5XEJ3206QY', '--secret', 'QS5zddodwd26OlSDjoxox3W0BzgFaGzV', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:19.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi036.front.sepia.ceph.com:8000', '--access-key', 'G693TB5XEJ3206QY', '--secret', 'QS5zddodwd26OlSDjoxox3W0BzgFaGzV', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:19.929 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi036.front.sepia.ceph.com:8000 --access-key G693TB5XEJ3206QY --secret QS5zddodwd26OlSDjoxox3W0BzgFaGzV --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-13T07:25:19.989 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:19.989 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:20.028 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:25:20.028 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T07:25:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "test-realm", 2022-09-13T07:25:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: "current_period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:25:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 2 2022-09-13T07:25:20.029 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:25:20.031 INFO:tasks.rgw_multisite:creating zone.. 2022-09-13T07:25:20.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'G693TB5XEJ3206QY', '--secret', 'QS5zddodwd26OlSDjoxox3W0BzgFaGzV', '--endpoints', 'http://smithi083.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:20.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'G693TB5XEJ3206QY', '--secret', 'QS5zddodwd26OlSDjoxox3W0BzgFaGzV', '--endpoints', 'http://smithi083.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:20.032 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key G693TB5XEJ3206QY --secret QS5zddodwd26OlSDjoxox3W0BzgFaGzV --endpoints http://smithi083.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-13T07:25:20.088 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:20.088 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:20.113 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:20.109+0000 7f7ee46d8600 0 failed reading obj info from .rgw.root:zone_info.7103daa4-1b45-47a1-bb49-5d9877f6d137: (2) No such file or directory 2022-09-13T07:25:20.113 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:20.109+0000 7f7ee46d8600 0 WARNING: could not read zone params for zone id=7103daa4-1b45-47a1-bb49-5d9877f6d137 name=test-zone1 2022-09-13T07:25:20.119 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:25:20.120 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T07:25:20.120 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "test-zone2", 2022-09-13T07:25:20.120 INFO:teuthology.orchestra.run.smithi083.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-13T07:25:20.120 INFO:teuthology.orchestra.run.smithi083.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-13T07:25:20.121 INFO:teuthology.orchestra.run.smithi083.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-13T07:25:20.121 INFO:teuthology.orchestra.run.smithi083.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-13T07:25:20.121 INFO:teuthology.orchestra.run.smithi083.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-13T07:25:20.121 INFO:teuthology.orchestra.run.smithi083.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-13T07:25:20.122 INFO:teuthology.orchestra.run.smithi083.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-13T07:25:20.122 INFO:teuthology.orchestra.run.smithi083.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-13T07:25:20.122 INFO:teuthology.orchestra.run.smithi083.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-13T07:25:20.122 INFO:teuthology.orchestra.run.smithi083.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-13T07:25:20.123 INFO:teuthology.orchestra.run.smithi083.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-13T07:25:20.123 INFO:teuthology.orchestra.run.smithi083.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-13T07:25:20.123 INFO:teuthology.orchestra.run.smithi083.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-13T07:25:20.123 INFO:teuthology.orchestra.run.smithi083.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-13T07:25:20.123 INFO:teuthology.orchestra.run.smithi083.stdout: "system_key": { 2022-09-13T07:25:20.124 INFO:teuthology.orchestra.run.smithi083.stdout: "access_key": "G693TB5XEJ3206QY", 2022-09-13T07:25:20.124 INFO:teuthology.orchestra.run.smithi083.stdout: "secret_key": "QS5zddodwd26OlSDjoxox3W0BzgFaGzV" 2022-09-13T07:25:20.124 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:20.124 INFO:teuthology.orchestra.run.smithi083.stdout: "placement_pools": [ 2022-09-13T07:25:20.125 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:20.125 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "default-placement", 2022-09-13T07:25:20.125 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:25:20.125 INFO:teuthology.orchestra.run.smithi083.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-13T07:25:20.126 INFO:teuthology.orchestra.run.smithi083.stdout: "storage_classes": { 2022-09-13T07:25:20.126 INFO:teuthology.orchestra.run.smithi083.stdout: "STANDARD": { 2022-09-13T07:25:20.126 INFO:teuthology.orchestra.run.smithi083.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2022-09-13T07:25:20.126 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:20.126 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:20.127 INFO:teuthology.orchestra.run.smithi083.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-13T07:25:20.127 INFO:teuthology.orchestra.run.smithi083.stdout: "index_type": 0 2022-09-13T07:25:20.127 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:20.127 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:20.128 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T07:25:20.128 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T07:25:20.128 INFO:teuthology.orchestra.run.smithi083.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-13T07:25:20.128 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:25:20.130 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2022-09-13T07:25:21.426 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'test-zone2.rgw.buckets.data' created 2022-09-13T07:25:21.447 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2022-09-13T07:25:21.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2022-09-13T07:25:22.454 INFO:teuthology.orchestra.run.smithi083.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2022-09-13T07:25:22.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:22.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:22.466 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-13T07:25:22.520 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:22.521 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:22.544 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:25:22.544 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T07:25:22.545 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "test-zone2", 2022-09-13T07:25:22.545 INFO:teuthology.orchestra.run.smithi083.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-13T07:25:22.545 INFO:teuthology.orchestra.run.smithi083.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-13T07:25:22.545 INFO:teuthology.orchestra.run.smithi083.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-13T07:25:22.546 INFO:teuthology.orchestra.run.smithi083.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-13T07:25:22.546 INFO:teuthology.orchestra.run.smithi083.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-13T07:25:22.546 INFO:teuthology.orchestra.run.smithi083.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-13T07:25:22.546 INFO:teuthology.orchestra.run.smithi083.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-13T07:25:22.547 INFO:teuthology.orchestra.run.smithi083.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-13T07:25:22.547 INFO:teuthology.orchestra.run.smithi083.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-13T07:25:22.547 INFO:teuthology.orchestra.run.smithi083.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-13T07:25:22.547 INFO:teuthology.orchestra.run.smithi083.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-13T07:25:22.548 INFO:teuthology.orchestra.run.smithi083.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-13T07:25:22.548 INFO:teuthology.orchestra.run.smithi083.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-13T07:25:22.548 INFO:teuthology.orchestra.run.smithi083.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-13T07:25:22.549 INFO:teuthology.orchestra.run.smithi083.stdout: "system_key": { 2022-09-13T07:25:22.549 INFO:teuthology.orchestra.run.smithi083.stdout: "access_key": "G693TB5XEJ3206QY", 2022-09-13T07:25:22.550 INFO:teuthology.orchestra.run.smithi083.stdout: "secret_key": "QS5zddodwd26OlSDjoxox3W0BzgFaGzV" 2022-09-13T07:25:22.550 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:22.550 INFO:teuthology.orchestra.run.smithi083.stdout: "placement_pools": [ 2022-09-13T07:25:22.551 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:22.551 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "default-placement", 2022-09-13T07:25:22.551 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:25:22.551 INFO:teuthology.orchestra.run.smithi083.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-13T07:25:22.552 INFO:teuthology.orchestra.run.smithi083.stdout: "storage_classes": { 2022-09-13T07:25:22.552 INFO:teuthology.orchestra.run.smithi083.stdout: "STANDARD": { 2022-09-13T07:25:22.552 INFO:teuthology.orchestra.run.smithi083.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-13T07:25:22.552 INFO:teuthology.orchestra.run.smithi083.stdout: "compression_type": "random" 2022-09-13T07:25:22.553 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:22.553 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:22.553 INFO:teuthology.orchestra.run.smithi083.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-13T07:25:22.553 INFO:teuthology.orchestra.run.smithi083.stdout: "index_type": 0 2022-09-13T07:25:22.554 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:22.554 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:22.554 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T07:25:22.555 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_id": "", 2022-09-13T07:25:22.555 INFO:teuthology.orchestra.run.smithi083.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-13T07:25:22.555 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:25:22.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:22.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:22.557 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-13T07:25:22.606 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:22.606 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:22.615 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:22.611+0000 7fbaba1bc600 0 period (3255cc65-e34b-4e83-a436-e06431413191 does not have zone 30ddd076-3f98-4411-9a5a-48d7fd04ad75 configured 2022-09-13T07:25:29.480 INFO:teuthology.orchestra.run.smithi083.stderr:Sending period to new master zone 7103daa4-1b45-47a1-bb49-5d9877f6d137 2022-09-13T07:25:29.511 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:25:29.511 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:25:29.512 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 2, 2022-09-13T07:25:29.512 INFO:teuthology.orchestra.run.smithi083.stdout: "predecessor_uuid": "0f1e9404-34a5-47ad-8196-42a4e6e2ff9c", 2022-09-13T07:25:29.512 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": [], 2022-09-13T07:25:29.512 INFO:teuthology.orchestra.run.smithi083.stdout: "period_map": { 2022-09-13T07:25:29.512 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:25:29.513 INFO:teuthology.orchestra.run.smithi083.stdout: "zonegroups": [ 2022-09-13T07:25:29.513 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:29.513 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "test-zonegroup", 2022-09-13T07:25:29.513 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "test-zonegroup", 2022-09-13T07:25:29.514 INFO:teuthology.orchestra.run.smithi083.stdout: "api_name": "", 2022-09-13T07:25:29.514 INFO:teuthology.orchestra.run.smithi083.stdout: "is_master": true, 2022-09-13T07:25:29.514 INFO:teuthology.orchestra.run.smithi083.stdout: "endpoints": [ 2022-09-13T07:25:29.514 INFO:teuthology.orchestra.run.smithi083.stdout: "http://smithi036.front.sepia.ceph.com:8000" 2022-09-13T07:25:29.514 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T07:25:29.515 INFO:teuthology.orchestra.run.smithi083.stdout: "hostnames": [], 2022-09-13T07:25:29.515 INFO:teuthology.orchestra.run.smithi083.stdout: "hostnames_s3website": [], 2022-09-13T07:25:29.515 INFO:teuthology.orchestra.run.smithi083.stdout: "master_zone": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T07:25:29.515 INFO:teuthology.orchestra.run.smithi083.stdout: "zones": [ 2022-09-13T07:25:29.516 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:29.516 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T07:25:29.516 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "test-zone2", 2022-09-13T07:25:29.516 INFO:teuthology.orchestra.run.smithi083.stdout: "endpoints": [ 2022-09-13T07:25:29.516 INFO:teuthology.orchestra.run.smithi083.stdout: "http://smithi083.front.sepia.ceph.com:8000" 2022-09-13T07:25:29.517 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T07:25:29.517 INFO:teuthology.orchestra.run.smithi083.stdout: "log_meta": false, 2022-09-13T07:25:29.517 INFO:teuthology.orchestra.run.smithi083.stdout: "log_data": true, 2022-09-13T07:25:29.517 INFO:teuthology.orchestra.run.smithi083.stdout: "bucket_index_max_shards": 11, 2022-09-13T07:25:29.518 INFO:teuthology.orchestra.run.smithi083.stdout: "read_only": false, 2022-09-13T07:25:29.518 INFO:teuthology.orchestra.run.smithi083.stdout: "tier_type": "", 2022-09-13T07:25:29.518 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_from_all": true, 2022-09-13T07:25:29.518 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_from": [], 2022-09-13T07:25:29.518 INFO:teuthology.orchestra.run.smithi083.stdout: "redirect_zone": "", 2022-09-13T07:25:29.519 INFO:teuthology.orchestra.run.smithi083.stdout: "supported_features": [ 2022-09-13T07:25:29.519 INFO:teuthology.orchestra.run.smithi083.stdout: "resharding" 2022-09-13T07:25:29.519 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:25:29.519 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:29.520 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:29.520 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T07:25:29.520 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "test-zone1", 2022-09-13T07:25:29.520 INFO:teuthology.orchestra.run.smithi083.stdout: "endpoints": [ 2022-09-13T07:25:29.520 INFO:teuthology.orchestra.run.smithi083.stdout: "http://smithi036.front.sepia.ceph.com:8000" 2022-09-13T07:25:29.521 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T07:25:29.521 INFO:teuthology.orchestra.run.smithi083.stdout: "log_meta": false, 2022-09-13T07:25:29.521 INFO:teuthology.orchestra.run.smithi083.stdout: "log_data": true, 2022-09-13T07:25:29.522 INFO:teuthology.orchestra.run.smithi083.stdout: "bucket_index_max_shards": 11, 2022-09-13T07:25:29.522 INFO:teuthology.orchestra.run.smithi083.stdout: "read_only": false, 2022-09-13T07:25:29.522 INFO:teuthology.orchestra.run.smithi083.stdout: "tier_type": "", 2022-09-13T07:25:29.522 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_from_all": true, 2022-09-13T07:25:29.523 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_from": [], 2022-09-13T07:25:29.523 INFO:teuthology.orchestra.run.smithi083.stdout: "redirect_zone": "", 2022-09-13T07:25:29.523 INFO:teuthology.orchestra.run.smithi083.stdout: "supported_features": [ 2022-09-13T07:25:29.524 INFO:teuthology.orchestra.run.smithi083.stdout: "resharding" 2022-09-13T07:25:29.524 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:25:29.524 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:29.524 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T07:25:29.525 INFO:teuthology.orchestra.run.smithi083.stdout: "placement_targets": [ 2022-09-13T07:25:29.525 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:29.525 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "default-placement", 2022-09-13T07:25:29.525 INFO:teuthology.orchestra.run.smithi083.stdout: "tags": [], 2022-09-13T07:25:29.525 INFO:teuthology.orchestra.run.smithi083.stdout: "storage_classes": [ 2022-09-13T07:25:29.526 INFO:teuthology.orchestra.run.smithi083.stdout: "STANDARD" 2022-09-13T07:25:29.526 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:25:29.526 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:29.527 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T07:25:29.527 INFO:teuthology.orchestra.run.smithi083.stdout: "default_placement": "default-placement", 2022-09-13T07:25:29.527 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T07:25:29.527 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_policy": { 2022-09-13T07:25:29.527 INFO:teuthology.orchestra.run.smithi083.stdout: "groups": [] 2022-09-13T07:25:29.528 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:29.528 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled_features": [ 2022-09-13T07:25:29.528 INFO:teuthology.orchestra.run.smithi083.stdout: "resharding" 2022-09-13T07:25:29.528 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:25:29.528 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:29.529 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T07:25:29.529 INFO:teuthology.orchestra.run.smithi083.stdout: "short_zone_ids": [ 2022-09-13T07:25:29.529 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:29.529 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T07:25:29.530 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 2827250327 2022-09-13T07:25:29.530 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:29.530 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:29.530 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T07:25:29.531 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 2837352077 2022-09-13T07:25:29.531 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:29.531 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:25:29.532 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:29.532 INFO:teuthology.orchestra.run.smithi083.stdout: "master_zonegroup": "test-zonegroup", 2022-09-13T07:25:29.532 INFO:teuthology.orchestra.run.smithi083.stdout: "master_zone": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T07:25:29.533 INFO:teuthology.orchestra.run.smithi083.stdout: "period_config": { 2022-09-13T07:25:29.533 INFO:teuthology.orchestra.run.smithi083.stdout: "bucket_quota": { 2022-09-13T07:25:29.533 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled": false, 2022-09-13T07:25:29.533 INFO:teuthology.orchestra.run.smithi083.stdout: "check_on_raw": false, 2022-09-13T07:25:29.534 INFO:teuthology.orchestra.run.smithi083.stdout: "max_size": -1, 2022-09-13T07:25:29.534 INFO:teuthology.orchestra.run.smithi083.stdout: "max_size_kb": 0, 2022-09-13T07:25:29.534 INFO:teuthology.orchestra.run.smithi083.stdout: "max_objects": -1 2022-09-13T07:25:29.535 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:29.535 INFO:teuthology.orchestra.run.smithi083.stdout: "user_quota": { 2022-09-13T07:25:29.535 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled": false, 2022-09-13T07:25:29.535 INFO:teuthology.orchestra.run.smithi083.stdout: "check_on_raw": false, 2022-09-13T07:25:29.536 INFO:teuthology.orchestra.run.smithi083.stdout: "max_size": -1, 2022-09-13T07:25:29.536 INFO:teuthology.orchestra.run.smithi083.stdout: "max_size_kb": 0, 2022-09-13T07:25:29.536 INFO:teuthology.orchestra.run.smithi083.stdout: "max_objects": -1 2022-09-13T07:25:29.536 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:29.537 INFO:teuthology.orchestra.run.smithi083.stdout: "user_ratelimit": { 2022-09-13T07:25:29.537 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_ops": 0, 2022-09-13T07:25:29.537 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_ops": 0, 2022-09-13T07:25:29.538 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_bytes": 0, 2022-09-13T07:25:29.538 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_bytes": 0, 2022-09-13T07:25:29.538 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled": false 2022-09-13T07:25:29.538 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:29.539 INFO:teuthology.orchestra.run.smithi083.stdout: "bucket_ratelimit": { 2022-09-13T07:25:29.539 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_ops": 0, 2022-09-13T07:25:29.539 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_ops": 0, 2022-09-13T07:25:29.539 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_bytes": 0, 2022-09-13T07:25:29.539 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_bytes": 0, 2022-09-13T07:25:29.540 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled": false 2022-09-13T07:25:29.540 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:29.541 INFO:teuthology.orchestra.run.smithi083.stdout: "anonymous_ratelimit": { 2022-09-13T07:25:29.541 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_ops": 0, 2022-09-13T07:25:29.542 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_ops": 0, 2022-09-13T07:25:29.542 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_bytes": 0, 2022-09-13T07:25:29.542 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_bytes": 0, 2022-09-13T07:25:29.543 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled": false 2022-09-13T07:25:29.543 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:29.543 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:29.543 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T07:25:29.544 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_name": "test-realm", 2022-09-13T07:25:29.544 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:25:29.544 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:25:29.545 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2022-09-13T07:25:29.545 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T07:25:29.546 INFO:tasks.rgw.c2.client.0.smithi083.stdout:2022-09-13T07:25:29.542+0000 7f18292fe700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 107343) UID: 0 2022-09-13T07:25:29.546 INFO:tasks.rgw.c2.client.0.smithi083.stdout:2022-09-13T07:25:29.542+0000 7f182d569600 -1 shutting down 2022-09-13T07:25:33.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-09-13T07:25:35.647 INFO:tasks.rgw.c2.client.0:Stopped 2022-09-13T07:25:35.647 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-09-13T07:25:35.648 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-09-13T07:25:35.651 INFO:tasks.rgw.c2.client.0:Started 2022-09-13T07:25:35.651 DEBUG:teuthology.orchestra.run.smithi083:> curl http://smithi083.front.sepia.ceph.com:8000 2022-09-13T07:25:35.676 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-13T07:25:35.677 INFO:teuthology.orchestra.run.smithi083.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-13T07:25:35.678 INFO:teuthology.orchestra.run.smithi083.stderr: Dload Upload Total Spent Left Speed 2022-09-13T07:25:35.678 INFO:teuthology.orchestra.run.smithi083.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi083.front.sepia.ceph.com port 8000: Connection refused 2022-09-13T07:25:36.679 DEBUG:teuthology.orchestra.run.smithi083:> curl http://smithi083.front.sepia.ceph.com:8000 2022-09-13T07:25:36.701 INFO:teuthology.orchestra.run.smithi083.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-13T07:25:36.702 INFO:teuthology.orchestra.run.smithi083.stderr: Dload Upload Total Spent Left Speed 2022-09-13T07:25:36.703 INFO:teuthology.orchestra.run.smithi083.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-13T07:25:36.705 INFO:teuthology.orchestra.run.smithi083.stdout:anonymous 2022-09-13T07:25:36.705 INFO:tasks.rgw_multisite:creating zone.. 2022-09-13T07:25:36.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'G693TB5XEJ3206QY', '--secret', 'QS5zddodwd26OlSDjoxox3W0BzgFaGzV', '--endpoints', 'http://smithi036.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:36.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'G693TB5XEJ3206QY', '--secret', 'QS5zddodwd26OlSDjoxox3W0BzgFaGzV', '--endpoints', 'http://smithi036.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:36.707 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key G693TB5XEJ3206QY --secret QS5zddodwd26OlSDjoxox3W0BzgFaGzV --endpoints http://smithi036.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T07:25:36.770 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:36.771 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:36.797 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:25:36.794+0000 7f5fb4e15600 0 failed reading obj info from .rgw.root:zone_info.30ddd076-3f98-4411-9a5a-48d7fd04ad75: (2) No such file or directory 2022-09-13T07:25:36.798 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:25:36.794+0000 7f5fb4e15600 0 WARNING: could not read zone params for zone id=30ddd076-3f98-4411-9a5a-48d7fd04ad75 name=test-zone2 2022-09-13T07:25:36.801 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:25:36.802 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9", 2022-09-13T07:25:36.802 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zone3", 2022-09-13T07:25:36.802 INFO:teuthology.orchestra.run.smithi036.stdout: "domain_root": "test-zone3.rgw.meta:root", 2022-09-13T07:25:36.802 INFO:teuthology.orchestra.run.smithi036.stdout: "control_pool": "test-zone3.rgw.control", 2022-09-13T07:25:36.803 INFO:teuthology.orchestra.run.smithi036.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2022-09-13T07:25:36.803 INFO:teuthology.orchestra.run.smithi036.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2022-09-13T07:25:36.803 INFO:teuthology.orchestra.run.smithi036.stdout: "log_pool": "test-zone3.rgw.log", 2022-09-13T07:25:36.804 INFO:teuthology.orchestra.run.smithi036.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2022-09-13T07:25:36.804 INFO:teuthology.orchestra.run.smithi036.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2022-09-13T07:25:36.804 INFO:teuthology.orchestra.run.smithi036.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2022-09-13T07:25:36.804 INFO:teuthology.orchestra.run.smithi036.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2022-09-13T07:25:36.805 INFO:teuthology.orchestra.run.smithi036.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2022-09-13T07:25:36.805 INFO:teuthology.orchestra.run.smithi036.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2022-09-13T07:25:36.805 INFO:teuthology.orchestra.run.smithi036.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2022-09-13T07:25:36.805 INFO:teuthology.orchestra.run.smithi036.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2022-09-13T07:25:36.805 INFO:teuthology.orchestra.run.smithi036.stdout: "otp_pool": "test-zone3.rgw.otp", 2022-09-13T07:25:36.806 INFO:teuthology.orchestra.run.smithi036.stdout: "system_key": { 2022-09-13T07:25:36.806 INFO:teuthology.orchestra.run.smithi036.stdout: "access_key": "G693TB5XEJ3206QY", 2022-09-13T07:25:36.806 INFO:teuthology.orchestra.run.smithi036.stdout: "secret_key": "QS5zddodwd26OlSDjoxox3W0BzgFaGzV" 2022-09-13T07:25:36.807 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:36.807 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_pools": [ 2022-09-13T07:25:36.807 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:36.807 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "default-placement", 2022-09-13T07:25:36.807 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:25:36.808 INFO:teuthology.orchestra.run.smithi036.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2022-09-13T07:25:36.808 INFO:teuthology.orchestra.run.smithi036.stdout: "storage_classes": { 2022-09-13T07:25:36.808 INFO:teuthology.orchestra.run.smithi036.stdout: "STANDARD": { 2022-09-13T07:25:36.808 INFO:teuthology.orchestra.run.smithi036.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2022-09-13T07:25:36.809 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:36.809 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:36.809 INFO:teuthology.orchestra.run.smithi036.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2022-09-13T07:25:36.810 INFO:teuthology.orchestra.run.smithi036.stdout: "index_type": 0 2022-09-13T07:25:36.810 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:36.810 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:36.810 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:36.811 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T07:25:36.811 INFO:teuthology.orchestra.run.smithi036.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2022-09-13T07:25:36.811 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:25:36.813 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2022-09-13T07:25:37.571 INFO:teuthology.orchestra.run.smithi036.stderr:pool 'test-zone3.rgw.buckets.data' created 2022-09-13T07:25:37.582 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2022-09-13T07:25:38.576 INFO:teuthology.orchestra.run.smithi036.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2022-09-13T07:25:38.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:38.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:38.593 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T07:25:38.648 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:38.649 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:38.673 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:25:38.674 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9", 2022-09-13T07:25:38.674 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zone3", 2022-09-13T07:25:38.674 INFO:teuthology.orchestra.run.smithi036.stdout: "domain_root": "test-zone3.rgw.meta:root", 2022-09-13T07:25:38.675 INFO:teuthology.orchestra.run.smithi036.stdout: "control_pool": "test-zone3.rgw.control", 2022-09-13T07:25:38.675 INFO:teuthology.orchestra.run.smithi036.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2022-09-13T07:25:38.675 INFO:teuthology.orchestra.run.smithi036.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2022-09-13T07:25:38.675 INFO:teuthology.orchestra.run.smithi036.stdout: "log_pool": "test-zone3.rgw.log", 2022-09-13T07:25:38.676 INFO:teuthology.orchestra.run.smithi036.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2022-09-13T07:25:38.676 INFO:teuthology.orchestra.run.smithi036.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2022-09-13T07:25:38.676 INFO:teuthology.orchestra.run.smithi036.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2022-09-13T07:25:38.676 INFO:teuthology.orchestra.run.smithi036.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2022-09-13T07:25:38.677 INFO:teuthology.orchestra.run.smithi036.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2022-09-13T07:25:38.677 INFO:teuthology.orchestra.run.smithi036.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2022-09-13T07:25:38.677 INFO:teuthology.orchestra.run.smithi036.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2022-09-13T07:25:38.678 INFO:teuthology.orchestra.run.smithi036.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2022-09-13T07:25:38.678 INFO:teuthology.orchestra.run.smithi036.stdout: "otp_pool": "test-zone3.rgw.otp", 2022-09-13T07:25:38.678 INFO:teuthology.orchestra.run.smithi036.stdout: "system_key": { 2022-09-13T07:25:38.678 INFO:teuthology.orchestra.run.smithi036.stdout: "access_key": "G693TB5XEJ3206QY", 2022-09-13T07:25:38.679 INFO:teuthology.orchestra.run.smithi036.stdout: "secret_key": "QS5zddodwd26OlSDjoxox3W0BzgFaGzV" 2022-09-13T07:25:38.679 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:38.679 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_pools": [ 2022-09-13T07:25:38.679 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:38.680 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "default-placement", 2022-09-13T07:25:38.680 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:25:38.680 INFO:teuthology.orchestra.run.smithi036.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2022-09-13T07:25:38.680 INFO:teuthology.orchestra.run.smithi036.stdout: "storage_classes": { 2022-09-13T07:25:38.680 INFO:teuthology.orchestra.run.smithi036.stdout: "STANDARD": { 2022-09-13T07:25:38.681 INFO:teuthology.orchestra.run.smithi036.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2022-09-13T07:25:38.681 INFO:teuthology.orchestra.run.smithi036.stdout: "compression_type": "random" 2022-09-13T07:25:38.681 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:38.682 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:38.682 INFO:teuthology.orchestra.run.smithi036.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2022-09-13T07:25:38.682 INFO:teuthology.orchestra.run.smithi036.stdout: "index_type": 0 2022-09-13T07:25:38.682 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:38.683 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:38.683 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:38.683 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "", 2022-09-13T07:25:38.683 INFO:teuthology.orchestra.run.smithi036.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2022-09-13T07:25:38.684 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:25:38.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:38.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:38.686 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T07:25:38.736 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:38.736 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:38.745 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:25:38.742+0000 7f36efb6f600 0 period (3255cc65-e34b-4e83-a436-e06431413191 does not have zone 4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9 configured 2022-09-13T07:25:39.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2022-09-13T07:25:45.614 INFO:teuthology.orchestra.run.smithi036.stderr:Sending period to new master zone 7103daa4-1b45-47a1-bb49-5d9877f6d137 2022-09-13T07:25:45.638 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:25:45.639 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:25:45.639 INFO:teuthology.orchestra.run.smithi036.stdout: "epoch": 3, 2022-09-13T07:25:45.639 INFO:teuthology.orchestra.run.smithi036.stdout: "predecessor_uuid": "0f1e9404-34a5-47ad-8196-42a4e6e2ff9c", 2022-09-13T07:25:45.640 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": [], 2022-09-13T07:25:45.640 INFO:teuthology.orchestra.run.smithi036.stdout: "period_map": { 2022-09-13T07:25:45.640 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:25:45.640 INFO:teuthology.orchestra.run.smithi036.stdout: "zonegroups": [ 2022-09-13T07:25:45.640 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:45.641 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "test-zonegroup", 2022-09-13T07:25:45.641 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zonegroup", 2022-09-13T07:25:45.641 INFO:teuthology.orchestra.run.smithi036.stdout: "api_name": "", 2022-09-13T07:25:45.641 INFO:teuthology.orchestra.run.smithi036.stdout: "is_master": true, 2022-09-13T07:25:45.642 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [ 2022-09-13T07:25:45.642 INFO:teuthology.orchestra.run.smithi036.stdout: "http://smithi036.front.sepia.ceph.com:8000" 2022-09-13T07:25:45.642 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:45.642 INFO:teuthology.orchestra.run.smithi036.stdout: "hostnames": [], 2022-09-13T07:25:45.642 INFO:teuthology.orchestra.run.smithi036.stdout: "hostnames_s3website": [], 2022-09-13T07:25:45.643 INFO:teuthology.orchestra.run.smithi036.stdout: "master_zone": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T07:25:45.643 INFO:teuthology.orchestra.run.smithi036.stdout: "zones": [ 2022-09-13T07:25:45.643 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:45.643 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T07:25:45.644 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zone2", 2022-09-13T07:25:45.644 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [ 2022-09-13T07:25:45.644 INFO:teuthology.orchestra.run.smithi036.stdout: "http://smithi083.front.sepia.ceph.com:8000" 2022-09-13T07:25:45.644 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:45.644 INFO:teuthology.orchestra.run.smithi036.stdout: "log_meta": false, 2022-09-13T07:25:45.645 INFO:teuthology.orchestra.run.smithi036.stdout: "log_data": true, 2022-09-13T07:25:45.645 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_index_max_shards": 11, 2022-09-13T07:25:45.645 INFO:teuthology.orchestra.run.smithi036.stdout: "read_only": false, 2022-09-13T07:25:45.645 INFO:teuthology.orchestra.run.smithi036.stdout: "tier_type": "", 2022-09-13T07:25:45.645 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from_all": true, 2022-09-13T07:25:45.646 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from": [], 2022-09-13T07:25:45.646 INFO:teuthology.orchestra.run.smithi036.stdout: "redirect_zone": "", 2022-09-13T07:25:45.646 INFO:teuthology.orchestra.run.smithi036.stdout: "supported_features": [ 2022-09-13T07:25:45.646 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T07:25:45.647 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:25:45.647 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:45.647 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:45.647 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9", 2022-09-13T07:25:45.647 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zone3", 2022-09-13T07:25:45.648 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [ 2022-09-13T07:25:45.648 INFO:teuthology.orchestra.run.smithi036.stdout: "http://smithi036.front.sepia.ceph.com:8001" 2022-09-13T07:25:45.648 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:45.648 INFO:teuthology.orchestra.run.smithi036.stdout: "log_meta": false, 2022-09-13T07:25:45.649 INFO:teuthology.orchestra.run.smithi036.stdout: "log_data": true, 2022-09-13T07:25:45.649 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_index_max_shards": 11, 2022-09-13T07:25:45.649 INFO:teuthology.orchestra.run.smithi036.stdout: "read_only": false, 2022-09-13T07:25:45.649 INFO:teuthology.orchestra.run.smithi036.stdout: "tier_type": "", 2022-09-13T07:25:45.649 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from_all": true, 2022-09-13T07:25:45.650 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from": [], 2022-09-13T07:25:45.650 INFO:teuthology.orchestra.run.smithi036.stdout: "redirect_zone": "", 2022-09-13T07:25:45.650 INFO:teuthology.orchestra.run.smithi036.stdout: "supported_features": [ 2022-09-13T07:25:45.650 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T07:25:45.650 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:25:45.651 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:45.651 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:45.651 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T07:25:45.651 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zone1", 2022-09-13T07:25:45.652 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [ 2022-09-13T07:25:45.652 INFO:teuthology.orchestra.run.smithi036.stdout: "http://smithi036.front.sepia.ceph.com:8000" 2022-09-13T07:25:45.652 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:45.653 INFO:teuthology.orchestra.run.smithi036.stdout: "log_meta": false, 2022-09-13T07:25:45.653 INFO:teuthology.orchestra.run.smithi036.stdout: "log_data": true, 2022-09-13T07:25:45.653 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_index_max_shards": 11, 2022-09-13T07:25:45.653 INFO:teuthology.orchestra.run.smithi036.stdout: "read_only": false, 2022-09-13T07:25:45.653 INFO:teuthology.orchestra.run.smithi036.stdout: "tier_type": "", 2022-09-13T07:25:45.654 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from_all": true, 2022-09-13T07:25:45.654 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from": [], 2022-09-13T07:25:45.654 INFO:teuthology.orchestra.run.smithi036.stdout: "redirect_zone": "", 2022-09-13T07:25:45.654 INFO:teuthology.orchestra.run.smithi036.stdout: "supported_features": [ 2022-09-13T07:25:45.655 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T07:25:45.655 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:25:45.655 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:45.655 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:45.655 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_targets": [ 2022-09-13T07:25:45.656 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:45.656 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "default-placement", 2022-09-13T07:25:45.656 INFO:teuthology.orchestra.run.smithi036.stdout: "tags": [], 2022-09-13T07:25:45.656 INFO:teuthology.orchestra.run.smithi036.stdout: "storage_classes": [ 2022-09-13T07:25:45.656 INFO:teuthology.orchestra.run.smithi036.stdout: "STANDARD" 2022-09-13T07:25:45.657 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:25:45.657 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:45.657 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:45.657 INFO:teuthology.orchestra.run.smithi036.stdout: "default_placement": "default-placement", 2022-09-13T07:25:45.657 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T07:25:45.658 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_policy": { 2022-09-13T07:25:45.658 INFO:teuthology.orchestra.run.smithi036.stdout: "groups": [] 2022-09-13T07:25:45.658 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:45.658 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled_features": [ 2022-09-13T07:25:45.659 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T07:25:45.659 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:25:45.659 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:45.659 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:45.659 INFO:teuthology.orchestra.run.smithi036.stdout: "short_zone_ids": [ 2022-09-13T07:25:45.660 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:45.660 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T07:25:45.660 INFO:teuthology.orchestra.run.smithi036.stdout: "val": 2827250327 2022-09-13T07:25:45.660 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:45.660 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:45.661 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9", 2022-09-13T07:25:45.661 INFO:teuthology.orchestra.run.smithi036.stdout: "val": 1723930414 2022-09-13T07:25:45.661 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:45.661 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:45.661 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T07:25:45.662 INFO:teuthology.orchestra.run.smithi036.stdout: "val": 2837352077 2022-09-13T07:25:45.662 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:45.662 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:25:45.662 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:45.664 INFO:teuthology.orchestra.run.smithi036.stdout: "master_zonegroup": "test-zonegroup", 2022-09-13T07:25:45.664 INFO:teuthology.orchestra.run.smithi036.stdout: "master_zone": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T07:25:45.664 INFO:teuthology.orchestra.run.smithi036.stdout: "period_config": { 2022-09-13T07:25:45.664 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_quota": { 2022-09-13T07:25:45.665 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false, 2022-09-13T07:25:45.665 INFO:teuthology.orchestra.run.smithi036.stdout: "check_on_raw": false, 2022-09-13T07:25:45.665 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size": -1, 2022-09-13T07:25:45.665 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size_kb": 0, 2022-09-13T07:25:45.665 INFO:teuthology.orchestra.run.smithi036.stdout: "max_objects": -1 2022-09-13T07:25:45.666 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:45.666 INFO:teuthology.orchestra.run.smithi036.stdout: "user_quota": { 2022-09-13T07:25:45.667 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false, 2022-09-13T07:25:45.667 INFO:teuthology.orchestra.run.smithi036.stdout: "check_on_raw": false, 2022-09-13T07:25:45.667 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size": -1, 2022-09-13T07:25:45.668 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size_kb": 0, 2022-09-13T07:25:45.668 INFO:teuthology.orchestra.run.smithi036.stdout: "max_objects": -1 2022-09-13T07:25:45.669 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:45.669 INFO:teuthology.orchestra.run.smithi036.stdout: "user_ratelimit": { 2022-09-13T07:25:45.669 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_ops": 0, 2022-09-13T07:25:45.670 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_ops": 0, 2022-09-13T07:25:45.670 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_bytes": 0, 2022-09-13T07:25:45.670 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_bytes": 0, 2022-09-13T07:25:45.671 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false 2022-09-13T07:25:45.671 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:45.672 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_ratelimit": { 2022-09-13T07:25:45.672 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_ops": 0, 2022-09-13T07:25:45.672 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_ops": 0, 2022-09-13T07:25:45.673 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_bytes": 0, 2022-09-13T07:25:45.673 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_bytes": 0, 2022-09-13T07:25:45.673 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false 2022-09-13T07:25:45.674 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:45.674 INFO:teuthology.orchestra.run.smithi036.stdout: "anonymous_ratelimit": { 2022-09-13T07:25:45.675 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_ops": 0, 2022-09-13T07:25:45.675 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_ops": 0, 2022-09-13T07:25:45.676 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_bytes": 0, 2022-09-13T07:25:45.676 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_bytes": 0, 2022-09-13T07:25:45.676 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false 2022-09-13T07:25:45.677 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:45.677 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:45.677 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T07:25:45.678 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_name": "test-realm", 2022-09-13T07:25:45.678 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:25:45.679 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:25:45.680 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2022-09-13T07:25:45.680 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T07:25:45.681 INFO:tasks.rgw.c1.client.1.smithi036.stdout:2022-09-13T07:25:45.677+0000 7f261ddef700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 107595) UID: 0 2022-09-13T07:25:45.682 INFO:tasks.rgw.c1.client.1.smithi036.stdout:2022-09-13T07:25:45.677+0000 7f262205a600 -1 shutting down 2022-09-13T07:25:50.371 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2022-09-13T07:25:51.783 INFO:tasks.rgw.c1.client.1:Stopped 2022-09-13T07:25:51.784 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-09-13T07:25:51.784 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-09-13T07:25:51.788 INFO:tasks.rgw.c1.client.1:Started 2022-09-13T07:25:51.789 DEBUG:teuthology.orchestra.run.smithi036:> curl http://smithi036.front.sepia.ceph.com:8001 2022-09-13T07:25:51.815 INFO:teuthology.orchestra.run.smithi036.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-13T07:25:51.816 INFO:teuthology.orchestra.run.smithi036.stderr: Dload Upload Total Spent Left Speed 2022-09-13T07:25:51.817 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-13T07:25:51.818 INFO:teuthology.orchestra.run.smithi036.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi036.front.sepia.ceph.com port 8001: Connection refused 2022-09-13T07:25:52.819 DEBUG:teuthology.orchestra.run.smithi036:> curl http://smithi036.front.sepia.ceph.com:8001 2022-09-13T07:25:52.845 INFO:teuthology.orchestra.run.smithi036.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-13T07:25:52.846 INFO:teuthology.orchestra.run.smithi036.stderr: Dload Upload Total Spent Left Speed 2022-09-13T07:25:52.846 INFO:teuthology.orchestra.run.smithi036.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-13T07:25:52.848 INFO:teuthology.orchestra.run.smithi036.stdout:anonymous 2022-09-13T07:25:52.848 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2022-09-13T07:25:52.849 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2022-09-13T07:25:52.861 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi 2022-09-13T07:25:52.881 INFO:tasks.rgw_multisite_tests:creating test user.. 2022-09-13T07:25:52.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:52.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T07:25:52.883 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T07:25:52.945 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:52.945 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:52.982 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:25:52.983 INFO:teuthology.orchestra.run.smithi036.stdout: "user_id": "rgw-multisite-test-user", 2022-09-13T07:25:52.984 INFO:teuthology.orchestra.run.smithi036.stdout: "display_name": "Multisite Test User", 2022-09-13T07:25:52.984 INFO:teuthology.orchestra.run.smithi036.stdout: "email": "", 2022-09-13T07:25:52.985 INFO:teuthology.orchestra.run.smithi036.stdout: "suspended": 0, 2022-09-13T07:25:52.985 INFO:teuthology.orchestra.run.smithi036.stdout: "max_buckets": 1000, 2022-09-13T07:25:52.985 INFO:teuthology.orchestra.run.smithi036.stdout: "subusers": [], 2022-09-13T07:25:52.986 INFO:teuthology.orchestra.run.smithi036.stdout: "keys": [ 2022-09-13T07:25:52.986 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:52.987 INFO:teuthology.orchestra.run.smithi036.stdout: "user": "rgw-multisite-test-user", 2022-09-13T07:25:52.987 INFO:teuthology.orchestra.run.smithi036.stdout: "access_key": "RICCPJUVKDGV9UEGFN6Z", 2022-09-13T07:25:52.988 INFO:teuthology.orchestra.run.smithi036.stdout: "secret_key": "KkyX7cb1MqTxVFsCIradvB7cdNTX0aPkucA9NvuB" 2022-09-13T07:25:52.988 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:52.988 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:52.989 INFO:teuthology.orchestra.run.smithi036.stdout: "swift_keys": [], 2022-09-13T07:25:52.989 INFO:teuthology.orchestra.run.smithi036.stdout: "caps": [ 2022-09-13T07:25:52.989 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:52.989 INFO:teuthology.orchestra.run.smithi036.stdout: "type": "roles", 2022-09-13T07:25:52.989 INFO:teuthology.orchestra.run.smithi036.stdout: "perm": "*" 2022-09-13T07:25:52.990 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:52.990 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T07:25:52.990 INFO:teuthology.orchestra.run.smithi036.stdout: "op_mask": "read, write, delete", 2022-09-13T07:25:52.990 INFO:teuthology.orchestra.run.smithi036.stdout: "default_placement": "", 2022-09-13T07:25:52.991 INFO:teuthology.orchestra.run.smithi036.stdout: "default_storage_class": "", 2022-09-13T07:25:52.991 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_tags": [], 2022-09-13T07:25:52.991 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_quota": { 2022-09-13T07:25:52.991 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false, 2022-09-13T07:25:52.992 INFO:teuthology.orchestra.run.smithi036.stdout: "check_on_raw": false, 2022-09-13T07:25:52.992 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size": -1, 2022-09-13T07:25:52.992 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size_kb": 0, 2022-09-13T07:25:52.992 INFO:teuthology.orchestra.run.smithi036.stdout: "max_objects": -1 2022-09-13T07:25:52.993 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:52.993 INFO:teuthology.orchestra.run.smithi036.stdout: "user_quota": { 2022-09-13T07:25:52.993 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false, 2022-09-13T07:25:52.993 INFO:teuthology.orchestra.run.smithi036.stdout: "check_on_raw": false, 2022-09-13T07:25:52.993 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size": -1, 2022-09-13T07:25:52.994 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size_kb": 0, 2022-09-13T07:25:52.994 INFO:teuthology.orchestra.run.smithi036.stdout: "max_objects": -1 2022-09-13T07:25:52.994 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:52.994 INFO:teuthology.orchestra.run.smithi036.stdout: "temp_url_keys": [], 2022-09-13T07:25:52.995 INFO:teuthology.orchestra.run.smithi036.stdout: "type": "rgw", 2022-09-13T07:25:52.995 INFO:teuthology.orchestra.run.smithi036.stdout: "mfa_ids": [] 2022-09-13T07:25:52.995 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:25:52.995 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T07:25:52.997 INFO:rgw_multi.tests:meta checkpoint 2022-09-13T07:25:52.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:25:52.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:25:52.998 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:25:53.048 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:53.048 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:53.087 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:25:53.102 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:25:53.102 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:53.102 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:25:53.103 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:25:53.103 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:53.103 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:53.103 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:25:53.104 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:25:53.104 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:53.104 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:53.105 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:25:53.105 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:25:53.106 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:25:53.106 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:25:53.106 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:25:53.106 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:25:53.107 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:25:53.107 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:25:53.108 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:25:53.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:25:53.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:25:53.109 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:25:53.166 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:53.166 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:53.208 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:53.205+0000 7fe7a2cdd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:25:53.209 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:53.206+0000 7fe7a2cdd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:25:53.230 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:25:53.230 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:25:53.231 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:25:53.231 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:25:53.231 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:25:53.231 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:25:53.232 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:25:53.232 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:53.232 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:25:53.232 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:53.233 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:25:53.233 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:25:53.233 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:25:53.233 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:25:53.233 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:25:53.234 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:25:53.234 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:25:53.234 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:25:53.234 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:25:53.235 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:53.235 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:53.235 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:53.235 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:25:53.236 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:25:53.236 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:25:53.236 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:25:53.236 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:25:53.236 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:25:53.237 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:25:53.237 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:25:53.237 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:25:53.237 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:53.237 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:53.238 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:53.238 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:25:53.238 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:25:53.238 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:25:53.239 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:25:53.239 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:25:53.239 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:25:53.239 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:25:53.239 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:25:53.240 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:25:53.240 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:53.240 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:53.240 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:53.240 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:25:53.241 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:25:53.241 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:25:53.241 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:25:53.241 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:25:53.242 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:25:53.242 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:25:53.242 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:25:53.242 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:25:53.242 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:53.243 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:53.243 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:25:53.243 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:53.243 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:25:53.243 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:25:53.244 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:25:53.244 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:53.244 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:25:53Z" 2022-09-13T07:25:53.244 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:25:53.254 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:25:56.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2022-09-13T07:25:58.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:25:58.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:25:58.257 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:25:58.313 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:25:58.313 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:25:58.359 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:58.356+0000 7fadb4066600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:25:58.360 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:25:58.357+0000 7fadb4066600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:25:58.381 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:25:58.382 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:25:58.382 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:25:58.383 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:25:58.383 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:25:58.384 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:25:58.384 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:25:58.384 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:58.385 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:25:58.385 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:58.386 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:25:58.386 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:25:58.386 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:25:58.387 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:25:58.387 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:25:58.388 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:25:58.388 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:25:58.388 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:25:58.389 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:25:58.389 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:58.389 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:58.390 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:58.390 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:25:58.390 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:25:58.391 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:25:58.391 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:25:58.391 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:25:58.391 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:25:58.392 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:25:58.392 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:25:58.392 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:25:58.392 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:58.392 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:58.393 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:58.393 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:25:58.393 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:25:58.393 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:25:58.394 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:25:58.394 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:25:58.394 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:25:58.394 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:25:58.395 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:25:58.395 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:25:58.395 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:58.395 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:58.395 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:25:58.396 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:25:58.396 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:25:58.396 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:25:58.396 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:25:58.397 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:25:58.397 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:25:58.397 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:25:58.397 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:25:58.397 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:25:58.398 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:58.398 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:25:58.398 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:25:58.399 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:58.399 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:25:58.399 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:25:58.400 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:25:58.400 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:25:58.400 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:25:58Z" 2022-09-13T07:25:58.401 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:25:58.403 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:26:03.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:03.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:03.406 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:26:03.464 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:26:03.465 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:26:03.494 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:03.491+0000 7fa220c8b600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:03.495 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:03.492+0000 7fa220c8b600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:03.510 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:26:03.511 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:26:03.511 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:26:03.512 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:26:03.512 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:26:03.513 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:26:03.513 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:03.513 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:03.514 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:26:03.514 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:03.515 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:26:03.515 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:03.516 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:03.516 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:03.517 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:03.517 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:26:03.518 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:03.518 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:03.518 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:03.519 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:03.519 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:03.519 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:03.520 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:26:03.520 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:03.520 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:03.520 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:03.521 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:03.521 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:03.521 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:03.522 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:03.522 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:03.522 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:03.522 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:03.523 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:03.523 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:26:03.523 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:03.523 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:03.524 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:03.524 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:03.524 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:03.525 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:03.525 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:03.525 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:03.526 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:03.526 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:03.527 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:03.527 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:26:03.527 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:03.528 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:03.528 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:03.529 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:03.529 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:03.529 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:03.530 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:03.531 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:03.531 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:03.532 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:03.532 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:26:03.532 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:03.533 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:26:03.533 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:26:03.534 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:26:03.534 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:03.534 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:26:03Z" 2022-09-13T07:26:03.535 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:26:03.537 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:26:08.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:08.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:08.540 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:26:08.596 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:26:08.596 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:26:08.639 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:08.636+0000 7ff9e8003600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:08.641 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:08.637+0000 7ff9e8003600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:08.657 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:26:08.658 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:26:08.658 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:26:08.658 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:26:08.659 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:26:08.659 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:26:08.659 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:08.659 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:08.660 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:26:08.660 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:08.660 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:26:08.661 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:08.661 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:08.661 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:08.662 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:08.662 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:26:08.662 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:08.663 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:08.663 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:08.663 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:08.663 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:08.664 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:08.664 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:26:08.664 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:08.664 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:08.665 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:08.665 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:08.665 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:08.665 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:08.666 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:08.666 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:08.666 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:08.666 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:08.667 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:08.667 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:26:08.667 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:08.667 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:08.668 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:08.668 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:08.669 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:08.669 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:08.669 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:08.669 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:08.669 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:08.670 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:08.670 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:08.670 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:26:08.671 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:08.671 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:08.671 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:08.671 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:08.671 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:08.672 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:08.672 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:08.672 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:08.672 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:08.673 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:08.673 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:26:08.673 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:08.673 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:26:08.674 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:26:08.674 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:26:08.674 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:08.674 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:26:08Z" 2022-09-13T07:26:08.675 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:26:08.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:26:13.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:13.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:13.679 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:26:13.736 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:26:13.736 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:26:13.780 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:13.777+0000 7f455453d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:13.781 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:13.778+0000 7f455453d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:13.796 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:26:13.796 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:26:13.796 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:26:13.796 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:26:13.797 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:26:13.797 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:26:13.797 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:13.797 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:13.798 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:26:13.798 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:13.798 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:26:13.798 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:13.799 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:13.799 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:13.799 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:13.799 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:26:13.799 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:13.800 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:13.800 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:13.800 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:13.801 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:13.801 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:13.801 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:26:13.801 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:13.802 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:13.802 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:13.802 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:13.802 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:13.802 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:13.803 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:13.803 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:13.803 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:13.803 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:13.804 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:13.804 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:26:13.804 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:13.804 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:13.804 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:13.805 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:13.805 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:13.805 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:13.805 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:13.805 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:13.806 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:13.806 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:13.806 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:13.806 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:26:13.807 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:13.807 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:13.807 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:13.807 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:13.807 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:13.808 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:13.808 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:13.808 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:13.808 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:13.809 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:13.809 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:26:13.809 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:13.809 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:26:13.809 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:26:13.810 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:26:13.810 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:13.810 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:26:13Z" 2022-09-13T07:26:13.810 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:26:13.811 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:26:18.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:18.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:18.814 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:26:18.870 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:26:18.871 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:26:18.909 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:18.907+0000 7f8f7c95f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:18.911 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:18.908+0000 7f8f7c95f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:18.925 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:26:18.925 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:26:18.925 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:26:18.926 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:26:18.926 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:26:18.926 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:26:18.926 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:18.927 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:18.927 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:26:18.927 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:18.927 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:26:18.928 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:18.928 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:18.928 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:18.928 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:18.929 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:26:18.929 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:18.929 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:18.929 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:18.929 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:18.930 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:18.930 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:18.930 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:26:18.931 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:18.931 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:18.931 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:18.931 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:18.931 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:18.932 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:18.932 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:18.932 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:18.932 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:18.933 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:18.933 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:18.933 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:26:18.934 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:18.934 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:18.934 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:18.935 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:18.935 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:18.935 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:18.935 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:18.936 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:18.936 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:18.936 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:18.936 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:18.936 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:26:18.937 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:18.937 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:18.937 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:18.937 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:18.938 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:18.938 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:18.938 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:18.938 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:18.939 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:18.939 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:18.939 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:26:18.939 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:18.940 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:26:18.940 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:26:18.940 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:26:18.940 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:18.940 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:26:18Z" 2022-09-13T07:26:18.941 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:26:18.942 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:26:23.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:23.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:23.945 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:26:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:26:24.002 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:26:24.034 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:24.032+0000 7f488e94f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:24.035 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:24.033+0000 7f488e94f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:24.047 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:26:24.047 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:26:24.048 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:26:24.048 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:26:24.048 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:26:24.048 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:26:24.049 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:24.049 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:24.049 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:26:24.049 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:24.050 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:26:24.050 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:24.050 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:24.050 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:24.051 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:24.051 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:26:24.051 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:24.051 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:24.052 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:24.052 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:24.052 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:24.052 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:24.053 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:26:24.053 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:24.053 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:24.053 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:24.053 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:24.054 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:24.054 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:24.055 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:24.055 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:24.055 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:24.055 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:24.056 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:24.056 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:26:24.056 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:24.056 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:24.056 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:24.057 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:24.057 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:24.057 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:24.057 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:24.058 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:24.058 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:24.058 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:24.058 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:24.058 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:26:24.059 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:24.059 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:24.059 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:24.059 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:24.060 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:24.060 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:24.060 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:24.060 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:24.061 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:24.061 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:24.061 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:26:24.061 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:24.062 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:26:24.062 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:26:24.062 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:26:24.062 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:24.063 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:26:24Z" 2022-09-13T07:26:24.063 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:26:24.064 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:26:29.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:29.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:29.067 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:26:29.136 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:26:29.136 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:26:29.172 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:29.169+0000 7f85a864a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:29.173 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:29.170+0000 7f85a864a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:29.191 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:26:29.192 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:26:29.192 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:26:29.193 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:26:29.193 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:26:29.193 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:26:29.194 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:29.195 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:29.196 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:26:29.196 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:29.197 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:26:29.197 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:29.198 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:29.198 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:29.199 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:29.199 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:26:29.200 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:29.200 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:29.201 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:29.201 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:29.201 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:29.202 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:29.202 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:26:29.203 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:29.203 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:29.204 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:29.204 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:29.205 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:29.205 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:29.206 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:29.206 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:29.207 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:29.207 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:29.208 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:29.208 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:26:29.208 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:29.209 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:29.209 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:29.210 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:29.210 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:29.211 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:29.211 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:29.212 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:29.212 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:29.213 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:29.213 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:29.213 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:26:29.214 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:29.214 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:29.215 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:29.215 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:29.216 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:29.216 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:29.216 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:29.217 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:29.217 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:29.218 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:29.218 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:26:29.219 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:29.219 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:26:29.219 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:26:29.219 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:26:29.220 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:29.220 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:26:29Z" 2022-09-13T07:26:29.220 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:26:29.222 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:26:34.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:34.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:34.225 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:26:34.281 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:26:34.282 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:26:34.321 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:34.318+0000 7f20f7ed7600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:34.322 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:34.319+0000 7f20f7ed7600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:34.338 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:26:34.338 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:26:34.338 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:26:34.339 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:26:34.339 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:26:34.339 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:26:34.339 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:34.340 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:34.340 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:26:34.340 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:34.340 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:26:34.342 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:34.342 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:34.342 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:34.343 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:34.344 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:26:34.344 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:34.344 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:34.344 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:34.345 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:34.345 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:34.345 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:34.346 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:26:34.346 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:34.346 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:34.347 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:34.347 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:34.348 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:34.348 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:34.348 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:34.349 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:34.349 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:34.350 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:34.351 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:34.351 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:26:34.351 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:34.351 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:34.352 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:34.352 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:34.352 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:34.352 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:34.353 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:34.353 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:34.353 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:34.354 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:34.354 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:34.354 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:26:34.354 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:34.355 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:34.355 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:34.355 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:34.355 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:34.355 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:34.356 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:34.356 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:34.356 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:34.356 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:26:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:26:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:26:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:26:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:26:34Z" 2022-09-13T07:26:34.358 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:26:34.360 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:26:39.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:39.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:39.362 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:26:39.418 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:26:39.419 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:26:39.456 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:39.452+0000 7fed7fec2600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:39.457 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:39.453+0000 7fed7fec2600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:39.477 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:26:39.478 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:26:39.478 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:26:39.478 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:26:39.479 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:26:39.479 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:26:39.479 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:39.479 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:39.480 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:26:39.480 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:39.480 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:26:39.481 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:39.481 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:39.481 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:39.481 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:39.482 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:26:39.482 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:39.482 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:39.482 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:39.483 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:39.483 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:39.483 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:39.483 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:26:39.484 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:39.484 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:39.484 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:39.484 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:39.484 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:39.485 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:39.485 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:39.485 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:39.485 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:39.486 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:39.486 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:39.486 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:26:39.486 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:39.486 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:39.487 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:39.487 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:39.487 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:39.487 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:39.488 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:39.488 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:39.488 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:39.488 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:39.488 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:39.489 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:26:39.489 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:39.489 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:39.489 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:39.490 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:39.490 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:39.490 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:39.491 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:39.491 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:39.491 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:39.491 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:39.491 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:26:39.492 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:39.492 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:26:39.492 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:26:39.492 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:26:39.492 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:39.493 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:26:39Z" 2022-09-13T07:26:39.493 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:26:39.494 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:26:44.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:44.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:44.497 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:26:44.554 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:26:44.554 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:26:44.584 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:44.580+0000 7f7f85b74600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:44.584 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:44.581+0000 7f7f85b74600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:44.596 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:26:44.597 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:26:44.597 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:26:44.597 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:26:44.598 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:26:44.598 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:26:44.598 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:44.598 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:44.599 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:26:44.599 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:44.599 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:26:44.599 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:44.600 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:44.600 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:44.600 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:44.601 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:26:44.601 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:44.601 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:44.601 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:44.601 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:44.602 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:44.602 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:44.602 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:26:44.603 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:44.603 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:44.603 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:44.603 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:44.604 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:44.604 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:44.604 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:44.604 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:44.604 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:44.605 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:44.605 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:44.605 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:26:44.605 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:44.606 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:44.606 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:44.606 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:44.606 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:44.607 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:44.607 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:44.607 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:44.608 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:44.608 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:44.608 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:44.608 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:26:44.609 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:44.609 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:44.609 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:44.609 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:44.610 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:44.610 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:44.610 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:44.610 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:44.611 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:44.611 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:44.611 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:26:44.611 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:44.612 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:26:44.612 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:26:44.612 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:26:44.612 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:44.612 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:26:44Z" 2022-09-13T07:26:44.613 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:26:44.615 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:26:49.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:49.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:49.618 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:26:49.675 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:26:49.675 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:26:49.718 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:49.714+0000 7f1a952c2600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:49.719 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:49.715+0000 7f1a952c2600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:49.733 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:26:49.733 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:26:49.733 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:26:49.734 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:26:49.734 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:26:49.734 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:26:49.734 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:49.735 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:49.735 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:26:49.735 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:49.735 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:26:49.736 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:49.736 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:49.736 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:49.736 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:49.737 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:26:49.737 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:49.737 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:49.738 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:49.738 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:49.738 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:49.738 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:49.739 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:26:49.739 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:49.739 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:49.739 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:49.740 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:49.740 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:49.740 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:49.740 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:49.741 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:49.741 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:49.741 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:49.741 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:49.741 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:26:49.742 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:49.742 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:49.742 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:49.743 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:49.743 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:49.743 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:49.743 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:49.744 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:49.744 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:49.744 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:49.744 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:49.744 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:26:49.745 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:49.745 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:49.745 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:49.746 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:49.746 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:49.746 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:49.746 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:49.747 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:49.747 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:49.747 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:49.747 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:26:49.748 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:49.748 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:26:49.748 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:26:49.749 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:26:49.749 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:49.749 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:26:49Z" 2022-09-13T07:26:49.749 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:26:49.751 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:26:54.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:54.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:54.754 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:26:54.811 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:26:54.811 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:26:54.841 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:54.838+0000 7f49f470e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:54.843 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:54.839+0000 7f49f470e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:54.854 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:26:54.854 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:26:54.855 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:26:54.855 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:26:54.855 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:26:54.855 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:26:54.855 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:54.856 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:54.856 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:26:54.856 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:54.856 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:26:54.857 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:54.857 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:54.857 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:54.858 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:54.859 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:26:54.859 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:54.859 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:54.859 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:54.860 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:54.860 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:54.860 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:54.860 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:26:54.861 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:54.861 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:54.861 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:54.861 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:54.862 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:54.862 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:54.862 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:54.862 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:54.863 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:54.863 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:54.863 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:54.863 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:26:54.864 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:54.864 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:54.864 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:54.864 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:54.864 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:54.865 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:54.865 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:54.865 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:54.865 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:54.866 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:54.866 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:26:54.866 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:26:54.866 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:26:54.867 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:26:54.867 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:26:54.867 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:26:54.867 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:26:54.868 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:26:54.868 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:26:54.868 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:26:54.868 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:54.869 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:26:54.869 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:26:54.869 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:54.869 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:26:54.869 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:26:54.870 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:26:54.870 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:26:54.870 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:26:54Z" 2022-09-13T07:26:54.870 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:26:54.872 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:26:59.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:59.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:26:59.875 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:26:59.931 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:26:59.932 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:26:59.983 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:59.981+0000 7faac7b4e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:26:59.984 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:26:59.982+0000 7faac7b4e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:00.007 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:27:00.007 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:27:00.007 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:27:00.007 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:27:00.008 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:27:00.008 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:27:00.008 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:00.008 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:00.009 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:27:00.009 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:00.009 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:27:00.009 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:00.010 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:00.010 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:00.010 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:00.010 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:27:00.011 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:00.011 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:00.011 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:00.011 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:00.011 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:00.012 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:00.012 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:27:00.012 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:00.013 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:00.013 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:00.013 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:00.013 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:00.014 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:00.014 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:00.014 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:00.014 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:00.015 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:00.015 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:00.015 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:27:00.015 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:00.015 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:00.016 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:00.016 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:00.016 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:00.016 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:00.017 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:00.017 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:00.017 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:00.017 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:00.018 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:00.018 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:27:00.018 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:00.018 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:00.019 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:00.019 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:00.019 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:00.019 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:00.020 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:00.020 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:00.020 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:00.020 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:00.021 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:27:00.021 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:00.021 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:27:00.021 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:27:00.021 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:27:00.022 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:00.022 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:26:59Z" 2022-09-13T07:27:00.022 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:27:00.023 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:27:05.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:05.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:05.026 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:05.084 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:05.085 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:05.114 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:05.111+0000 7f8d1d455600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:05.115 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:05.113+0000 7f8d1d455600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:05.128 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:27:05.128 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:27:05.128 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:27:05.129 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:27:05.129 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:27:05.130 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:27:05.130 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:05.131 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:05.131 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:27:05.131 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:05.132 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:27:05.132 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:05.132 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:05.132 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:05.133 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:05.133 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:27:05.133 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:05.133 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:05.134 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:05.134 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:05.134 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:05.134 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:05.135 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:27:05.135 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:05.135 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:05.135 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:05.135 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:05.136 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:05.136 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:05.136 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:05.136 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:05.137 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:05.137 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:05.137 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:05.138 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:27:05.138 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:05.138 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:05.138 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:05.139 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:05.139 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:05.139 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:05.139 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:05.140 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:05.140 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:05.141 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:05.141 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:05.141 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:27:05.142 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:05.142 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:05.142 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:05.143 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:05.143 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:05.144 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:05.144 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:05.144 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:05.145 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:05.145 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:05.145 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:27:05.146 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:05.146 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:27:05.147 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:27:05.147 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:27:05.147 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:05.148 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:27:05Z" 2022-09-13T07:27:05.148 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:27:05.149 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:27:10.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:10.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:10.151 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:10.208 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:10.209 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:10.249 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:10.246+0000 7f331c6cc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:10.247+0000 7f331c6cc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:10.270 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:27:10.271 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:27:10.271 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:27:10.271 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:10.273 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:10.273 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:27:10.273 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:10.273 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:27:10.274 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:10.274 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:10.274 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:10.274 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:10.275 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:27:10.275 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:10.276 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:10.276 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:10.276 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:10.276 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:10.277 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:10.277 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:27:10.277 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:10.277 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:10.278 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:10.278 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:10.278 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:10.278 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:10.279 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:10.279 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:10.279 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:10.279 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:10.280 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:10.280 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:27:10.280 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:10.280 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:10.280 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:10.281 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:10.281 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:10.281 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:10.281 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:10.282 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:10.282 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:10.282 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:10.282 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:10.283 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:27:10.283 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:10.283 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:10.283 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:10.283 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:10.284 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:10.284 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:10.285 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:10.285 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:10.285 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:10.286 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:10.286 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:27:10.287 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:10.287 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:27:10.288 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:27:10.288 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:27:10.289 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:10.289 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:27:10Z" 2022-09-13T07:27:10.289 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:27:10.292 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:27:15.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:15.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:15.294 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:15.352 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:15.352 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:15.402 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:15.399+0000 7f41dfde3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:15.404 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:15.400+0000 7f41dfde3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:15.424 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:27:15.425 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:27:15.425 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:27:15.426 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:27:15.426 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:27:15.427 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:27:15.427 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:15.427 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:15.428 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:27:15.428 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:15.429 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:27:15.429 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:15.430 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:15.431 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:15.432 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:15.432 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:27:15.432 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:15.433 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:15.433 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:15.434 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:15.434 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:15.435 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:15.435 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:27:15.436 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:15.436 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:15.436 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:15.437 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:15.437 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:15.438 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:15.438 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:15.439 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:15.439 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:15.440 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:15.440 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:15.440 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:27:15.441 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:15.441 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:15.442 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:15.443 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:15.443 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:15.444 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:15.444 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:15.444 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:15.444 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:15.444 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:15.445 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:15.445 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:27:15.445 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:15.445 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:15.446 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:15.446 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:15.446 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:15.446 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:15.447 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:15.447 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:15.447 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:15.447 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:15.448 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:27:15.448 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:15.449 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:27:15.449 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:27:15.449 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:27:15.450 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:15.450 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:27:15Z" 2022-09-13T07:27:15.450 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:27:15.451 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:27:20.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:20.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:20.454 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:20.511 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:20.512 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:20.553 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:20.550+0000 7fd2e3ab6600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:20.554 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:20.551+0000 7fd2e3ab6600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:20.581 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:27:20.582 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:27:20.582 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:27:20.583 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:27:20.583 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:27:20.583 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:27:20.583 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:20.584 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:20.584 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:27:20.584 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:20.584 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:27:20.584 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:20.585 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:20.585 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:20.585 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:20.585 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:27:20.586 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:20.586 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:20.586 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:20.586 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:20.586 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:20.587 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:20.587 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:27:20.587 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:20.587 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:20.588 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:20.588 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:20.588 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:20.588 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:20.588 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:20.589 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:20.589 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:20.589 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:20.589 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:20.590 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:27:20.590 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:20.590 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:20.590 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:20.591 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:20.591 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:20.591 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:20.592 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:20.592 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:20.592 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:20.592 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:20.592 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:20.593 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:27:20.593 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:20.593 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:20.593 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:20.594 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:20.594 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:20.594 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:20.594 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:20.594 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:20.595 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:20.595 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:20.595 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:27:20.595 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:20.596 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:27:20.596 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:27:20.596 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:27:20.596 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:20.596 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:27:20Z" 2022-09-13T07:27:20.597 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:27:20.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:27:25.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:25.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:25.602 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:25.672 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:25.714 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:25.705+0000 7f6247d67600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:25.714 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:25.706+0000 7f6247d67600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:25.723 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:27:25.724 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:27:25.724 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:27:25.724 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:27:25.724 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:27:25.725 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:27:25.725 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:25.725 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:25.726 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:27:25.726 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:25.726 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:27:25.727 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:25.727 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:25.728 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:25.728 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:25.728 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:27:25.728 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:25.729 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:25.729 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:25.729 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:25.730 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:25.730 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:25.730 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:27:25.731 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:25.731 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:25.731 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:25.731 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:25.731 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:25.732 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:25.732 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:25.732 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:25.732 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:25.733 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:25.733 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:25.733 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:27:25.734 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:25.734 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:25.735 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:25.735 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:25.736 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:25.736 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:25.737 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:25.737 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:25.739 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:25.740 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:25.740 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:25.741 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:27:25.741 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:25.741 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:25.742 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:25.742 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:25.742 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:25.743 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:25.743 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:25.743 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:25.744 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:25.744 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:25.744 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:27:25.745 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:25.745 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:27:25.746 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:27:25.746 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:27:25.746 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:25.747 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:27:25Z" 2022-09-13T07:27:25.747 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:27:25.749 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:27:30.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:30.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:30.752 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:30.809 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:30.810 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:30.853 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:30.850+0000 7f05b0a51600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:30.854 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:30.851+0000 7f05b0a51600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:30.872 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:27:30.872 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:27:30.873 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:27:30.873 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:27:30.873 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:27:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:27:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:27:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:27:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:27:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:27:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:27:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:27:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:30.886 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:30.886 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:30.886 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:30.886 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:27:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:30.888 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:27:30.888 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:27:30.888 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:27:30.888 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:30.889 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:27:30Z" 2022-09-13T07:27:30.889 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:27:30.890 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:27:35.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:35.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:35.893 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:35.950 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:35.951 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:35.988 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:35.985+0000 7f624fbc9600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:35.989 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:35.987+0000 7f624fbc9600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:36.001 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:27:36.002 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:27:36.002 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:27:36.002 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:27:36.002 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:27:36.003 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:27:36.003 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:36.003 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:36.003 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:27:36.004 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:36.004 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:27:36.004 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:36.004 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:36.004 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:36.005 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:36.005 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:27:36.005 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:36.005 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:36.006 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:36.006 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:36.006 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:36.006 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:36.006 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:27:36.007 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:36.007 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:36.007 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:36.007 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:36.008 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:36.008 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:36.008 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:36.008 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:36.009 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:36.009 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:36.009 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:36.010 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:27:36.010 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:36.010 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:36.010 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:36.010 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:36.011 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:36.011 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:36.011 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:36.011 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:36.012 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:36.012 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:36.012 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:36.012 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:27:36.012 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:36.013 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:36.013 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:36.013 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:36.013 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:36.014 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:36.014 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:36.014 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:36.014 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:36.015 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:36.015 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:27:36.015 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:36.015 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:27:36.015 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:27:36.016 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:27:36.016 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:36.016 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:27:35Z" 2022-09-13T07:27:36.016 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:27:36.018 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:27:41.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:41.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:41.020 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:41.077 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:41.078 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:41.109+0000 7fd28808a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:41.112 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:41.110+0000 7fd28808a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:41.134 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:27:41.135 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:27:41.135 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:27:41.136 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:27:41.136 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:27:41.137 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:27:41.137 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:41.138 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:41.138 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:27:41.138 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:41.139 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:27:41.139 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:41.140 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:41.140 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:41.140 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:41.140 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:27:41.141 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:41.141 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:41.141 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:41.141 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:41.142 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:41.142 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:41.142 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:27:41.142 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:41.143 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:41.143 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:41.143 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:41.143 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:41.143 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:41.144 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:41.144 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:41.144 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:41.144 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:41.145 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:41.145 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:27:41.145 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:41.145 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:41.145 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:41.146 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:41.146 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:41.146 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:41.146 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:41.147 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:41.147 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:41.147 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:41.147 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:41.147 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:27:41.148 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:41.148 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:41.148 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:41.148 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:41.148 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:41.149 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:41.149 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:41.149 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:41.149 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:41.150 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:41.150 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:27:41.150 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:41.150 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:27:41.151 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:27:41.151 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:27:41.151 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:41.151 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:27:41Z" 2022-09-13T07:27:41.151 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:27:41.153 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:27:46.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:46.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:46.155 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:46.212 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:46.212 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:46.254 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:46.251+0000 7f506c694600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:46.255 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:46.252+0000 7f506c694600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:46.273 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:27:46.273 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:27:46.274 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:27:46.274 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:27:46.274 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:27:46.275 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:27:46.275 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:46.276 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:46.276 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:27:46.276 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:46.277 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:27:46.277 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:46.278 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:46.278 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:46.279 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:46.279 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:27:46.279 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:46.280 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:46.280 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:46.281 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:46.281 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:46.282 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:46.282 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:27:46.283 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:46.283 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:46.283 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:46.284 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:46.284 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:46.284 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:46.284 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:46.285 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:46.285 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:46.285 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:46.285 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:46.286 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:27:46.286 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:46.286 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:46.286 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:46.287 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:46.287 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:46.287 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:46.287 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:46.288 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:46.288 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:46.288 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:46.288 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:46.288 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:27:46.289 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:46.289 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:46.289 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:46.289 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:46.290 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:46.290 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:46.290 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:46.290 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:46.290 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:46.291 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:46.291 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:27:46.291 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:46.291 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:27:46.292 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:27:46.292 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:27:46.292 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:46.292 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:27:46Z" 2022-09-13T07:27:46.293 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:27:46.294 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663053952.976234_34.1 target= 2022-09-13T07:27:51.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:51.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:51.298 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:51.355 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:51.355 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:51.386 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:51.382+0000 7f2dd5a17600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:51.388 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:51.383+0000 7f2dd5a17600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:51.398 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:27:51.399 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:27:51.399 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:27:51.399 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:27:51.399 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:27:51.400 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:27:51.400 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:51.400 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:51.400 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:27:51.401 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:51.401 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:27:51.401 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:51.401 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:51.401 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:27:51.402 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:51.402 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:27:51.402 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:51.402 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:27:51.403 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:51.403 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:51.403 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:51.403 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:51.404 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:27:51.404 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:51.404 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:51.404 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:51.405 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:51.405 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:51.405 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:51.405 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:51.405 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:51.406 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:51.406 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:51.406 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:51.406 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:27:51.407 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:51.407 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:51.407 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:51.407 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:51.408 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:51.408 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:51.408 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:51.408 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:51.408 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:51.409 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:51.409 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:51.409 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:27:51.409 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:51.410 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:51.410 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:51.411 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:51.411 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:51.411 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:27:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:27:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:27:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:27:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:27:51Z" 2022-09-13T07:27:51.414 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:27:51.416 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:27:51.416 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:27:51.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:51.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:51.418 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:51.480 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:51.480 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:51.519 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:27:51.515+0000 7f191f2c6600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:51.520 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:27:51.517+0000 7f191f2c6600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:51.538 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:27:51.538 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:27:51.539 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:27:51.539 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:27:51.539 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:27:51.540 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:27:51.540 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:27:51.540 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:51.540 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:27:51.540 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:51.541 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:27:51.541 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:27:51.541 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:27:51.541 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:27:51.542 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:27:51.542 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:27:51.542 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:27:51.542 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:27:51.543 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:27:51.543 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:27:51.543 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:51.543 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:51.543 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:27:51.544 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:27:51.544 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:27:51.544 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:27:51.544 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:27:51.545 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:27:51.545 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:27:51.545 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:27:51.545 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:27:51.545 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:27:51.546 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:51.546 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:51.546 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:27:51.547 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:27:51.547 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:27:51.547 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:27:51.547 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:27:51.548 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:27:51.548 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:27:51.548 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:27:51.548 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:27:51.549 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:27:51.549 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:51.549 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:51.549 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:27:51.549 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:27:51.550 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:27:51.550 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:27:51.550 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:27:51.550 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:27:51.551 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:27:51.551 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:27:51.552 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:27:51.552 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:27:51.552 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:27:51.552 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:27:51.553 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:51.553 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:27:51.553 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:27:51.553 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:27:51.553 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:51.554 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:27:51Z" 2022-09-13T07:27:51.554 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:27:51.555 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:27:51.556 INFO:rgw_multi.tests:meta checkpoint 2022-09-13T07:27:51.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:51.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:51.557 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:51.616 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:51.616 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:51.641 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:27:51.658 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:27:51.659 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:51.659 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:27:51.660 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:27:51.660 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:51.660 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:51.661 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:27:51.661 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:27:51.661 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:51.662 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:51.662 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:27:51.662 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:27:51.663 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:51.663 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:51.664 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:27:51.664 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:27:51.665 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:27:51.665 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:27:51.668 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:27:51.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:51.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:51.670 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:51.720 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:51.720 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:51.771 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:51.766+0000 7f19dfc30600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:51.771 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:51.767+0000 7f19dfc30600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:51.793 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:27:51.794 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:27:51.795 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:27:51.795 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:27:51.796 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:27:51.796 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:27:51.796 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:51.796 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:51.797 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:27:51.797 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:51.797 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:27:51.798 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:51.798 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:51.799 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:27:51.799 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:51.799 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:27:51.799 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:51.800 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:27:51.800 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:51.800 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:51.801 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:51.801 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:51.801 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:27:51.802 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:51.802 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:51.802 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:51.803 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:51.803 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:51.803 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:51.803 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:51.804 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:51.804 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:51.804 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:51.804 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:51.805 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:27:51.805 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:51.805 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:51.805 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:51.806 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:51.806 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:51.807 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:51.807 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:51.807 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:51.808 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:51.808 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:51.808 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:51.809 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:27:51.809 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:51.809 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:51.809 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:51.809 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:51.810 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:51.810 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:51.811 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:51.811 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:51.811 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:51.811 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:51.812 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:27:51.812 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:51.812 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:27:51.812 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:27:51.813 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:27:51.813 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:51.813 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:27:51Z" 2022-09-13T07:27:51.813 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:27:51.815 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:27:51.815 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:27:51.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:51.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:51.817 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:51.868 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:51.869 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:51.902 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:27:51.899+0000 7fdabb967600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:51.903 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:27:51.900+0000 7fdabb967600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:51.915 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:27:51.916 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:27:51.916 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:27:51.916 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:27:51.917 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:27:51.917 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:27:51.917 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:27:51.917 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:51.918 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:27:51.918 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:51.918 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:27:51.919 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:27:51.919 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:27:51.919 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:27:51.919 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:27:51.920 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:27:51.920 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:27:51.920 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:27:51.920 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:27:51.921 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:27:51.921 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:51.921 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:51.921 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:27:51.922 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:27:51.922 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:27:51.922 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:27:51.922 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:27:51.923 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:27:51.923 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:27:51.923 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:27:51.924 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:27:51.924 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:27:51.924 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:51.924 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:51.924 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:27:51.925 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:27:51.925 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:27:51.926 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:27:51.926 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:27:51.926 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:27:51.926 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:27:51.927 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:27:51.927 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:27:51.927 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:27:51.927 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:51.928 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:51.928 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:27:51.928 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:27:51.928 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:27:51.928 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:27:51.929 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:27:51.929 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:27:51.930 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:27:51.930 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:27:51.930 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:27:51.930 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:27:51.931 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:27:51.931 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:27:51.931 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:51.931 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:27:51.932 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:27:51.932 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:27:51.932 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:51.932 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:27:51Z" 2022-09-13T07:27:51.933 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:27:51.934 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:27:51.935 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2022-09-13T07:27:51.963 DEBUG:boto:Using access key provided by client. 2022-09-13T07:27:51.966 DEBUG:boto:Using secret key provided by client. 2022-09-13T07:27:51.967 DEBUG:boto:Using access key provided by client. 2022-09-13T07:27:51.968 DEBUG:boto:Using secret key provided by client. 2022-09-13T07:27:51.969 DEBUG:boto:Using access key provided by client. 2022-09-13T07:27:51.970 DEBUG:boto:Using secret key provided by client. 2022-09-13T07:27:51.971 DEBUG:boto:Using access key provided by client. 2022-09-13T07:27:51.971 DEBUG:boto:Using secret key provided by client. 2022-09-13T07:27:51.972 DEBUG:boto:Using access key provided by client. 2022-09-13T07:27:51.973 DEBUG:boto:Using secret key provided by client. 2022-09-13T07:27:51.974 DEBUG:boto:Using access key provided by client. 2022-09-13T07:27:51.974 DEBUG:boto:Using secret key provided by client. 2022-09-13T07:27:51.975 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-1 2022-09-13T07:27:51.976 DEBUG:boto:path=/yxvvuh-1/ 2022-09-13T07:27:51.976 DEBUG:boto:auth_path=/yxvvuh-1/ 2022-09-13T07:27:51.977 DEBUG:boto:Method: PUT 2022-09-13T07:27:51.978 DEBUG:boto:Path: /yxvvuh-1/ 2022-09-13T07:27:51.978 DEBUG:boto:Data: 2022-09-13T07:27:51.979 DEBUG:boto:Headers: {} 2022-09-13T07:27:51.979 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:27:51.980 DEBUG:boto:Port: 8000 2022-09-13T07:27:51.980 DEBUG:boto:Params: {} 2022-09-13T07:27:51.981 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T07:27:51.981 DEBUG:boto:Token: None 2022-09-13T07:27:51.982 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:27:51 GMT /yxvvuh-1/ 2022-09-13T07:27:51.983 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c8+x0UA05ut/tNP83VqlN6cqCuc= 2022-09-13T07:27:51.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:27:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c8+x0UA05ut/tNP83VqlN6cqCuc=', 'Content-Length': '0'} 2022-09-13T07:27:54.522 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d12bd31863b3f5a0-00632030f7-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:27:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:27:54.525 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-2 2022-09-13T07:27:54.526 DEBUG:boto:path=/yxvvuh-2/ 2022-09-13T07:27:54.527 DEBUG:boto:auth_path=/yxvvuh-2/ 2022-09-13T07:27:54.528 DEBUG:boto:Method: PUT 2022-09-13T07:27:54.529 DEBUG:boto:Path: /yxvvuh-2/ 2022-09-13T07:27:54.530 DEBUG:boto:Data: 2022-09-13T07:27:54.531 DEBUG:boto:Headers: {} 2022-09-13T07:27:54.532 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:27:54.533 DEBUG:boto:Port: 8000 2022-09-13T07:27:54.533 DEBUG:boto:Params: {} 2022-09-13T07:27:54.534 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T07:27:54.535 DEBUG:boto:Token: None 2022-09-13T07:27:54.536 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:27:54 GMT /yxvvuh-2/ 2022-09-13T07:27:54.536 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ytgxzuexehVJgoIIP6mFImo4/LA= 2022-09-13T07:27:54.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:27:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ytgxzuexehVJgoIIP6mFImo4/LA=', 'Content-Length': '0'} 2022-09-13T07:27:56.698 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007f0cffc7e0a71260-00632030fa-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:27:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:27:56.699 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-3 2022-09-13T07:27:56.700 DEBUG:boto:path=/yxvvuh-3/ 2022-09-13T07:27:56.700 DEBUG:boto:auth_path=/yxvvuh-3/ 2022-09-13T07:27:56.700 DEBUG:boto:Method: PUT 2022-09-13T07:27:56.701 DEBUG:boto:Path: /yxvvuh-3/ 2022-09-13T07:27:56.701 DEBUG:boto:Data: 2022-09-13T07:27:56.702 DEBUG:boto:Headers: {} 2022-09-13T07:27:56.702 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:27:56.703 DEBUG:boto:Port: 8001 2022-09-13T07:27:56.704 DEBUG:boto:Params: {} 2022-09-13T07:27:56.705 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-13T07:27:56.705 DEBUG:boto:Token: None 2022-09-13T07:27:56.706 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:27:56 GMT /yxvvuh-3/ 2022-09-13T07:27:56.707 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S0EVLj/H53KPjsZyEQZihTl0QiU= 2022-09-13T07:27:56.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:27:56 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S0EVLj/H53KPjsZyEQZihTl0QiU=', 'Content-Length': '0'} 2022-09-13T07:27:56.744 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d80ba4f1e8c8f20c-00632030fc-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:27:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:27:56.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:56.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:56.746 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:56.800 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:56.801 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:56.851 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:27:56.863 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:27:56.864 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:56.864 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:27:56.864 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:27:56.865 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:56.865 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:56.866 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:27:56.866 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:27:56.867 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:56.867 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:56.868 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:27:56.868 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:27:56.868 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:27:56.868 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:27:56.869 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054076.730183_41.1", 2022-09-13T07:27:56.869 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:27:56.730183Z" 2022-09-13T07:27:56.869 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:27:56.870 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:27:56.872 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054076.730183_41.1'} 2022-09-13T07:27:56.872 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:27:56.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:56.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:27:56.875 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:27:56.930 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:27:56.931 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:27:56.963 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:56.959+0000 7fd6d31e3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:56.964 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:27:56.960+0000 7fd6d31e3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:27:56.975 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:27:56.976 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:27:56.977 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:27:56.978 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:27:56.979 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:27:56.980 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:27:56.981 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:56.981 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:56.982 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:27:56.983 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:56.983 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:27:56.984 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:56.985 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:56.986 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:27:56.987 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:56.988 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:27:56.989 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:56.990 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:27:56.991 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:56.992 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:56.993 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:56.994 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:56.995 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:27:56.995 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:56.996 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:56.997 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:56.999 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:57.000 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:57.001 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:57.002 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:57.002 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:57.003 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:57.004 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:57.004 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:57.005 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:27:57.005 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:57.006 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:57.006 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:27:57.007 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:57.008 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:57.008 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:57.009 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:27:57.009 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:57.009 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:57.010 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:57.011 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:27:57.011 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:27:57.011 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:27:57.012 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:27:57.012 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054074.509319_30.1", 2022-09-13T07:27:57.013 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:27:57.013 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:27:57.014 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:27:57.014 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:27:54.509319Z", 2022-09-13T07:27:57.015 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:27:57.015 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:57.016 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:27:57.016 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:27:57.017 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:57.017 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:27:57.018 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:27:57.018 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:27:57.019 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:27:57.019 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:27:56Z" 2022-09-13T07:27:57.020 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:27:57.022 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054074.509319_30.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:27:54.509319Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:27:56Z" } 2022-09-13T07:27:57.022 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054074.509319_30.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:27:54.509319Z', 'realm_epoch': 2}}] 2022-09-13T07:27:57.023 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054076.730183_41.1'} 2022-09-13T07:27:57.023 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054074.509319_30.1'} 2022-09-13T07:27:57.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1663054076.730183_41.1 target=1_1663054074.509319_30.1 2022-09-13T07:28:02.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:02.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:02.027 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:02.085 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:02.123 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:02.119+0000 7fc7b4991600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:02.124 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:02.121+0000 7fc7b4991600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:02.141 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:28:02.141 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:28:02.142 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:28:02.142 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:28:02.143 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:28:02.143 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:02.144 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:02.144 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:02.145 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:28:02.145 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:02.146 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:28:02.146 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:02.147 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:02.147 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:02.148 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:02.148 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:28:02.149 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:02.149 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:02.150 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:02.150 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:02.152 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:28:02.152 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:02.156 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:02.156 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:28:02.159 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:02.159 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:02.160 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:02.160 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:02.161 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:02.161 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:02.162 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:02.162 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:02.163 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:02.163 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:02.164 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:02.164 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:28:02.165 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:02.165 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:02.166 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054076.730183_41.1", 2022-09-13T07:28:02.166 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:02.167 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:02.167 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:02.168 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:27:56.730183Z", 2022-09-13T07:28:02.168 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:02.169 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:02.169 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:02.170 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:28:02.170 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:02.171 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:28:02.171 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:28:02.172 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:28:02.172 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:02.173 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:28:02Z" 2022-09-13T07:28:02.174 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:28:02.176 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054076.730183_41.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:27:56.730183Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:02Z" } 2022-09-13T07:28:02.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054076.730183_41.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:27:56.730183Z', 'realm_epoch': 2}}] 2022-09-13T07:28:02.178 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054076.730183_41.1'} 2022-09-13T07:28:02.178 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054076.730183_41.1'} 2022-09-13T07:28:02.179 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:28:02.180 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:28:02.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:02.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:02.183 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:02.242 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:02.243 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:02.274 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:02.270+0000 7f2947fd9600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:02.275 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:02.271+0000 7f2947fd9600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:02.286 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:28:02.287 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:28:02.288 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:28:02.288 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:28:02.288 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:28:02.288 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:02.289 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:02.289 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:02.289 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:28:02.289 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:02.290 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:28:02.290 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:02.290 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:02.290 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:02.291 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:02.291 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:28:02.291 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:02.291 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:02.292 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:02.292 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:02.292 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:02.292 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:02.293 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:28:02.293 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:02.293 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:02.294 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:02.294 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:02.294 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:02.294 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:02.295 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:28:02.295 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:02.295 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:02.296 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:02.296 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:02.296 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:28:02.296 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:02.297 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:02.297 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:02.297 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:02.297 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:02.298 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:02.298 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:28:02.298 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:02.298 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:02.299 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:02.299 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:02.299 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:28:02.299 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:02.300 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:02.300 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054076.730183_41.1", 2022-09-13T07:28:02.301 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:02.301 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:02.301 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:02.301 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:27:56.730183Z", 2022-09-13T07:28:02.302 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:02.302 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:02.302 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:02.302 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:28:02.303 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:02.303 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:28:02.303 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:28:02.304 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:28:02.304 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:02.304 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:28:02Z" 2022-09-13T07:28:02.305 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:28:02.306 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054076.730183_41.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:27:56.730183Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:02Z" } 2022-09-13T07:28:02.307 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054076.730183_41.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:27:56.730183Z', 'realm_epoch': 2}}] 2022-09-13T07:28:02.307 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054076.730183_41.1'} 2022-09-13T07:28:02.308 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054076.730183_41.1'} 2022-09-13T07:28:02.308 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:28:02.310 DEBUG:boto:path=/yxvvuh-1/ 2022-09-13T07:28:02.311 DEBUG:boto:auth_path=/yxvvuh-1/ 2022-09-13T07:28:02.311 DEBUG:boto:Method: HEAD 2022-09-13T07:28:02.312 DEBUG:boto:Path: /yxvvuh-1/ 2022-09-13T07:28:02.312 DEBUG:boto:Data: 2022-09-13T07:28:02.313 DEBUG:boto:Headers: {} 2022-09-13T07:28:02.313 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:02.314 DEBUG:boto:Port: 8000 2022-09-13T07:28:02.315 DEBUG:boto:Params: {} 2022-09-13T07:28:02.315 DEBUG:boto:Token: None 2022-09-13T07:28:02.316 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:02 GMT /yxvvuh-1/ 2022-09-13T07:28:02.316 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EcmEWbMtfr8R/g67TgunQZ1+E+E= 2022-09-13T07:28:02.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:02 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EcmEWbMtfr8R/g67TgunQZ1+E+E=', 'Content-Length': '0'} 2022-09-13T07:28:02.320 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e23096ba3f56af8-0063203102-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:02.321 DEBUG:boto:path=/yxvvuh-2/ 2022-09-13T07:28:02.321 DEBUG:boto:auth_path=/yxvvuh-2/ 2022-09-13T07:28:02.322 DEBUG:boto:Method: HEAD 2022-09-13T07:28:02.322 DEBUG:boto:Path: /yxvvuh-2/ 2022-09-13T07:28:02.323 DEBUG:boto:Data: 2022-09-13T07:28:02.324 DEBUG:boto:Headers: {} 2022-09-13T07:28:02.324 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:02.324 DEBUG:boto:Port: 8000 2022-09-13T07:28:02.325 DEBUG:boto:Params: {} 2022-09-13T07:28:02.326 DEBUG:boto:Token: None 2022-09-13T07:28:02.326 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:02 GMT /yxvvuh-2/ 2022-09-13T07:28:02.327 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2etpUGrFMzqiHMgbSt9L3OXra70= 2022-09-13T07:28:02.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:02 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2etpUGrFMzqiHMgbSt9L3OXra70=', 'Content-Length': '0'} 2022-09-13T07:28:02.330 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000907a7ca0f130c947-0063203102-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:02.331 DEBUG:boto:path=/yxvvuh-3/ 2022-09-13T07:28:02.331 DEBUG:boto:auth_path=/yxvvuh-3/ 2022-09-13T07:28:02.332 DEBUG:boto:Method: HEAD 2022-09-13T07:28:02.332 DEBUG:boto:Path: /yxvvuh-3/ 2022-09-13T07:28:02.333 DEBUG:boto:Data: 2022-09-13T07:28:02.333 DEBUG:boto:Headers: {} 2022-09-13T07:28:02.334 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:02.334 DEBUG:boto:Port: 8000 2022-09-13T07:28:02.335 DEBUG:boto:Params: {} 2022-09-13T07:28:02.335 DEBUG:boto:Token: None 2022-09-13T07:28:02.336 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:02 GMT /yxvvuh-3/ 2022-09-13T07:28:02.336 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VPl3rLWbvcaUr7d15qQv4QuXuDk= 2022-09-13T07:28:02.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:02 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VPl3rLWbvcaUr7d15qQv4QuXuDk=', 'Content-Length': '0'} 2022-09-13T07:28:02.341 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ada7fa0f2799aea-0063203102-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:02.343 DEBUG:boto:path=/yxvvuh-1/ 2022-09-13T07:28:02.343 DEBUG:boto:auth_path=/yxvvuh-1/ 2022-09-13T07:28:02.344 DEBUG:boto:Method: HEAD 2022-09-13T07:28:02.344 DEBUG:boto:Path: /yxvvuh-1/ 2022-09-13T07:28:02.345 DEBUG:boto:Data: 2022-09-13T07:28:02.345 DEBUG:boto:Headers: {} 2022-09-13T07:28:02.346 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:02.346 DEBUG:boto:Port: 8000 2022-09-13T07:28:02.347 DEBUG:boto:Params: {} 2022-09-13T07:28:02.347 DEBUG:boto:Token: None 2022-09-13T07:28:02.348 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:02 GMT /yxvvuh-1/ 2022-09-13T07:28:02.349 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EcmEWbMtfr8R/g67TgunQZ1+E+E= 2022-09-13T07:28:02.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:02 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EcmEWbMtfr8R/g67TgunQZ1+E+E=', 'Content-Length': '0'} 2022-09-13T07:28:02.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c83a591f3c7bfdd4-0063203102-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:02.353 DEBUG:boto:path=/yxvvuh-2/ 2022-09-13T07:28:02.353 DEBUG:boto:auth_path=/yxvvuh-2/ 2022-09-13T07:28:02.354 DEBUG:boto:Method: HEAD 2022-09-13T07:28:02.354 DEBUG:boto:Path: /yxvvuh-2/ 2022-09-13T07:28:02.355 DEBUG:boto:Data: 2022-09-13T07:28:02.355 DEBUG:boto:Headers: {} 2022-09-13T07:28:02.356 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:02.356 DEBUG:boto:Port: 8000 2022-09-13T07:28:02.357 DEBUG:boto:Params: {} 2022-09-13T07:28:02.357 DEBUG:boto:Token: None 2022-09-13T07:28:02.358 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:02 GMT /yxvvuh-2/ 2022-09-13T07:28:02.358 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2etpUGrFMzqiHMgbSt9L3OXra70= 2022-09-13T07:28:02.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:02 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2etpUGrFMzqiHMgbSt9L3OXra70=', 'Content-Length': '0'} 2022-09-13T07:28:02.361 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af1f0e4d834e8bb8-0063203102-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:02.362 DEBUG:boto:path=/yxvvuh-3/ 2022-09-13T07:28:02.363 DEBUG:boto:auth_path=/yxvvuh-3/ 2022-09-13T07:28:02.363 DEBUG:boto:Method: HEAD 2022-09-13T07:28:02.365 DEBUG:boto:Path: /yxvvuh-3/ 2022-09-13T07:28:02.366 DEBUG:boto:Data: 2022-09-13T07:28:02.368 DEBUG:boto:Headers: {} 2022-09-13T07:28:02.369 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:02.370 DEBUG:boto:Port: 8000 2022-09-13T07:28:02.371 DEBUG:boto:Params: {} 2022-09-13T07:28:02.372 DEBUG:boto:Token: None 2022-09-13T07:28:02.373 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:02 GMT /yxvvuh-3/ 2022-09-13T07:28:02.375 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VPl3rLWbvcaUr7d15qQv4QuXuDk= 2022-09-13T07:28:02.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:02 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VPl3rLWbvcaUr7d15qQv4QuXuDk=', 'Content-Length': '0'} 2022-09-13T07:28:02.384 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b7fcbcd183cd8d3-0063203102-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:02.385 DEBUG:boto:path=/yxvvuh-1/ 2022-09-13T07:28:02.386 DEBUG:boto:auth_path=/yxvvuh-1/ 2022-09-13T07:28:02.387 DEBUG:boto:Method: HEAD 2022-09-13T07:28:02.387 DEBUG:boto:Path: /yxvvuh-1/ 2022-09-13T07:28:02.388 DEBUG:boto:Data: 2022-09-13T07:28:02.388 DEBUG:boto:Headers: {} 2022-09-13T07:28:02.389 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:02.389 DEBUG:boto:Port: 8001 2022-09-13T07:28:02.390 DEBUG:boto:Params: {} 2022-09-13T07:28:02.390 DEBUG:boto:Token: None 2022-09-13T07:28:02.391 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:02 GMT /yxvvuh-1/ 2022-09-13T07:28:02.392 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EcmEWbMtfr8R/g67TgunQZ1+E+E= 2022-09-13T07:28:02.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:02 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EcmEWbMtfr8R/g67TgunQZ1+E+E=', 'Content-Length': '0'} 2022-09-13T07:28:02.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000053e5d2d6db57eae-0063203102-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:02.395 DEBUG:boto:path=/yxvvuh-2/ 2022-09-13T07:28:02.396 DEBUG:boto:auth_path=/yxvvuh-2/ 2022-09-13T07:28:02.396 DEBUG:boto:Method: HEAD 2022-09-13T07:28:02.397 DEBUG:boto:Path: /yxvvuh-2/ 2022-09-13T07:28:02.397 DEBUG:boto:Data: 2022-09-13T07:28:02.398 DEBUG:boto:Headers: {} 2022-09-13T07:28:02.398 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:02.399 DEBUG:boto:Port: 8001 2022-09-13T07:28:02.399 DEBUG:boto:Params: {} 2022-09-13T07:28:02.400 DEBUG:boto:Token: None 2022-09-13T07:28:02.400 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:02 GMT /yxvvuh-2/ 2022-09-13T07:28:02.401 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2etpUGrFMzqiHMgbSt9L3OXra70= 2022-09-13T07:28:02.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:02 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2etpUGrFMzqiHMgbSt9L3OXra70=', 'Content-Length': '0'} 2022-09-13T07:28:02.404 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfddd189185b124e-0063203102-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:02.405 DEBUG:boto:path=/yxvvuh-3/ 2022-09-13T07:28:02.405 DEBUG:boto:auth_path=/yxvvuh-3/ 2022-09-13T07:28:02.406 DEBUG:boto:Method: HEAD 2022-09-13T07:28:02.406 DEBUG:boto:Path: /yxvvuh-3/ 2022-09-13T07:28:02.407 DEBUG:boto:Data: 2022-09-13T07:28:02.407 DEBUG:boto:Headers: {} 2022-09-13T07:28:02.408 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:02.408 DEBUG:boto:Port: 8001 2022-09-13T07:28:02.409 DEBUG:boto:Params: {} 2022-09-13T07:28:02.409 DEBUG:boto:Token: None 2022-09-13T07:28:02.410 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:02 GMT /yxvvuh-3/ 2022-09-13T07:28:02.411 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VPl3rLWbvcaUr7d15qQv4QuXuDk= 2022-09-13T07:28:02.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:02 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VPl3rLWbvcaUr7d15qQv4QuXuDk=', 'Content-Length': '0'} 2022-09-13T07:28:02.413 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067ccc322490c808a-0063203102-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:02.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2022-09-13T07:28:02.418 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-4 2022-09-13T07:28:02.419 DEBUG:boto:path=/yxvvuh-4/ 2022-09-13T07:28:02.419 DEBUG:boto:auth_path=/yxvvuh-4/ 2022-09-13T07:28:02.420 DEBUG:boto:Method: PUT 2022-09-13T07:28:02.421 DEBUG:boto:Path: /yxvvuh-4/ 2022-09-13T07:28:02.422 DEBUG:boto:Data: 2022-09-13T07:28:02.423 DEBUG:boto:Headers: {} 2022-09-13T07:28:02.424 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:02.424 DEBUG:boto:Port: 8000 2022-09-13T07:28:02.425 DEBUG:boto:Params: {} 2022-09-13T07:28:02.426 DEBUG:boto:Token: None 2022-09-13T07:28:02.427 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:02 GMT /yxvvuh-4/ 2022-09-13T07:28:02.428 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CB/zRO9bWZfNUOeD4lwxKRzrrjE= 2022-09-13T07:28:02.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:02 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CB/zRO9bWZfNUOeD4lwxKRzrrjE=', 'Content-Length': '0'} 2022-09-13T07:28:02.451 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004f1900d0a8a6457f-0063203102-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:02.452 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-5 2022-09-13T07:28:02.453 DEBUG:boto:path=/yxvvuh-5/ 2022-09-13T07:28:02.453 DEBUG:boto:auth_path=/yxvvuh-5/ 2022-09-13T07:28:02.454 DEBUG:boto:Method: PUT 2022-09-13T07:28:02.454 DEBUG:boto:Path: /yxvvuh-5/ 2022-09-13T07:28:02.455 DEBUG:boto:Data: 2022-09-13T07:28:02.455 DEBUG:boto:Headers: {} 2022-09-13T07:28:02.457 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:02.457 DEBUG:boto:Port: 8000 2022-09-13T07:28:02.457 DEBUG:boto:Params: {} 2022-09-13T07:28:02.458 DEBUG:boto:Token: None 2022-09-13T07:28:02.459 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:02 GMT /yxvvuh-5/ 2022-09-13T07:28:02.460 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i5aYUar6LNf2fy8E5FLf5ICRtH4= 2022-09-13T07:28:02.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:02 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i5aYUar6LNf2fy8E5FLf5ICRtH4=', 'Content-Length': '0'} 2022-09-13T07:28:02.495 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cbbf99cd7119ff62-0063203102-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:02.496 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-6 2022-09-13T07:28:02.497 DEBUG:boto:path=/yxvvuh-6/ 2022-09-13T07:28:02.497 DEBUG:boto:auth_path=/yxvvuh-6/ 2022-09-13T07:28:02.498 DEBUG:boto:Method: PUT 2022-09-13T07:28:02.498 DEBUG:boto:Path: /yxvvuh-6/ 2022-09-13T07:28:02.499 DEBUG:boto:Data: 2022-09-13T07:28:02.499 DEBUG:boto:Headers: {} 2022-09-13T07:28:02.500 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:02.501 DEBUG:boto:Port: 8001 2022-09-13T07:28:02.501 DEBUG:boto:Params: {} 2022-09-13T07:28:02.502 DEBUG:boto:Token: None 2022-09-13T07:28:02.502 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:02 GMT /yxvvuh-6/ 2022-09-13T07:28:02.503 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0DVvtdi7VOGJuc+vPdTF+GBv28Q= 2022-09-13T07:28:02.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:02 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0DVvtdi7VOGJuc+vPdTF+GBv28Q=', 'Content-Length': '0'} 2022-09-13T07:28:02.536 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006622f8776b42aa0d-0063203102-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:02.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:02.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:02.539 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:02.598 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:02.599 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:02.631 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:28:02.653 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:28:02.654 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:02.654 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:02.655 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:28:02.655 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:02.656 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:02.656 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:02.657 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:28:02.657 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:02.658 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:02.658 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:02.658 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:28:02.659 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:02.659 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:02.660 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054082.522898_53.1", 2022-09-13T07:28:02.660 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:02.522898Z" 2022-09-13T07:28:02.661 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:02.661 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:28:02.663 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054082.522898_53.1'} 2022-09-13T07:28:02.664 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:28:02.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:02.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:02.667 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:02.727 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:02.728 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:02.746 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:02.742+0000 7fafed641600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:02.747 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:02.743+0000 7fafed641600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:02.758 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:28:02.759 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:28:02.759 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:28:02.760 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:28:02.760 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:28:02.760 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:02.760 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:02.761 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:02.761 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:28:02.761 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:02.762 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:28:02.762 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:02.762 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:02.763 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:02.763 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:02.763 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:28:02.763 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:02.764 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:02.764 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:02.764 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:02.764 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:02.765 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:02.765 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:28:02.765 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:02.766 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:02.766 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:02.766 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:02.766 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:02.767 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:02.767 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:02.767 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:02.768 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:02.768 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:02.768 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:02.768 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:28:02.769 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:02.769 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:02.769 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:02.769 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:02.770 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:02.770 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:02.770 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:02.771 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:02.771 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:02.771 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:02.771 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:02.772 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:28:02.772 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:02.772 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:02.772 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054082.516114_51.1", 2022-09-13T07:28:02.773 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:02.773 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:02.773 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:02.773 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:02.516114Z", 2022-09-13T07:28:02.774 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:02.774 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:02.774 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:02.775 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:28:02.775 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:02.776 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:28:02.776 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:28:02.776 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:28:02.777 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:02.777 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:28:02Z" 2022-09-13T07:28:02.777 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:28:02.780 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054082.516114_51.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:02.516114Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:02Z" } 2022-09-13T07:28:02.781 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054082.516114_51.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:02.516114Z', 'realm_epoch': 2}}] 2022-09-13T07:28:02.782 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054082.522898_53.1'} 2022-09-13T07:28:02.782 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054082.516114_51.1'} 2022-09-13T07:28:02.783 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1663054082.522898_53.1 target=1_1663054082.516114_51.1 2022-09-13T07:28:07.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:07.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:07.786 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:07.844 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:07.845 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:07.889 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:07.885+0000 7fba4b3b8600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:07.890 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:07.887+0000 7fba4b3b8600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:07.913 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:28:07.913 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:28:07.913 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:28:07.914 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:28:07.914 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:28:07.914 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:07.915 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:07.915 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:07.916 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:28:07.917 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:07.917 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:28:07.917 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:07.917 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:07.918 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:07.918 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:07.918 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:28:07.919 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:07.919 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:07.919 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:07.920 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:07.920 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:07.920 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:07.920 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:28:07.921 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:07.921 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:07.921 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:07.921 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:07.922 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:07.922 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:07.922 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:07.923 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:07.923 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:07.923 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:07.923 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:07.924 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:28:07.924 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:07.924 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:07.925 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:07.925 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:07.925 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:07.925 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:07.926 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:07.926 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:07.926 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:07.926 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:07.927 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:07.927 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:28:07.927 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:07.927 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:07.928 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054082.522898_53.1", 2022-09-13T07:28:07.928 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:07.928 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:07.928 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:07.929 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:02.522898Z", 2022-09-13T07:28:07.929 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:07.929 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:07.930 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:07.930 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:28:07.930 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:07.930 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:28:07.931 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:28:07.931 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:28:07.931 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:07.931 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:28:07Z" 2022-09-13T07:28:07.932 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:28:07.933 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054082.522898_53.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:02.522898Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:07Z" } 2022-09-13T07:28:07.934 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054082.522898_53.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:02.522898Z', 'realm_epoch': 2}}] 2022-09-13T07:28:07.934 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054082.522898_53.1'} 2022-09-13T07:28:07.935 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054082.522898_53.1'} 2022-09-13T07:28:07.935 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:28:07.936 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:28:07.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:07.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:07.938 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:07.998 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:07.998 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:08.029 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:08.026+0000 7f407e3f5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:08.030 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:08.027+0000 7f407e3f5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:08.044 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:28:08.045 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:28:08.046 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:28:08.046 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:28:08.047 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:28:08.047 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:08.047 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:08.048 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:08.048 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:28:08.048 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:08.048 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:28:08.049 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:08.049 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:08.049 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:08.050 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:08.050 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:28:08.051 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:08.052 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:08.052 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:08.053 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:08.053 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:08.054 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:08.054 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:28:08.055 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:08.055 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:08.056 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:08.057 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:08.059 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:08.060 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:08.060 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:28:08.061 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:08.061 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:08.062 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:08.062 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:08.063 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:28:08.063 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:08.064 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:08.064 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:08.064 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:08.065 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:08.065 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:08.066 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:28:08.067 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:08.067 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:08.068 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:08.069 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:08.069 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:28:08.070 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:08.070 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:08.070 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054082.516114_51.1", 2022-09-13T07:28:08.071 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:08.071 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:08.072 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:08.073 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:02.516114Z", 2022-09-13T07:28:08.073 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:08.074 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:08.074 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:08.075 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:28:08.075 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:08.076 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:28:08.076 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:28:08.077 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:28:08.078 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:08.078 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:28:08Z" 2022-09-13T07:28:08.079 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:28:08.081 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054082.516114_51.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:02.516114Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:08Z" } 2022-09-13T07:28:08.082 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054082.516114_51.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:02.516114Z', 'realm_epoch': 2}}] 2022-09-13T07:28:08.083 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054082.522898_53.1'} 2022-09-13T07:28:08.084 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054082.516114_51.1'} 2022-09-13T07:28:08.085 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1663054082.522898_53.1 target=1_1663054082.516114_51.1 2022-09-13T07:28:13.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:13.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:13.089 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:13.149 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:13.232 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:13.234 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:13.194+0000 7f99209c6600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:13.234 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:13.196+0000 7f99209c6600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:13.235 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:28:13.236 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:28:13.236 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:28:13.236 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:28:13.236 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:28:13.237 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:13.237 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:13.237 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:13.237 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:28:13.238 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:13.238 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:28:13.238 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:13.239 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:13.239 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:13.239 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:13.239 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:28:13.240 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:13.240 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:13.241 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:13.241 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:13.241 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:13.241 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:13.242 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:28:13.242 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:13.242 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:13.243 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:13.243 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:13.243 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:13.243 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:13.244 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:28:13.244 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:13.244 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:13.244 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:13.245 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:13.245 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:28:13.245 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:13.245 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:13.246 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:13.246 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:13.246 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:13.246 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:13.247 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:28:13.247 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:13.247 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:13.247 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:13.248 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:13.248 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:28:13.249 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:13.249 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:13.249 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054082.522898_53.1", 2022-09-13T07:28:13.250 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:13.250 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:13.251 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:13.251 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:02.522898Z", 2022-09-13T07:28:13.252 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:13.252 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:13.253 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:13.253 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:28:13.254 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:13.254 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:28:13.255 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:28:13.255 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:28:13.256 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:13.256 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:28:13Z" 2022-09-13T07:28:13.257 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:28:13.257 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054082.522898_53.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:02.522898Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:13Z" } 2022-09-13T07:28:13.258 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054082.522898_53.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:02.522898Z', 'realm_epoch': 2}}] 2022-09-13T07:28:13.259 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054082.522898_53.1'} 2022-09-13T07:28:13.260 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054082.522898_53.1'} 2022-09-13T07:28:13.261 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:28:13.265 DEBUG:boto:path=/yxvvuh-4/ 2022-09-13T07:28:13.266 DEBUG:boto:auth_path=/yxvvuh-4/ 2022-09-13T07:28:13.267 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.268 DEBUG:boto:Path: /yxvvuh-4/ 2022-09-13T07:28:13.269 DEBUG:boto:Data: 2022-09-13T07:28:13.270 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.271 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:13.273 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.273 DEBUG:boto:Params: {} 2022-09-13T07:28:13.275 DEBUG:boto:Token: None 2022-09-13T07:28:13.276 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-4/ 2022-09-13T07:28:13.277 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MZbbkCPWiiCkxrTDTcsq8OkgCj4= 2022-09-13T07:28:13.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MZbbkCPWiiCkxrTDTcsq8OkgCj4=', 'Content-Length': '0'} 2022-09-13T07:28:13.289 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009226c7e497821701-006320310d-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.290 DEBUG:boto:path=/yxvvuh-5/ 2022-09-13T07:28:13.291 DEBUG:boto:auth_path=/yxvvuh-5/ 2022-09-13T07:28:13.292 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.293 DEBUG:boto:Path: /yxvvuh-5/ 2022-09-13T07:28:13.299 DEBUG:boto:Data: 2022-09-13T07:28:13.300 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.300 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:13.301 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.301 DEBUG:boto:Params: {} 2022-09-13T07:28:13.302 DEBUG:boto:Token: None 2022-09-13T07:28:13.302 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-5/ 2022-09-13T07:28:13.303 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hnSei0sXDHpn+mj19cG44HPDLhY= 2022-09-13T07:28:13.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hnSei0sXDHpn+mj19cG44HPDLhY=', 'Content-Length': '0'} 2022-09-13T07:28:13.310 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b342d25502ffd5d-006320310d-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.311 DEBUG:boto:path=/yxvvuh-6/ 2022-09-13T07:28:13.312 DEBUG:boto:auth_path=/yxvvuh-6/ 2022-09-13T07:28:13.312 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.313 DEBUG:boto:Path: /yxvvuh-6/ 2022-09-13T07:28:13.313 DEBUG:boto:Data: 2022-09-13T07:28:13.314 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.314 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:13.315 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.315 DEBUG:boto:Params: {} 2022-09-13T07:28:13.316 DEBUG:boto:Token: None 2022-09-13T07:28:13.316 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-6/ 2022-09-13T07:28:13.317 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0N4PFomvzzMlblMn3A81MLY3MwA= 2022-09-13T07:28:13.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0N4PFomvzzMlblMn3A81MLY3MwA=', 'Content-Length': '0'} 2022-09-13T07:28:13.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000672496380973c63a-006320310d-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.327 DEBUG:boto:path=/yxvvuh-4/ 2022-09-13T07:28:13.328 DEBUG:boto:auth_path=/yxvvuh-4/ 2022-09-13T07:28:13.329 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.331 DEBUG:boto:Path: /yxvvuh-4/ 2022-09-13T07:28:13.332 DEBUG:boto:Data: 2022-09-13T07:28:13.333 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.334 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:13.335 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.336 DEBUG:boto:Params: {} 2022-09-13T07:28:13.337 DEBUG:boto:Token: None 2022-09-13T07:28:13.338 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-4/ 2022-09-13T07:28:13.340 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MZbbkCPWiiCkxrTDTcsq8OkgCj4= 2022-09-13T07:28:13.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MZbbkCPWiiCkxrTDTcsq8OkgCj4=', 'Content-Length': '0'} 2022-09-13T07:28:13.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039fc3287d0c2f216-006320310d-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.355 DEBUG:boto:path=/yxvvuh-5/ 2022-09-13T07:28:13.356 DEBUG:boto:auth_path=/yxvvuh-5/ 2022-09-13T07:28:13.357 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.358 DEBUG:boto:Path: /yxvvuh-5/ 2022-09-13T07:28:13.359 DEBUG:boto:Data: 2022-09-13T07:28:13.359 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.360 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:13.361 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.361 DEBUG:boto:Params: {} 2022-09-13T07:28:13.362 DEBUG:boto:Token: None 2022-09-13T07:28:13.363 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-5/ 2022-09-13T07:28:13.364 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hnSei0sXDHpn+mj19cG44HPDLhY= 2022-09-13T07:28:13.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hnSei0sXDHpn+mj19cG44HPDLhY=', 'Content-Length': '0'} 2022-09-13T07:28:13.371 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090689f5447c8ab98-006320310d-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.373 DEBUG:boto:path=/yxvvuh-6/ 2022-09-13T07:28:13.373 DEBUG:boto:auth_path=/yxvvuh-6/ 2022-09-13T07:28:13.374 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.380 DEBUG:boto:Path: /yxvvuh-6/ 2022-09-13T07:28:13.381 DEBUG:boto:Data: 2022-09-13T07:28:13.382 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.383 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:13.384 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.385 DEBUG:boto:Params: {} 2022-09-13T07:28:13.385 DEBUG:boto:Token: None 2022-09-13T07:28:13.386 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-6/ 2022-09-13T07:28:13.387 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0N4PFomvzzMlblMn3A81MLY3MwA= 2022-09-13T07:28:13.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0N4PFomvzzMlblMn3A81MLY3MwA=', 'Content-Length': '0'} 2022-09-13T07:28:13.396 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008bd6213bfee9aa52-006320310d-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.397 DEBUG:boto:path=/yxvvuh-4/ 2022-09-13T07:28:13.397 DEBUG:boto:auth_path=/yxvvuh-4/ 2022-09-13T07:28:13.398 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.398 DEBUG:boto:Path: /yxvvuh-4/ 2022-09-13T07:28:13.399 DEBUG:boto:Data: 2022-09-13T07:28:13.399 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.400 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:13.400 DEBUG:boto:Port: 8001 2022-09-13T07:28:13.401 DEBUG:boto:Params: {} 2022-09-13T07:28:13.402 DEBUG:boto:Token: None 2022-09-13T07:28:13.402 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-4/ 2022-09-13T07:28:13.403 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MZbbkCPWiiCkxrTDTcsq8OkgCj4= 2022-09-13T07:28:13.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MZbbkCPWiiCkxrTDTcsq8OkgCj4=', 'Content-Length': '0'} 2022-09-13T07:28:13.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfb253cbaa706eca-006320310d-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.411 DEBUG:boto:path=/yxvvuh-5/ 2022-09-13T07:28:13.411 DEBUG:boto:auth_path=/yxvvuh-5/ 2022-09-13T07:28:13.412 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.412 DEBUG:boto:Path: /yxvvuh-5/ 2022-09-13T07:28:13.413 DEBUG:boto:Data: 2022-09-13T07:28:13.413 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.414 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:13.414 DEBUG:boto:Port: 8001 2022-09-13T07:28:13.415 DEBUG:boto:Params: {} 2022-09-13T07:28:13.415 DEBUG:boto:Token: None 2022-09-13T07:28:13.416 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-5/ 2022-09-13T07:28:13.417 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hnSei0sXDHpn+mj19cG44HPDLhY= 2022-09-13T07:28:13.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hnSei0sXDHpn+mj19cG44HPDLhY=', 'Content-Length': '0'} 2022-09-13T07:28:13.420 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048e21fc0c79e007f-006320310d-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.421 DEBUG:boto:path=/yxvvuh-6/ 2022-09-13T07:28:13.421 DEBUG:boto:auth_path=/yxvvuh-6/ 2022-09-13T07:28:13.422 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.422 DEBUG:boto:Path: /yxvvuh-6/ 2022-09-13T07:28:13.423 DEBUG:boto:Data: 2022-09-13T07:28:13.423 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.424 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:13.424 DEBUG:boto:Port: 8001 2022-09-13T07:28:13.425 DEBUG:boto:Params: {} 2022-09-13T07:28:13.425 DEBUG:boto:Token: None 2022-09-13T07:28:13.426 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-6/ 2022-09-13T07:28:13.427 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0N4PFomvzzMlblMn3A81MLY3MwA= 2022-09-13T07:28:13.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0N4PFomvzzMlblMn3A81MLY3MwA=', 'Content-Length': '0'} 2022-09-13T07:28:13.434 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2ea4919fb252a68-006320310d-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.435 DEBUG:boto:path=/yxvvuh-4/ 2022-09-13T07:28:13.436 DEBUG:boto:auth_path=/yxvvuh-4/ 2022-09-13T07:28:13.437 DEBUG:boto:Method: PUT 2022-09-13T07:28:13.437 DEBUG:boto:Path: /yxvvuh-4/ 2022-09-13T07:28:13.438 DEBUG:boto:Data: 2022-09-13T07:28:13.438 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.439 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:13.439 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.440 DEBUG:boto:Params: {} 2022-09-13T07:28:13.440 DEBUG:boto:Token: None 2022-09-13T07:28:13.441 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-4/ 2022-09-13T07:28:13.441 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UGc8fnCNRdItmOHUgTt+f3Jj+zg= 2022-09-13T07:28:13.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UGc8fnCNRdItmOHUgTt+f3Jj+zg=', 'Content-Length': '0'} 2022-09-13T07:28:13.446 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fdc5e3859ea52d2d-006320310d-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.447 DEBUG:boto:path=/yxvvuh-5/ 2022-09-13T07:28:13.447 DEBUG:boto:auth_path=/yxvvuh-5/ 2022-09-13T07:28:13.448 DEBUG:boto:Method: PUT 2022-09-13T07:28:13.448 DEBUG:boto:Path: /yxvvuh-5/ 2022-09-13T07:28:13.449 DEBUG:boto:Data: 2022-09-13T07:28:13.449 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.450 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:13.450 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.451 DEBUG:boto:Params: {} 2022-09-13T07:28:13.452 DEBUG:boto:Token: None 2022-09-13T07:28:13.453 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-5/ 2022-09-13T07:28:13.453 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZZBKSHjX2VyI3la7DEg1TNgJL08= 2022-09-13T07:28:13.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZZBKSHjX2VyI3la7DEg1TNgJL08=', 'Content-Length': '0'} 2022-09-13T07:28:13.458 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d80ddcfc8c1a8497-006320310d-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.459 DEBUG:boto:path=/yxvvuh-6/ 2022-09-13T07:28:13.459 DEBUG:boto:auth_path=/yxvvuh-6/ 2022-09-13T07:28:13.460 DEBUG:boto:Method: PUT 2022-09-13T07:28:13.460 DEBUG:boto:Path: /yxvvuh-6/ 2022-09-13T07:28:13.461 DEBUG:boto:Data: 2022-09-13T07:28:13.461 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.462 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:13.462 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.463 DEBUG:boto:Params: {} 2022-09-13T07:28:13.463 DEBUG:boto:Token: None 2022-09-13T07:28:13.464 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-6/ 2022-09-13T07:28:13.465 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fmf4XY18eAbgWQlg3vJJ0t7P21g= 2022-09-13T07:28:13.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fmf4XY18eAbgWQlg3vJJ0t7P21g=', 'Content-Length': '0'} 2022-09-13T07:28:13.470 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bf2d6cba81c7aeb2-006320310d-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.471 DEBUG:boto:path=/yxvvuh-4/ 2022-09-13T07:28:13.471 DEBUG:boto:auth_path=/yxvvuh-4/ 2022-09-13T07:28:13.472 DEBUG:boto:Method: PUT 2022-09-13T07:28:13.473 DEBUG:boto:Path: /yxvvuh-4/ 2022-09-13T07:28:13.474 DEBUG:boto:Data: 2022-09-13T07:28:13.474 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.475 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:13.476 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.476 DEBUG:boto:Params: {} 2022-09-13T07:28:13.477 DEBUG:boto:Token: None 2022-09-13T07:28:13.478 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-4/ 2022-09-13T07:28:13.478 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UGc8fnCNRdItmOHUgTt+f3Jj+zg= 2022-09-13T07:28:13.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UGc8fnCNRdItmOHUgTt+f3Jj+zg=', 'Content-Length': '0'} 2022-09-13T07:28:13.485 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000324c463111f8140d-006320310d-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.486 DEBUG:boto:path=/yxvvuh-5/ 2022-09-13T07:28:13.487 DEBUG:boto:auth_path=/yxvvuh-5/ 2022-09-13T07:28:13.487 DEBUG:boto:Method: PUT 2022-09-13T07:28:13.488 DEBUG:boto:Path: /yxvvuh-5/ 2022-09-13T07:28:13.488 DEBUG:boto:Data: 2022-09-13T07:28:13.489 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.489 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:13.490 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.491 DEBUG:boto:Params: {} 2022-09-13T07:28:13.491 DEBUG:boto:Token: None 2022-09-13T07:28:13.492 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-5/ 2022-09-13T07:28:13.492 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZZBKSHjX2VyI3la7DEg1TNgJL08= 2022-09-13T07:28:13.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZZBKSHjX2VyI3la7DEg1TNgJL08=', 'Content-Length': '0'} 2022-09-13T07:28:13.541 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000d42577fc7f3d89a-006320310d-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.541 DEBUG:boto:path=/yxvvuh-6/ 2022-09-13T07:28:13.542 DEBUG:boto:auth_path=/yxvvuh-6/ 2022-09-13T07:28:13.543 DEBUG:boto:Method: PUT 2022-09-13T07:28:13.543 DEBUG:boto:Path: /yxvvuh-6/ 2022-09-13T07:28:13.544 DEBUG:boto:Data: 2022-09-13T07:28:13.544 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.544 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:13.545 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.545 DEBUG:boto:Params: {} 2022-09-13T07:28:13.546 DEBUG:boto:Token: None 2022-09-13T07:28:13.547 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-6/ 2022-09-13T07:28:13.547 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fmf4XY18eAbgWQlg3vJJ0t7P21g= 2022-09-13T07:28:13.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fmf4XY18eAbgWQlg3vJJ0t7P21g=', 'Content-Length': '0'} 2022-09-13T07:28:13.594 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000099772af3d76faf7d-006320310d-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.595 DEBUG:boto:path=/yxvvuh-4/ 2022-09-13T07:28:13.596 DEBUG:boto:auth_path=/yxvvuh-4/ 2022-09-13T07:28:13.596 DEBUG:boto:Method: PUT 2022-09-13T07:28:13.597 DEBUG:boto:Path: /yxvvuh-4/ 2022-09-13T07:28:13.597 DEBUG:boto:Data: 2022-09-13T07:28:13.598 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.598 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:13.599 DEBUG:boto:Port: 8001 2022-09-13T07:28:13.599 DEBUG:boto:Params: {} 2022-09-13T07:28:13.600 DEBUG:boto:Token: None 2022-09-13T07:28:13.600 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-4/ 2022-09-13T07:28:13.601 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UGc8fnCNRdItmOHUgTt+f3Jj+zg= 2022-09-13T07:28:13.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UGc8fnCNRdItmOHUgTt+f3Jj+zg=', 'Content-Length': '0'} 2022-09-13T07:28:13.607 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000042d9b7153b9ff63c-006320310d-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.608 DEBUG:boto:path=/yxvvuh-5/ 2022-09-13T07:28:13.609 DEBUG:boto:auth_path=/yxvvuh-5/ 2022-09-13T07:28:13.609 DEBUG:boto:Method: PUT 2022-09-13T07:28:13.610 DEBUG:boto:Path: /yxvvuh-5/ 2022-09-13T07:28:13.611 DEBUG:boto:Data: 2022-09-13T07:28:13.612 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.613 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:13.613 DEBUG:boto:Port: 8001 2022-09-13T07:28:13.614 DEBUG:boto:Params: {} 2022-09-13T07:28:13.615 DEBUG:boto:Token: None 2022-09-13T07:28:13.616 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-5/ 2022-09-13T07:28:13.617 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZZBKSHjX2VyI3la7DEg1TNgJL08= 2022-09-13T07:28:13.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZZBKSHjX2VyI3la7DEg1TNgJL08=', 'Content-Length': '0'} 2022-09-13T07:28:13.664 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bac970f9930d2167-006320310d-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.665 DEBUG:boto:path=/yxvvuh-6/ 2022-09-13T07:28:13.666 DEBUG:boto:auth_path=/yxvvuh-6/ 2022-09-13T07:28:13.667 DEBUG:boto:Method: PUT 2022-09-13T07:28:13.667 DEBUG:boto:Path: /yxvvuh-6/ 2022-09-13T07:28:13.668 DEBUG:boto:Data: 2022-09-13T07:28:13.668 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.669 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:13.669 DEBUG:boto:Port: 8001 2022-09-13T07:28:13.670 DEBUG:boto:Params: {} 2022-09-13T07:28:13.671 DEBUG:boto:Token: None 2022-09-13T07:28:13.672 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-6/ 2022-09-13T07:28:13.672 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fmf4XY18eAbgWQlg3vJJ0t7P21g= 2022-09-13T07:28:13.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fmf4XY18eAbgWQlg3vJJ0t7P21g=', 'Content-Length': '0'} 2022-09-13T07:28:13.721 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004d525c457f07927f-006320310d-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.722 DEBUG:boto:path=/yxvvuh-4/ 2022-09-13T07:28:13.722 DEBUG:boto:auth_path=/yxvvuh-4/ 2022-09-13T07:28:13.723 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.724 DEBUG:boto:Path: /yxvvuh-4/ 2022-09-13T07:28:13.724 DEBUG:boto:Data: 2022-09-13T07:28:13.725 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.725 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:13.726 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.726 DEBUG:boto:Params: {} 2022-09-13T07:28:13.727 DEBUG:boto:Token: None 2022-09-13T07:28:13.728 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-4/ 2022-09-13T07:28:13.728 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MZbbkCPWiiCkxrTDTcsq8OkgCj4= 2022-09-13T07:28:13.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MZbbkCPWiiCkxrTDTcsq8OkgCj4=', 'Content-Length': '0'} 2022-09-13T07:28:13.732 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000000953e848a23abf-006320310d-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.732 DEBUG:boto:path=/yxvvuh-5/ 2022-09-13T07:28:13.733 DEBUG:boto:auth_path=/yxvvuh-5/ 2022-09-13T07:28:13.733 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.734 DEBUG:boto:Path: /yxvvuh-5/ 2022-09-13T07:28:13.734 DEBUG:boto:Data: 2022-09-13T07:28:13.735 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.735 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:13.736 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.737 DEBUG:boto:Params: {} 2022-09-13T07:28:13.737 DEBUG:boto:Token: None 2022-09-13T07:28:13.738 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-5/ 2022-09-13T07:28:13.739 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hnSei0sXDHpn+mj19cG44HPDLhY= 2022-09-13T07:28:13.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hnSei0sXDHpn+mj19cG44HPDLhY=', 'Content-Length': '0'} 2022-09-13T07:28:13.742 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e536d76dd21b0b66-006320310d-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.742 DEBUG:boto:path=/yxvvuh-6/ 2022-09-13T07:28:13.743 DEBUG:boto:auth_path=/yxvvuh-6/ 2022-09-13T07:28:13.743 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.744 DEBUG:boto:Path: /yxvvuh-6/ 2022-09-13T07:28:13.745 DEBUG:boto:Data: 2022-09-13T07:28:13.746 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.747 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:13.747 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.748 DEBUG:boto:Params: {} 2022-09-13T07:28:13.748 DEBUG:boto:Token: None 2022-09-13T07:28:13.749 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-6/ 2022-09-13T07:28:13.750 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0N4PFomvzzMlblMn3A81MLY3MwA= 2022-09-13T07:28:13.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0N4PFomvzzMlblMn3A81MLY3MwA=', 'Content-Length': '0'} 2022-09-13T07:28:13.753 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f8d432ae836daa1-006320310d-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.754 DEBUG:boto:path=/yxvvuh-4/ 2022-09-13T07:28:13.755 DEBUG:boto:auth_path=/yxvvuh-4/ 2022-09-13T07:28:13.756 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.756 DEBUG:boto:Path: /yxvvuh-4/ 2022-09-13T07:28:13.757 DEBUG:boto:Data: 2022-09-13T07:28:13.758 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.758 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:13.759 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.760 DEBUG:boto:Params: {} 2022-09-13T07:28:13.760 DEBUG:boto:Token: None 2022-09-13T07:28:13.761 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-4/ 2022-09-13T07:28:13.762 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MZbbkCPWiiCkxrTDTcsq8OkgCj4= 2022-09-13T07:28:13.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MZbbkCPWiiCkxrTDTcsq8OkgCj4=', 'Content-Length': '0'} 2022-09-13T07:28:13.765 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6cdd82c0d21b901-006320310d-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.766 DEBUG:boto:path=/yxvvuh-5/ 2022-09-13T07:28:13.766 DEBUG:boto:auth_path=/yxvvuh-5/ 2022-09-13T07:28:13.767 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.767 DEBUG:boto:Path: /yxvvuh-5/ 2022-09-13T07:28:13.768 DEBUG:boto:Data: 2022-09-13T07:28:13.769 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.769 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:13.770 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.770 DEBUG:boto:Params: {} 2022-09-13T07:28:13.771 DEBUG:boto:Token: None 2022-09-13T07:28:13.772 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-5/ 2022-09-13T07:28:13.772 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hnSei0sXDHpn+mj19cG44HPDLhY= 2022-09-13T07:28:13.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hnSei0sXDHpn+mj19cG44HPDLhY=', 'Content-Length': '0'} 2022-09-13T07:28:13.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a4e6e1fbd089f12-006320310d-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.777 DEBUG:boto:path=/yxvvuh-6/ 2022-09-13T07:28:13.778 DEBUG:boto:auth_path=/yxvvuh-6/ 2022-09-13T07:28:13.778 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.779 DEBUG:boto:Path: /yxvvuh-6/ 2022-09-13T07:28:13.779 DEBUG:boto:Data: 2022-09-13T07:28:13.780 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.781 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:13.781 DEBUG:boto:Port: 8000 2022-09-13T07:28:13.782 DEBUG:boto:Params: {} 2022-09-13T07:28:13.782 DEBUG:boto:Token: None 2022-09-13T07:28:13.783 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-6/ 2022-09-13T07:28:13.783 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0N4PFomvzzMlblMn3A81MLY3MwA= 2022-09-13T07:28:13.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0N4PFomvzzMlblMn3A81MLY3MwA=', 'Content-Length': '0'} 2022-09-13T07:28:13.787 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014b2fd50135d8001-006320310d-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.788 DEBUG:boto:path=/yxvvuh-4/ 2022-09-13T07:28:13.789 DEBUG:boto:auth_path=/yxvvuh-4/ 2022-09-13T07:28:13.790 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.791 DEBUG:boto:Path: /yxvvuh-4/ 2022-09-13T07:28:13.791 DEBUG:boto:Data: 2022-09-13T07:28:13.792 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.794 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:13.795 DEBUG:boto:Port: 8001 2022-09-13T07:28:13.795 DEBUG:boto:Params: {} 2022-09-13T07:28:13.796 DEBUG:boto:Token: None 2022-09-13T07:28:13.797 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-4/ 2022-09-13T07:28:13.798 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MZbbkCPWiiCkxrTDTcsq8OkgCj4= 2022-09-13T07:28:13.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MZbbkCPWiiCkxrTDTcsq8OkgCj4=', 'Content-Length': '0'} 2022-09-13T07:28:13.803 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e772099442066f5-006320310d-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.804 DEBUG:boto:path=/yxvvuh-5/ 2022-09-13T07:28:13.805 DEBUG:boto:auth_path=/yxvvuh-5/ 2022-09-13T07:28:13.805 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.807 DEBUG:boto:Path: /yxvvuh-5/ 2022-09-13T07:28:13.807 DEBUG:boto:Data: 2022-09-13T07:28:13.809 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.810 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:13.812 DEBUG:boto:Port: 8001 2022-09-13T07:28:13.813 DEBUG:boto:Params: {} 2022-09-13T07:28:13.815 DEBUG:boto:Token: None 2022-09-13T07:28:13.816 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-5/ 2022-09-13T07:28:13.816 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hnSei0sXDHpn+mj19cG44HPDLhY= 2022-09-13T07:28:13.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hnSei0sXDHpn+mj19cG44HPDLhY=', 'Content-Length': '0'} 2022-09-13T07:28:13.820 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056a6746f39b41635-006320310d-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.821 DEBUG:boto:path=/yxvvuh-6/ 2022-09-13T07:28:13.822 DEBUG:boto:auth_path=/yxvvuh-6/ 2022-09-13T07:28:13.823 DEBUG:boto:Method: HEAD 2022-09-13T07:28:13.824 DEBUG:boto:Path: /yxvvuh-6/ 2022-09-13T07:28:13.825 DEBUG:boto:Data: 2022-09-13T07:28:13.826 DEBUG:boto:Headers: {} 2022-09-13T07:28:13.828 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:13.829 DEBUG:boto:Port: 8001 2022-09-13T07:28:13.831 DEBUG:boto:Params: {} 2022-09-13T07:28:13.831 DEBUG:boto:Token: None 2022-09-13T07:28:13.833 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:13 GMT /yxvvuh-6/ 2022-09-13T07:28:13.834 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0N4PFomvzzMlblMn3A81MLY3MwA= 2022-09-13T07:28:13.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0N4PFomvzzMlblMn3A81MLY3MwA=', 'Content-Length': '0'} 2022-09-13T07:28:13.837 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c40576cf7565ab9-006320310d-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:13.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:13.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:13.839 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:13.893 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:13.894 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:13.922 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:28:13.935 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:28:13.935 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:13.935 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:13.936 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:28:13.936 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:13.936 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:13.937 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:13.937 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:28:13.937 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:13.937 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:13.938 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:13.938 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:28:13.938 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:13.939 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:13.939 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054082.522898_53.1", 2022-09-13T07:28:13.939 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:02.522898Z" 2022-09-13T07:28:13.940 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:13.940 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:28:13.941 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054082.522898_53.1'} 2022-09-13T07:28:13.942 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:28:13.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:13.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:13.944 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:14.001 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:14.001 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:14.045 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:14.041+0000 7fdc67500600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:14.046 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:14.042+0000 7fdc67500600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:14.059 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:28:14.059 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:28:14.059 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:28:14.060 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:28:14.060 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:28:14.060 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:14.060 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:14.061 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:14.061 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:28:14.062 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:14.062 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:28:14.062 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:14.063 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:14.063 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:14.064 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:14.064 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:28:14.065 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:14.065 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:14.066 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:14.066 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:14.067 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:14.067 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:14.068 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:28:14.068 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:14.069 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:14.069 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:14.070 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:14.070 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:14.071 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:14.071 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:14.072 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:14.072 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:14.072 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:14.073 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:14.073 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:28:14.074 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:14.074 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:14.075 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:14.075 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:14.076 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:14.076 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:14.077 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:14.077 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:14.077 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:14.078 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:14.078 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:14.079 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:28:14.079 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:14.080 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:14.080 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054082.522898_53.1", 2022-09-13T07:28:14.080 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:14.080 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:14.081 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:14.081 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:02.522898Z", 2022-09-13T07:28:14.081 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:14.082 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:14.082 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:14.082 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:28:14.083 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:14.083 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:28:14.083 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:28:14.084 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:28:14.084 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:14.084 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:28:14Z" 2022-09-13T07:28:14.085 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:28:14.086 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054082.522898_53.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:02.522898Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:14Z" } 2022-09-13T07:28:14.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054082.522898_53.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:02.522898Z', 'realm_epoch': 2}}] 2022-09-13T07:28:14.087 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054082.522898_53.1'} 2022-09-13T07:28:14.088 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054082.522898_53.1'} 2022-09-13T07:28:14.088 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:28:14.089 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:28:14.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:14.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:14.091 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:14.144 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:14.145 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:14.190 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:14.188+0000 7fbc2b69a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:14.191 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:14.189+0000 7fbc2b69a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:14.206 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:28:14.206 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:28:14.206 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:28:14.207 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:28:14.207 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:28:14.207 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:14.207 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:14.208 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:14.208 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:28:14.208 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:14.208 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:28:14.209 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:14.209 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:14.209 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:14.209 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:14.210 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:28:14.210 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:14.210 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:14.211 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:14.211 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:14.211 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:14.211 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:14.212 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:28:14.212 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:14.212 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:14.213 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:14.213 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:14.213 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:14.213 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:14.213 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:28:14.214 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:14.214 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:14.214 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:14.215 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:14.215 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:28:14.215 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:14.215 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:14.216 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:14.216 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:14.216 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:14.216 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:14.217 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:28:14.217 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:14.217 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:14.217 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:14.218 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:14.218 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:28:14.218 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:14.218 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:14.219 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054082.522898_53.1", 2022-09-13T07:28:14.219 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:14.219 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:14.219 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:14.220 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:02.522898Z", 2022-09-13T07:28:14.220 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:14.220 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:14.220 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:14.221 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:28:14.221 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:14.221 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:28:14.221 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:28:14.222 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:28:14.222 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:14.222 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:28:14Z" 2022-09-13T07:28:14.223 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:28:14.224 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054082.522898_53.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:02.522898Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:14Z" } 2022-09-13T07:28:14.224 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054082.522898_53.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:02.522898Z', 'realm_epoch': 2}}] 2022-09-13T07:28:14.225 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054082.522898_53.1'} 2022-09-13T07:28:14.225 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054082.522898_53.1'} 2022-09-13T07:28:14.226 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:28:14.226 DEBUG:boto:path=/yxvvuh-4/ 2022-09-13T07:28:14.227 DEBUG:boto:auth_path=/yxvvuh-4/ 2022-09-13T07:28:14.227 DEBUG:boto:Method: HEAD 2022-09-13T07:28:14.228 DEBUG:boto:Path: /yxvvuh-4/ 2022-09-13T07:28:14.228 DEBUG:boto:Data: 2022-09-13T07:28:14.229 DEBUG:boto:Headers: {} 2022-09-13T07:28:14.229 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:14.230 DEBUG:boto:Port: 8000 2022-09-13T07:28:14.230 DEBUG:boto:Params: {} 2022-09-13T07:28:14.231 DEBUG:boto:Token: None 2022-09-13T07:28:14.231 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:14 GMT /yxvvuh-4/ 2022-09-13T07:28:14.232 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/Fc7vzRYPqH14BhHIMbw0u2FPno= 2022-09-13T07:28:14.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/Fc7vzRYPqH14BhHIMbw0u2FPno=', 'Content-Length': '0'} 2022-09-13T07:28:14.235 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d19b43560a59e059-006320310e-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:14.235 DEBUG:boto:path=/yxvvuh-5/ 2022-09-13T07:28:14.236 DEBUG:boto:auth_path=/yxvvuh-5/ 2022-09-13T07:28:14.236 DEBUG:boto:Method: HEAD 2022-09-13T07:28:14.237 DEBUG:boto:Path: /yxvvuh-5/ 2022-09-13T07:28:14.237 DEBUG:boto:Data: 2022-09-13T07:28:14.238 DEBUG:boto:Headers: {} 2022-09-13T07:28:14.238 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:14.239 DEBUG:boto:Port: 8000 2022-09-13T07:28:14.239 DEBUG:boto:Params: {} 2022-09-13T07:28:14.240 DEBUG:boto:Token: None 2022-09-13T07:28:14.240 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:14 GMT /yxvvuh-5/ 2022-09-13T07:28:14.241 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+/7cHLaMmlQuGSlzdyNokJZGP4s= 2022-09-13T07:28:14.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+/7cHLaMmlQuGSlzdyNokJZGP4s=', 'Content-Length': '0'} 2022-09-13T07:28:14.244 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000800668aec96ddccd-006320310e-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:14.244 DEBUG:boto:path=/yxvvuh-6/ 2022-09-13T07:28:14.245 DEBUG:boto:auth_path=/yxvvuh-6/ 2022-09-13T07:28:14.245 DEBUG:boto:Method: HEAD 2022-09-13T07:28:14.246 DEBUG:boto:Path: /yxvvuh-6/ 2022-09-13T07:28:14.246 DEBUG:boto:Data: 2022-09-13T07:28:14.247 DEBUG:boto:Headers: {} 2022-09-13T07:28:14.247 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:14.248 DEBUG:boto:Port: 8000 2022-09-13T07:28:14.248 DEBUG:boto:Params: {} 2022-09-13T07:28:14.249 DEBUG:boto:Token: None 2022-09-13T07:28:14.249 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:14 GMT /yxvvuh-6/ 2022-09-13T07:28:14.250 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fwvuX9SOC0vuIb3sI3Dap6sOV0U= 2022-09-13T07:28:14.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fwvuX9SOC0vuIb3sI3Dap6sOV0U=', 'Content-Length': '0'} 2022-09-13T07:28:14.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a51a623a39e6c94e-006320310e-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:14.253 DEBUG:boto:path=/yxvvuh-4/ 2022-09-13T07:28:14.254 DEBUG:boto:auth_path=/yxvvuh-4/ 2022-09-13T07:28:14.254 DEBUG:boto:Method: HEAD 2022-09-13T07:28:14.255 DEBUG:boto:Path: /yxvvuh-4/ 2022-09-13T07:28:14.255 DEBUG:boto:Data: 2022-09-13T07:28:14.256 DEBUG:boto:Headers: {} 2022-09-13T07:28:14.256 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:14.257 DEBUG:boto:Port: 8000 2022-09-13T07:28:14.257 DEBUG:boto:Params: {} 2022-09-13T07:28:14.258 DEBUG:boto:Token: None 2022-09-13T07:28:14.258 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:14 GMT /yxvvuh-4/ 2022-09-13T07:28:14.259 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/Fc7vzRYPqH14BhHIMbw0u2FPno= 2022-09-13T07:28:14.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/Fc7vzRYPqH14BhHIMbw0u2FPno=', 'Content-Length': '0'} 2022-09-13T07:28:14.262 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004738457261ceb899-006320310e-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:14.262 DEBUG:boto:path=/yxvvuh-5/ 2022-09-13T07:28:14.263 DEBUG:boto:auth_path=/yxvvuh-5/ 2022-09-13T07:28:14.264 DEBUG:boto:Method: HEAD 2022-09-13T07:28:14.264 DEBUG:boto:Path: /yxvvuh-5/ 2022-09-13T07:28:14.265 DEBUG:boto:Data: 2022-09-13T07:28:14.265 DEBUG:boto:Headers: {} 2022-09-13T07:28:14.266 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:14.266 DEBUG:boto:Port: 8000 2022-09-13T07:28:14.267 DEBUG:boto:Params: {} 2022-09-13T07:28:14.267 DEBUG:boto:Token: None 2022-09-13T07:28:14.268 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:14 GMT /yxvvuh-5/ 2022-09-13T07:28:14.268 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+/7cHLaMmlQuGSlzdyNokJZGP4s= 2022-09-13T07:28:14.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+/7cHLaMmlQuGSlzdyNokJZGP4s=', 'Content-Length': '0'} 2022-09-13T07:28:14.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e461b52603f48b61-006320310e-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:14.272 DEBUG:boto:path=/yxvvuh-6/ 2022-09-13T07:28:14.272 DEBUG:boto:auth_path=/yxvvuh-6/ 2022-09-13T07:28:14.273 DEBUG:boto:Method: HEAD 2022-09-13T07:28:14.273 DEBUG:boto:Path: /yxvvuh-6/ 2022-09-13T07:28:14.274 DEBUG:boto:Data: 2022-09-13T07:28:14.274 DEBUG:boto:Headers: {} 2022-09-13T07:28:14.275 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:14.275 DEBUG:boto:Port: 8000 2022-09-13T07:28:14.276 DEBUG:boto:Params: {} 2022-09-13T07:28:14.276 DEBUG:boto:Token: None 2022-09-13T07:28:14.277 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:14 GMT /yxvvuh-6/ 2022-09-13T07:28:14.277 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fwvuX9SOC0vuIb3sI3Dap6sOV0U= 2022-09-13T07:28:14.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fwvuX9SOC0vuIb3sI3Dap6sOV0U=', 'Content-Length': '0'} 2022-09-13T07:28:14.280 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddb59a7e04909653-006320310e-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:14.281 DEBUG:boto:path=/yxvvuh-4/ 2022-09-13T07:28:14.281 DEBUG:boto:auth_path=/yxvvuh-4/ 2022-09-13T07:28:14.282 DEBUG:boto:Method: HEAD 2022-09-13T07:28:14.282 DEBUG:boto:Path: /yxvvuh-4/ 2022-09-13T07:28:14.283 DEBUG:boto:Data: 2022-09-13T07:28:14.283 DEBUG:boto:Headers: {} 2022-09-13T07:28:14.284 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:14.284 DEBUG:boto:Port: 8001 2022-09-13T07:28:14.285 DEBUG:boto:Params: {} 2022-09-13T07:28:14.285 DEBUG:boto:Token: None 2022-09-13T07:28:14.286 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:14 GMT /yxvvuh-4/ 2022-09-13T07:28:14.286 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/Fc7vzRYPqH14BhHIMbw0u2FPno= 2022-09-13T07:28:14.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/Fc7vzRYPqH14BhHIMbw0u2FPno=', 'Content-Length': '0'} 2022-09-13T07:28:14.289 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4ac744e396134d4-006320310e-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:14.289 DEBUG:boto:path=/yxvvuh-5/ 2022-09-13T07:28:14.290 DEBUG:boto:auth_path=/yxvvuh-5/ 2022-09-13T07:28:14.291 DEBUG:boto:Method: HEAD 2022-09-13T07:28:14.291 DEBUG:boto:Path: /yxvvuh-5/ 2022-09-13T07:28:14.292 DEBUG:boto:Data: 2022-09-13T07:28:14.292 DEBUG:boto:Headers: {} 2022-09-13T07:28:14.293 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:14.293 DEBUG:boto:Port: 8001 2022-09-13T07:28:14.294 DEBUG:boto:Params: {} 2022-09-13T07:28:14.294 DEBUG:boto:Token: None 2022-09-13T07:28:14.295 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:14 GMT /yxvvuh-5/ 2022-09-13T07:28:14.295 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+/7cHLaMmlQuGSlzdyNokJZGP4s= 2022-09-13T07:28:14.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+/7cHLaMmlQuGSlzdyNokJZGP4s=', 'Content-Length': '0'} 2022-09-13T07:28:14.299 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0f7510215f422e0-006320310e-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:14.299 DEBUG:boto:path=/yxvvuh-6/ 2022-09-13T07:28:14.300 DEBUG:boto:auth_path=/yxvvuh-6/ 2022-09-13T07:28:14.301 DEBUG:boto:Method: HEAD 2022-09-13T07:28:14.302 DEBUG:boto:Path: /yxvvuh-6/ 2022-09-13T07:28:14.302 DEBUG:boto:Data: 2022-09-13T07:28:14.303 DEBUG:boto:Headers: {} 2022-09-13T07:28:14.303 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:14.304 DEBUG:boto:Port: 8001 2022-09-13T07:28:14.304 DEBUG:boto:Params: {} 2022-09-13T07:28:14.305 DEBUG:boto:Token: None 2022-09-13T07:28:14.306 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:14 GMT /yxvvuh-6/ 2022-09-13T07:28:14.307 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fwvuX9SOC0vuIb3sI3Dap6sOV0U= 2022-09-13T07:28:14.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fwvuX9SOC0vuIb3sI3Dap6sOV0U=', 'Content-Length': '0'} 2022-09-13T07:28:14.310 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009cd478120ede9942-006320310e-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:14.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2022-09-13T07:28:14.312 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-7 2022-09-13T07:28:14.313 DEBUG:boto:path=/yxvvuh-7/ 2022-09-13T07:28:14.313 DEBUG:boto:auth_path=/yxvvuh-7/ 2022-09-13T07:28:14.314 DEBUG:boto:Method: PUT 2022-09-13T07:28:14.314 DEBUG:boto:Path: /yxvvuh-7/ 2022-09-13T07:28:14.315 DEBUG:boto:Data: 2022-09-13T07:28:14.316 DEBUG:boto:Headers: {} 2022-09-13T07:28:14.316 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:14.317 DEBUG:boto:Port: 8000 2022-09-13T07:28:14.317 DEBUG:boto:Params: {} 2022-09-13T07:28:14.318 DEBUG:boto:Token: None 2022-09-13T07:28:14.318 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:14 GMT /yxvvuh-7/ 2022-09-13T07:28:14.319 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fhfmWKfT5Ssrhwz9upR19SgbqAk= 2022-09-13T07:28:14.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fhfmWKfT5Ssrhwz9upR19SgbqAk=', 'Content-Length': '0'} 2022-09-13T07:28:14.348 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000054698346e4aea4b4-006320310e-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:14.349 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-8 2022-09-13T07:28:14.349 DEBUG:boto:path=/yxvvuh-8/ 2022-09-13T07:28:14.350 DEBUG:boto:auth_path=/yxvvuh-8/ 2022-09-13T07:28:14.350 DEBUG:boto:Method: PUT 2022-09-13T07:28:14.351 DEBUG:boto:Path: /yxvvuh-8/ 2022-09-13T07:28:14.351 DEBUG:boto:Data: 2022-09-13T07:28:14.352 DEBUG:boto:Headers: {} 2022-09-13T07:28:14.352 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:14.352 DEBUG:boto:Port: 8000 2022-09-13T07:28:14.353 DEBUG:boto:Params: {} 2022-09-13T07:28:14.353 DEBUG:boto:Token: None 2022-09-13T07:28:14.354 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:14 GMT /yxvvuh-8/ 2022-09-13T07:28:14.355 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nXOWj5kXWDkE9wTlAMPtyAePnmg= 2022-09-13T07:28:14.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nXOWj5kXWDkE9wTlAMPtyAePnmg=', 'Content-Length': '0'} 2022-09-13T07:28:14.379 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008bd6b2ef49a26110-006320310e-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:14.380 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-9 2022-09-13T07:28:14.381 DEBUG:boto:path=/yxvvuh-9/ 2022-09-13T07:28:14.381 DEBUG:boto:auth_path=/yxvvuh-9/ 2022-09-13T07:28:14.382 DEBUG:boto:Method: PUT 2022-09-13T07:28:14.382 DEBUG:boto:Path: /yxvvuh-9/ 2022-09-13T07:28:14.383 DEBUG:boto:Data: 2022-09-13T07:28:14.383 DEBUG:boto:Headers: {} 2022-09-13T07:28:14.384 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:14.385 DEBUG:boto:Port: 8001 2022-09-13T07:28:14.385 DEBUG:boto:Params: {} 2022-09-13T07:28:14.386 DEBUG:boto:Token: None 2022-09-13T07:28:14.386 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:14 GMT /yxvvuh-9/ 2022-09-13T07:28:14.387 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nat7DLydlC8wLm00w0/IFoS7ZUg= 2022-09-13T07:28:14.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nat7DLydlC8wLm00w0/IFoS7ZUg=', 'Content-Length': '0'} 2022-09-13T07:28:14.413 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a41bac73194167c3-006320310e-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:14.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:14.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:14.415 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:14.468 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:14.468 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:14.504 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:28:14.522 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:28:14.523 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:14.523 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:14.523 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:28:14.524 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:14.524 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:14.525 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:14.525 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:28:14.526 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:14.526 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:14.527 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:14.527 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:28:14.528 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:14.528 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:14.529 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054094.401732_65.1", 2022-09-13T07:28:14.529 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:14.401732Z" 2022-09-13T07:28:14.529 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:14.530 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:28:14.532 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054094.401732_65.1'} 2022-09-13T07:28:14.533 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:28:14.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:14.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:14.534 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:14.590 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:14.591 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:14.643 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:14.639+0000 7f1a47dfb600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:14.644 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:14.640+0000 7f1a47dfb600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:14.672 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:28:14.673 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:28:14.673 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:28:14.673 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:28:14.674 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:28:14.674 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:14.675 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:14.675 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:14.675 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:28:14.676 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:14.676 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:28:14.677 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:14.678 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:14.679 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:14.679 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:14.680 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:28:14.680 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:14.681 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:14.681 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:14.682 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:14.682 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:14.682 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:14.683 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:28:14.683 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:14.684 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:14.684 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:14.685 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:14.686 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:14.687 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:14.688 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:14.688 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:14.689 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:14.689 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:14.689 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:14.690 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:28:14.690 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:14.691 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:14.691 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:14.691 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:14.692 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:14.692 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:14.692 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:14.693 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:14.693 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:14.693 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:14.693 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:14.694 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:28:14.694 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:14.695 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:14.695 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054094.395645_62.1", 2022-09-13T07:28:14.695 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:14.695 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:14.696 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:14.696 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:14.395645Z", 2022-09-13T07:28:14.696 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:14.697 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:14.697 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:14.697 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:28:14.698 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:14.698 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:28:14.699 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:28:14.699 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:28:14.699 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:14.699 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:28:14Z" 2022-09-13T07:28:14.700 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:28:14.701 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054094.395645_62.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:14.395645Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:14Z" } 2022-09-13T07:28:14.702 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054094.395645_62.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:14.395645Z', 'realm_epoch': 2}}] 2022-09-13T07:28:14.702 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054094.401732_65.1'} 2022-09-13T07:28:14.703 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054094.395645_62.1'} 2022-09-13T07:28:14.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1663054094.401732_65.1 target=1_1663054094.395645_62.1 2022-09-13T07:28:19.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:19.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:19.707 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:19.765 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:19.766 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:19.795 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:19.792+0000 7ffb91f7f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:19.796 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:19.793+0000 7ffb91f7f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:19.812 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:28:19.813 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:28:19.813 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:28:19.814 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:28:19.814 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:28:19.815 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:19.815 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:19.815 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:19.816 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:28:19.816 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:19.817 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:28:19.817 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:19.818 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:19.818 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:19.819 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:19.819 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:28:19.820 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:19.820 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:19.821 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:19.821 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:19.822 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:19.822 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:19.823 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:28:19.823 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:19.824 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:19.824 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:19.824 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:19.825 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:19.825 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:19.826 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:19.826 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:19.827 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:19.827 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:19.827 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:19.828 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:28:19.828 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:19.828 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:19.828 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:19.829 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:19.829 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:19.830 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:19.830 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:19.831 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:19.831 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:19.831 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:19.832 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:19.832 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:28:19.832 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:19.832 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:19.833 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054094.401732_65.1", 2022-09-13T07:28:19.833 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:19.833 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:19.833 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:19.834 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:14.401732Z", 2022-09-13T07:28:19.834 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:19.834 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:19.834 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:19.835 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:28:19.835 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:19.835 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:28:19.835 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:28:19.836 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:28:19.836 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:19.836 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:28:19Z" 2022-09-13T07:28:19.836 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:28:19.837 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054094.401732_65.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:14.401732Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:19Z" } 2022-09-13T07:28:19.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054094.401732_65.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:14.401732Z', 'realm_epoch': 2}}] 2022-09-13T07:28:19.838 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054094.401732_65.1'} 2022-09-13T07:28:19.839 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054094.401732_65.1'} 2022-09-13T07:28:19.839 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:28:19.839 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:28:19.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:19.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:19.841 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:19.900 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:19.901 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:19.938 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:19.934+0000 7f394d7cc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:19.938 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:19.936+0000 7f394d7cc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:19.950 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:28:19.951 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:28:19.951 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:28:19.952 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:28:19.952 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:28:19.953 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:19.953 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:19.954 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:19.955 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:28:19.956 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:19.957 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:28:19.957 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:19.958 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:19.958 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:19.959 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:19.959 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:28:19.960 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:19.960 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:19.960 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:19.961 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:19.961 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:19.962 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:19.962 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:28:19.963 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:19.964 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:19.964 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:19.965 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:19.965 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:19.965 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:19.966 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:28:19.966 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:19.966 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:19.967 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:19.967 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:19.967 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:28:19.967 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:19.968 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:19.968 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:19.968 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:19.968 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:19.969 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:19.969 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:28:19.970 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:19.970 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:19.971 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:19.972 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:19.973 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:28:19.974 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:19.974 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:19.975 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054094.401732_65.1", 2022-09-13T07:28:19.976 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:19.976 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:19.976 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:19.977 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:14.401732Z", 2022-09-13T07:28:19.977 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:19.978 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:19.978 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:19.979 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:28:19.979 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:19.980 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:28:19.980 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:28:19.981 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:28:19.982 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:19.983 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:28:19Z" 2022-09-13T07:28:19.983 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:28:19.984 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054094.401732_65.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:14.401732Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:19Z" } 2022-09-13T07:28:19.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054094.401732_65.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:14.401732Z', 'realm_epoch': 2}}] 2022-09-13T07:28:19.985 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054094.401732_65.1'} 2022-09-13T07:28:19.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054094.401732_65.1'} 2022-09-13T07:28:19.986 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:28:19.987 DEBUG:boto:path=/yxvvuh-7/ 2022-09-13T07:28:19.987 DEBUG:boto:auth_path=/yxvvuh-7/ 2022-09-13T07:28:19.988 DEBUG:boto:Method: HEAD 2022-09-13T07:28:19.988 DEBUG:boto:Path: /yxvvuh-7/ 2022-09-13T07:28:19.989 DEBUG:boto:Data: 2022-09-13T07:28:19.990 DEBUG:boto:Headers: {} 2022-09-13T07:28:19.991 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:19.992 DEBUG:boto:Port: 8000 2022-09-13T07:28:19.993 DEBUG:boto:Params: {} 2022-09-13T07:28:19.994 DEBUG:boto:Token: None 2022-09-13T07:28:19.995 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:19 GMT /yxvvuh-7/ 2022-09-13T07:28:19.996 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:J5tBgv9r0CsbkETAVHPo+PTKx5k= 2022-09-13T07:28:19.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:19 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:J5tBgv9r0CsbkETAVHPo+PTKx5k=', 'Content-Length': '0'} 2022-09-13T07:28:20.001 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009de4ae6c295e5e5d-0063203113-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:20.001 DEBUG:boto:path=/yxvvuh-8/ 2022-09-13T07:28:20.002 DEBUG:boto:auth_path=/yxvvuh-8/ 2022-09-13T07:28:20.002 DEBUG:boto:Method: HEAD 2022-09-13T07:28:20.003 DEBUG:boto:Path: /yxvvuh-8/ 2022-09-13T07:28:20.004 DEBUG:boto:Data: 2022-09-13T07:28:20.004 DEBUG:boto:Headers: {} 2022-09-13T07:28:20.005 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:20.005 DEBUG:boto:Port: 8000 2022-09-13T07:28:20.006 DEBUG:boto:Params: {} 2022-09-13T07:28:20.007 DEBUG:boto:Token: None 2022-09-13T07:28:20.008 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:20 GMT /yxvvuh-8/ 2022-09-13T07:28:20.008 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uIN60lL164fdyrMi4tuEnn58xBg= 2022-09-13T07:28:20.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uIN60lL164fdyrMi4tuEnn58xBg=', 'Content-Length': '0'} 2022-09-13T07:28:20.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f498bce2dce7d87-0063203114-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:20.013 DEBUG:boto:path=/yxvvuh-9/ 2022-09-13T07:28:20.014 DEBUG:boto:auth_path=/yxvvuh-9/ 2022-09-13T07:28:20.015 DEBUG:boto:Method: HEAD 2022-09-13T07:28:20.016 DEBUG:boto:Path: /yxvvuh-9/ 2022-09-13T07:28:20.018 DEBUG:boto:Data: 2022-09-13T07:28:20.019 DEBUG:boto:Headers: {} 2022-09-13T07:28:20.020 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:20.021 DEBUG:boto:Port: 8000 2022-09-13T07:28:20.022 DEBUG:boto:Params: {} 2022-09-13T07:28:20.023 DEBUG:boto:Token: None 2022-09-13T07:28:20.024 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:20 GMT /yxvvuh-9/ 2022-09-13T07:28:20.026 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bU3fn95gxA1TJcvV6oSq4HUzLJA= 2022-09-13T07:28:20.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bU3fn95gxA1TJcvV6oSq4HUzLJA=', 'Content-Length': '0'} 2022-09-13T07:28:20.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003f0a645c8a5e647-0063203114-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:20.032 DEBUG:boto:path=/yxvvuh-7/ 2022-09-13T07:28:20.033 DEBUG:boto:auth_path=/yxvvuh-7/ 2022-09-13T07:28:20.035 DEBUG:boto:Method: HEAD 2022-09-13T07:28:20.036 DEBUG:boto:Path: /yxvvuh-7/ 2022-09-13T07:28:20.037 DEBUG:boto:Data: 2022-09-13T07:28:20.038 DEBUG:boto:Headers: {} 2022-09-13T07:28:20.039 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:20.040 DEBUG:boto:Port: 8000 2022-09-13T07:28:20.046 DEBUG:boto:Params: {} 2022-09-13T07:28:20.047 DEBUG:boto:Token: None 2022-09-13T07:28:20.048 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:20 GMT /yxvvuh-7/ 2022-09-13T07:28:20.049 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QgZ5tZkGYxakT2Cje5V1q+Ydz98= 2022-09-13T07:28:20.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QgZ5tZkGYxakT2Cje5V1q+Ydz98=', 'Content-Length': '0'} 2022-09-13T07:28:20.053 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002531f98d5325f334-0063203114-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:20.054 DEBUG:boto:path=/yxvvuh-8/ 2022-09-13T07:28:20.055 DEBUG:boto:auth_path=/yxvvuh-8/ 2022-09-13T07:28:20.056 DEBUG:boto:Method: HEAD 2022-09-13T07:28:20.057 DEBUG:boto:Path: /yxvvuh-8/ 2022-09-13T07:28:20.058 DEBUG:boto:Data: 2022-09-13T07:28:20.059 DEBUG:boto:Headers: {} 2022-09-13T07:28:20.060 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:20.061 DEBUG:boto:Port: 8000 2022-09-13T07:28:20.062 DEBUG:boto:Params: {} 2022-09-13T07:28:20.063 DEBUG:boto:Token: None 2022-09-13T07:28:20.064 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:20 GMT /yxvvuh-8/ 2022-09-13T07:28:20.065 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uIN60lL164fdyrMi4tuEnn58xBg= 2022-09-13T07:28:20.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uIN60lL164fdyrMi4tuEnn58xBg=', 'Content-Length': '0'} 2022-09-13T07:28:20.069 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1a2b5e3bc1f555e-0063203114-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:20.070 DEBUG:boto:path=/yxvvuh-9/ 2022-09-13T07:28:20.071 DEBUG:boto:auth_path=/yxvvuh-9/ 2022-09-13T07:28:20.072 DEBUG:boto:Method: HEAD 2022-09-13T07:28:20.073 DEBUG:boto:Path: /yxvvuh-9/ 2022-09-13T07:28:20.074 DEBUG:boto:Data: 2022-09-13T07:28:20.075 DEBUG:boto:Headers: {} 2022-09-13T07:28:20.076 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:20.077 DEBUG:boto:Port: 8000 2022-09-13T07:28:20.077 DEBUG:boto:Params: {} 2022-09-13T07:28:20.079 DEBUG:boto:Token: None 2022-09-13T07:28:20.080 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:20 GMT /yxvvuh-9/ 2022-09-13T07:28:20.081 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bU3fn95gxA1TJcvV6oSq4HUzLJA= 2022-09-13T07:28:20.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bU3fn95gxA1TJcvV6oSq4HUzLJA=', 'Content-Length': '0'} 2022-09-13T07:28:20.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e862feffd5312ed-0063203114-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:20.088 DEBUG:boto:path=/yxvvuh-7/ 2022-09-13T07:28:20.089 DEBUG:boto:auth_path=/yxvvuh-7/ 2022-09-13T07:28:20.089 DEBUG:boto:Method: HEAD 2022-09-13T07:28:20.090 DEBUG:boto:Path: /yxvvuh-7/ 2022-09-13T07:28:20.091 DEBUG:boto:Data: 2022-09-13T07:28:20.092 DEBUG:boto:Headers: {} 2022-09-13T07:28:20.093 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:20.094 DEBUG:boto:Port: 8001 2022-09-13T07:28:20.094 DEBUG:boto:Params: {} 2022-09-13T07:28:20.095 DEBUG:boto:Token: None 2022-09-13T07:28:20.096 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:20 GMT /yxvvuh-7/ 2022-09-13T07:28:20.096 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QgZ5tZkGYxakT2Cje5V1q+Ydz98= 2022-09-13T07:28:20.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QgZ5tZkGYxakT2Cje5V1q+Ydz98=', 'Content-Length': '0'} 2022-09-13T07:28:20.101 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ab4b382f019b2f9-0063203114-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:20.101 DEBUG:boto:path=/yxvvuh-8/ 2022-09-13T07:28:20.102 DEBUG:boto:auth_path=/yxvvuh-8/ 2022-09-13T07:28:20.102 DEBUG:boto:Method: HEAD 2022-09-13T07:28:20.103 DEBUG:boto:Path: /yxvvuh-8/ 2022-09-13T07:28:20.103 DEBUG:boto:Data: 2022-09-13T07:28:20.104 DEBUG:boto:Headers: {} 2022-09-13T07:28:20.105 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:20.105 DEBUG:boto:Port: 8001 2022-09-13T07:28:20.106 DEBUG:boto:Params: {} 2022-09-13T07:28:20.106 DEBUG:boto:Token: None 2022-09-13T07:28:20.107 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:20 GMT /yxvvuh-8/ 2022-09-13T07:28:20.107 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uIN60lL164fdyrMi4tuEnn58xBg= 2022-09-13T07:28:20.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uIN60lL164fdyrMi4tuEnn58xBg=', 'Content-Length': '0'} 2022-09-13T07:28:20.114 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a07d9577a7cc0cda-0063203114-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:20.115 DEBUG:boto:path=/yxvvuh-9/ 2022-09-13T07:28:20.115 DEBUG:boto:auth_path=/yxvvuh-9/ 2022-09-13T07:28:20.116 DEBUG:boto:Method: HEAD 2022-09-13T07:28:20.116 DEBUG:boto:Path: /yxvvuh-9/ 2022-09-13T07:28:20.117 DEBUG:boto:Data: 2022-09-13T07:28:20.117 DEBUG:boto:Headers: {} 2022-09-13T07:28:20.118 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:20.118 DEBUG:boto:Port: 8001 2022-09-13T07:28:20.119 DEBUG:boto:Params: {} 2022-09-13T07:28:20.119 DEBUG:boto:Token: None 2022-09-13T07:28:20.120 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:20 GMT /yxvvuh-9/ 2022-09-13T07:28:20.121 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bU3fn95gxA1TJcvV6oSq4HUzLJA= 2022-09-13T07:28:20.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bU3fn95gxA1TJcvV6oSq4HUzLJA=', 'Content-Length': '0'} 2022-09-13T07:28:20.128 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d3a006a80fbf418-0063203114-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:20.129 DEBUG:boto:path=/yxvvuh-7/ 2022-09-13T07:28:20.130 DEBUG:boto:auth_path=/yxvvuh-7/ 2022-09-13T07:28:20.131 DEBUG:boto:Method: DELETE 2022-09-13T07:28:20.132 DEBUG:boto:Path: /yxvvuh-7/ 2022-09-13T07:28:20.132 DEBUG:boto:Data: 2022-09-13T07:28:20.133 DEBUG:boto:Headers: {} 2022-09-13T07:28:20.133 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:20.134 DEBUG:boto:Port: 8000 2022-09-13T07:28:20.134 DEBUG:boto:Params: {} 2022-09-13T07:28:20.135 DEBUG:boto:Token: None 2022-09-13T07:28:20.135 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:28:20 GMT /yxvvuh-7/ 2022-09-13T07:28:20.136 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:V+aRyyv3alxxVy0S1KbQhRatMoM= 2022-09-13T07:28:20.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:V+aRyyv3alxxVy0S1KbQhRatMoM=', 'Content-Length': '0'} 2022-09-13T07:28:20.169 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000da0fe7ea2c497973-0063203114-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:28:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:20.170 DEBUG:boto:path=/yxvvuh-8/ 2022-09-13T07:28:20.170 DEBUG:boto:auth_path=/yxvvuh-8/ 2022-09-13T07:28:20.171 DEBUG:boto:Method: DELETE 2022-09-13T07:28:20.171 DEBUG:boto:Path: /yxvvuh-8/ 2022-09-13T07:28:20.172 DEBUG:boto:Data: 2022-09-13T07:28:20.172 DEBUG:boto:Headers: {} 2022-09-13T07:28:20.173 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:20.173 DEBUG:boto:Port: 8000 2022-09-13T07:28:20.173 DEBUG:boto:Params: {} 2022-09-13T07:28:20.174 DEBUG:boto:Token: None 2022-09-13T07:28:20.175 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:28:20 GMT /yxvvuh-8/ 2022-09-13T07:28:20.175 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vvPg2y3C+VBZPwJ9WedddNO8C+c= 2022-09-13T07:28:20.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vvPg2y3C+VBZPwJ9WedddNO8C+c=', 'Content-Length': '0'} 2022-09-13T07:28:20.245 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000046c58c540beadb78-0063203114-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:28:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:20.246 DEBUG:boto:path=/yxvvuh-9/ 2022-09-13T07:28:20.247 DEBUG:boto:auth_path=/yxvvuh-9/ 2022-09-13T07:28:20.247 DEBUG:boto:Method: DELETE 2022-09-13T07:28:20.248 DEBUG:boto:Path: /yxvvuh-9/ 2022-09-13T07:28:20.248 DEBUG:boto:Data: 2022-09-13T07:28:20.249 DEBUG:boto:Headers: {} 2022-09-13T07:28:20.249 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:20.250 DEBUG:boto:Port: 8001 2022-09-13T07:28:20.251 DEBUG:boto:Params: {} 2022-09-13T07:28:20.251 DEBUG:boto:Token: None 2022-09-13T07:28:20.252 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:28:20 GMT /yxvvuh-9/ 2022-09-13T07:28:20.252 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:g0865FsZca3utzm8ZkL9mlKO1DA= 2022-09-13T07:28:20.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:g0865FsZca3utzm8ZkL9mlKO1DA=', 'Content-Length': '0'} 2022-09-13T07:28:20.325 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000067fcf5a400d3a050-0063203114-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:28:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:20.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:20.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:20.327 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:20.377 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:20.378 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:20.425 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:28:20.437 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:28:20.437 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:20.438 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:20.438 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:28:20.438 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:20.439 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:20.439 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:20.439 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:28:20.439 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:20.440 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:20.440 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:20.440 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:28:20.440 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:20.441 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:20.441 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:28:20.441 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T07:28:20.441 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:20.442 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:28:20.443 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:20.444 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:28:20.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:20.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:20.445 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:20.497 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:20.498 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:20.546 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:20.543+0000 7f9a08973600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:20.547 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:20.545+0000 7f9a08973600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:20.569 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:28:20.570 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:28:20.571 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:28:20.572 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:28:20.572 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:28:20.573 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:20.573 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:20.573 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:20.574 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:28:20.574 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:20.575 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:28:20.575 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:20.576 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:20.576 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:20.577 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:20.577 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:28:20.578 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:20.579 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:20.579 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:20.580 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:20.580 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:20.581 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:20.581 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:28:20.582 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:20.582 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:20.583 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:20.583 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:20.586 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:20.586 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:20.586 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:20.587 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:20.587 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:20.587 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:20.587 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:20.588 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:28:20.588 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:20.588 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:20.588 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:20.589 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:20.589 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:20.589 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:20.590 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:20.590 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:20.591 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:20.591 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:20.591 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:20.592 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:28:20.592 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:20.593 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:20.593 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.147809_66.1", 2022-09-13T07:28:20.593 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:20.594 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:20.594 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:20.595 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.147809Z", 2022-09-13T07:28:20.595 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:20.596 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:20.596 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:20.596 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:28:20.597 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:20.597 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:28:20.598 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:28:20.598 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:28:20.598 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:20.599 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:28:20Z" 2022-09-13T07:28:20.599 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:28:20.601 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.147809_66.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.147809Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:20Z" } 2022-09-13T07:28:20.602 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.147809_66.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.147809Z', 'realm_epoch': 2}}] 2022-09-13T07:28:20.603 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:20.604 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054100.147809_66.1'} 2022-09-13T07:28:20.604 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1663054100.288347_77.1 target=1_1663054100.147809_66.1 2022-09-13T07:28:25.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:25.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:25.609 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:25.667 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:25.668 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:25.710 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:25.707+0000 7f2dd95de600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:25.711 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:25.708+0000 7f2dd95de600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:25.728 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:28:25.728 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:28:25.729 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:28:25.729 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:28:25.730 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:28:25.730 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:25.731 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:25.731 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:25.732 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:28:25.732 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:25.732 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:28:25.733 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:25.733 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:25.733 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:25.733 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:25.734 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:28:25.734 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:25.734 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:25.735 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:25.735 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:25.735 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:25.735 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:25.736 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:28:25.736 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:25.736 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:25.736 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:25.737 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:25.738 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:25.738 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:25.738 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:25.739 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:25.740 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:25.740 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:25.741 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:25.741 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:28:25.741 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:25.741 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:25.742 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:25.742 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:25.742 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:25.743 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:25.743 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:25.743 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:25.744 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:25.744 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:25.744 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:25.744 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:28:25.745 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:25.745 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:25.745 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:28:25.745 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:25.746 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:25.746 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:25.746 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:28:25.746 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:25.747 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:25.747 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:25.747 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:28:25.747 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:25.748 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:28:25.748 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:28:25.748 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:28:25.748 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:25.749 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:28:25Z" 2022-09-13T07:28:25.749 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:28:25.750 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:25Z" } 2022-09-13T07:28:25.751 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:28:25.752 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:25.752 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:25.753 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:28:25.753 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:28:25.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:25.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:25.755 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:25.814 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:25.815 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:25.851 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:25.848+0000 7f73e91bd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:25.853 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:25.849+0000 7f73e91bd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:25.869 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:28:25.870 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:28:25.870 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:28:25.870 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:28:25.871 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:28:25.871 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:25.871 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:25.871 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:25.872 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:28:25.872 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:25.873 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:28:25.873 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:25.874 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:25.874 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:25.875 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:25.875 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:28:25.876 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:25.876 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:25.877 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:25.877 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:25.878 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:25.878 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:25.879 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:28:25.879 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:25.880 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:25.880 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:25.881 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:25.881 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:25.882 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:25.882 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:28:25.883 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:25.883 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:25.884 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:25.884 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:25.884 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:28:25.885 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:25.885 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:25.886 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:25.886 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:25.887 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:25.887 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:25.888 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:28:25.888 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:25.889 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:25.889 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:25.890 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:25.890 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:28:25.891 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:25.891 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:25.892 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:28:25.892 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:25.893 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:25.893 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:25.893 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:28:25.894 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:25.894 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:25.895 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:25.895 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:28:25.896 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:25.896 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:28:25.897 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:28:25.897 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:28:25.898 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:25.898 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:28:25Z" 2022-09-13T07:28:25.899 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:28:25.901 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:25Z" } 2022-09-13T07:28:25.902 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:28:25.903 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:25.904 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:25.904 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:28:25.905 DEBUG:boto:path=/yxvvuh-7/ 2022-09-13T07:28:25.906 DEBUG:boto:auth_path=/yxvvuh-7/ 2022-09-13T07:28:25.907 DEBUG:boto:Method: HEAD 2022-09-13T07:28:25.908 DEBUG:boto:Path: /yxvvuh-7/ 2022-09-13T07:28:25.909 DEBUG:boto:Data: 2022-09-13T07:28:25.910 DEBUG:boto:Headers: {} 2022-09-13T07:28:25.911 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:25.912 DEBUG:boto:Port: 8000 2022-09-13T07:28:25.913 DEBUG:boto:Params: {} 2022-09-13T07:28:25.914 DEBUG:boto:Token: None 2022-09-13T07:28:25.915 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:25 GMT /yxvvuh-7/ 2022-09-13T07:28:25.916 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pv2tbdKPNQztshOdQyk7JE4SdUM= 2022-09-13T07:28:25.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:25 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pv2tbdKPNQztshOdQyk7JE4SdUM=', 'Content-Length': '0'} 2022-09-13T07:28:25.919 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx0000044f8502c80cf3fb4-0063203119-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:25.920 DEBUG:boto:path=/yxvvuh-8/ 2022-09-13T07:28:25.921 DEBUG:boto:auth_path=/yxvvuh-8/ 2022-09-13T07:28:25.922 DEBUG:boto:Method: HEAD 2022-09-13T07:28:25.923 DEBUG:boto:Path: /yxvvuh-8/ 2022-09-13T07:28:25.924 DEBUG:boto:Data: 2022-09-13T07:28:25.925 DEBUG:boto:Headers: {} 2022-09-13T07:28:25.925 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:25.926 DEBUG:boto:Port: 8000 2022-09-13T07:28:25.927 DEBUG:boto:Params: {} 2022-09-13T07:28:25.928 DEBUG:boto:Token: None 2022-09-13T07:28:25.929 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:25 GMT /yxvvuh-8/ 2022-09-13T07:28:25.930 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KZz5Mnr+a4Zmqn8npV++MWfXwQg= 2022-09-13T07:28:25.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:25 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KZz5Mnr+a4Zmqn8npV++MWfXwQg=', 'Content-Length': '0'} 2022-09-13T07:28:25.933 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx00000b0773b9905f9476c-0063203119-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:25.934 DEBUG:boto:path=/yxvvuh-9/ 2022-09-13T07:28:25.935 DEBUG:boto:auth_path=/yxvvuh-9/ 2022-09-13T07:28:25.936 DEBUG:boto:Method: HEAD 2022-09-13T07:28:25.937 DEBUG:boto:Path: /yxvvuh-9/ 2022-09-13T07:28:25.937 DEBUG:boto:Data: 2022-09-13T07:28:25.938 DEBUG:boto:Headers: {} 2022-09-13T07:28:25.939 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:25.940 DEBUG:boto:Port: 8000 2022-09-13T07:28:25.941 DEBUG:boto:Params: {} 2022-09-13T07:28:25.942 DEBUG:boto:Token: None 2022-09-13T07:28:25.943 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:25 GMT /yxvvuh-9/ 2022-09-13T07:28:25.944 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EI2a0BRUFmGVG0vCREfTiriVzN8= 2022-09-13T07:28:25.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:25 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EI2a0BRUFmGVG0vCREfTiriVzN8=', 'Content-Length': '0'} 2022-09-13T07:28:25.946 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx000007c5fb1366b59e512-0063203119-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:25.947 DEBUG:boto:path=/yxvvuh-7/ 2022-09-13T07:28:25.948 DEBUG:boto:auth_path=/yxvvuh-7/ 2022-09-13T07:28:25.948 DEBUG:boto:Method: HEAD 2022-09-13T07:28:25.949 DEBUG:boto:Path: /yxvvuh-7/ 2022-09-13T07:28:25.949 DEBUG:boto:Data: 2022-09-13T07:28:25.950 DEBUG:boto:Headers: {} 2022-09-13T07:28:25.950 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:25.951 DEBUG:boto:Port: 8000 2022-09-13T07:28:25.951 DEBUG:boto:Params: {} 2022-09-13T07:28:25.952 DEBUG:boto:Token: None 2022-09-13T07:28:25.953 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:25 GMT /yxvvuh-7/ 2022-09-13T07:28:25.953 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pv2tbdKPNQztshOdQyk7JE4SdUM= 2022-09-13T07:28:25.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:25 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pv2tbdKPNQztshOdQyk7JE4SdUM=', 'Content-Length': '0'} 2022-09-13T07:28:25.955 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx000009313c378a0bf8cbc-0063203119-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:25.956 DEBUG:boto:path=/yxvvuh-8/ 2022-09-13T07:28:25.956 DEBUG:boto:auth_path=/yxvvuh-8/ 2022-09-13T07:28:25.957 DEBUG:boto:Method: HEAD 2022-09-13T07:28:25.957 DEBUG:boto:Path: /yxvvuh-8/ 2022-09-13T07:28:25.958 DEBUG:boto:Data: 2022-09-13T07:28:25.958 DEBUG:boto:Headers: {} 2022-09-13T07:28:25.959 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:25.959 DEBUG:boto:Port: 8000 2022-09-13T07:28:25.960 DEBUG:boto:Params: {} 2022-09-13T07:28:25.960 DEBUG:boto:Token: None 2022-09-13T07:28:25.961 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:25 GMT /yxvvuh-8/ 2022-09-13T07:28:25.962 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KZz5Mnr+a4Zmqn8npV++MWfXwQg= 2022-09-13T07:28:25.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:25 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KZz5Mnr+a4Zmqn8npV++MWfXwQg=', 'Content-Length': '0'} 2022-09-13T07:28:25.964 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx0000011dd25ae543583ea-0063203119-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:25.964 DEBUG:boto:path=/yxvvuh-9/ 2022-09-13T07:28:25.965 DEBUG:boto:auth_path=/yxvvuh-9/ 2022-09-13T07:28:25.965 DEBUG:boto:Method: HEAD 2022-09-13T07:28:25.966 DEBUG:boto:Path: /yxvvuh-9/ 2022-09-13T07:28:25.966 DEBUG:boto:Data: 2022-09-13T07:28:25.967 DEBUG:boto:Headers: {} 2022-09-13T07:28:25.968 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:25.968 DEBUG:boto:Port: 8000 2022-09-13T07:28:25.969 DEBUG:boto:Params: {} 2022-09-13T07:28:25.969 DEBUG:boto:Token: None 2022-09-13T07:28:25.970 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:25 GMT /yxvvuh-9/ 2022-09-13T07:28:25.970 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EI2a0BRUFmGVG0vCREfTiriVzN8= 2022-09-13T07:28:25.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:25 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EI2a0BRUFmGVG0vCREfTiriVzN8=', 'Content-Length': '0'} 2022-09-13T07:28:25.973 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx00000264268630a9d8028-0063203119-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:25.974 DEBUG:boto:path=/yxvvuh-7/ 2022-09-13T07:28:25.974 DEBUG:boto:auth_path=/yxvvuh-7/ 2022-09-13T07:28:25.975 DEBUG:boto:Method: HEAD 2022-09-13T07:28:25.976 DEBUG:boto:Path: /yxvvuh-7/ 2022-09-13T07:28:25.976 DEBUG:boto:Data: 2022-09-13T07:28:25.977 DEBUG:boto:Headers: {} 2022-09-13T07:28:25.977 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:25.978 DEBUG:boto:Port: 8001 2022-09-13T07:28:25.979 DEBUG:boto:Params: {} 2022-09-13T07:28:25.979 DEBUG:boto:Token: None 2022-09-13T07:28:25.980 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:25 GMT /yxvvuh-7/ 2022-09-13T07:28:25.980 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pv2tbdKPNQztshOdQyk7JE4SdUM= 2022-09-13T07:28:25.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:25 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pv2tbdKPNQztshOdQyk7JE4SdUM=', 'Content-Length': '0'} 2022-09-13T07:28:25.982 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx00000309790f59afe3c9a-0063203119-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:25.983 DEBUG:boto:path=/yxvvuh-8/ 2022-09-13T07:28:25.984 DEBUG:boto:auth_path=/yxvvuh-8/ 2022-09-13T07:28:25.984 DEBUG:boto:Method: HEAD 2022-09-13T07:28:25.985 DEBUG:boto:Path: /yxvvuh-8/ 2022-09-13T07:28:25.985 DEBUG:boto:Data: 2022-09-13T07:28:25.986 DEBUG:boto:Headers: {} 2022-09-13T07:28:25.986 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:25.987 DEBUG:boto:Port: 8001 2022-09-13T07:28:25.987 DEBUG:boto:Params: {} 2022-09-13T07:28:25.988 DEBUG:boto:Token: None 2022-09-13T07:28:25.989 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:25 GMT /yxvvuh-8/ 2022-09-13T07:28:25.989 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KZz5Mnr+a4Zmqn8npV++MWfXwQg= 2022-09-13T07:28:25.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:25 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KZz5Mnr+a4Zmqn8npV++MWfXwQg=', 'Content-Length': '0'} 2022-09-13T07:28:25.991 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx0000066dc73b78af20b4b-0063203119-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:25.992 DEBUG:boto:path=/yxvvuh-9/ 2022-09-13T07:28:25.992 DEBUG:boto:auth_path=/yxvvuh-9/ 2022-09-13T07:28:25.993 DEBUG:boto:Method: HEAD 2022-09-13T07:28:25.993 DEBUG:boto:Path: /yxvvuh-9/ 2022-09-13T07:28:25.994 DEBUG:boto:Data: 2022-09-13T07:28:25.994 DEBUG:boto:Headers: {} 2022-09-13T07:28:25.995 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:25.995 DEBUG:boto:Port: 8001 2022-09-13T07:28:25.996 DEBUG:boto:Params: {} 2022-09-13T07:28:25.996 DEBUG:boto:Token: None 2022-09-13T07:28:25.997 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:25 GMT /yxvvuh-9/ 2022-09-13T07:28:25.998 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EI2a0BRUFmGVG0vCREfTiriVzN8= 2022-09-13T07:28:25.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:25 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EI2a0BRUFmGVG0vCREfTiriVzN8=', 'Content-Length': '0'} 2022-09-13T07:28:26.000 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx000007e7c8ba9cb7a2720-0063203119-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2022-09-13T07:28:26.001 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-10 2022-09-13T07:28:26.002 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:26.002 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:26.003 DEBUG:boto:Method: PUT 2022-09-13T07:28:26.003 DEBUG:boto:Path: /yxvvuh-10/ 2022-09-13T07:28:26.004 DEBUG:boto:Data: 2022-09-13T07:28:26.004 DEBUG:boto:Headers: {} 2022-09-13T07:28:26.005 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:26.005 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.006 DEBUG:boto:Params: {} 2022-09-13T07:28:26.006 DEBUG:boto:Token: None 2022-09-13T07:28:26.007 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-10/ 2022-09-13T07:28:26.007 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Vsy9zKFdWl/RqWAKYGWER601S/4= 2022-09-13T07:28:26.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Vsy9zKFdWl/RqWAKYGWER601S/4=', 'Content-Length': '0'} 2022-09-13T07:28:26.023 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000522907623ea4e4d8-006320311a-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.024 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-11 2022-09-13T07:28:26.025 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:26.026 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:26.027 DEBUG:boto:Method: PUT 2022-09-13T07:28:26.028 DEBUG:boto:Path: /yxvvuh-11/ 2022-09-13T07:28:26.029 DEBUG:boto:Data: 2022-09-13T07:28:26.030 DEBUG:boto:Headers: {} 2022-09-13T07:28:26.031 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:26.032 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.032 DEBUG:boto:Params: {} 2022-09-13T07:28:26.033 DEBUG:boto:Token: None 2022-09-13T07:28:26.034 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-11/ 2022-09-13T07:28:26.035 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FVHjEbt1FlLv5QDIw5yEV4DJ+EI= 2022-09-13T07:28:26.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FVHjEbt1FlLv5QDIw5yEV4DJ+EI=', 'Content-Length': '0'} 2022-09-13T07:28:26.067 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000e4d1dbb1881995f-006320311a-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.068 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-12 2022-09-13T07:28:26.068 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:26.069 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:26.069 DEBUG:boto:Method: PUT 2022-09-13T07:28:26.070 DEBUG:boto:Path: /yxvvuh-12/ 2022-09-13T07:28:26.070 DEBUG:boto:Data: 2022-09-13T07:28:26.071 DEBUG:boto:Headers: {} 2022-09-13T07:28:26.072 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:26.072 DEBUG:boto:Port: 8001 2022-09-13T07:28:26.073 DEBUG:boto:Params: {} 2022-09-13T07:28:26.073 DEBUG:boto:Token: None 2022-09-13T07:28:26.074 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-12/ 2022-09-13T07:28:26.074 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TzWqAZm5K0eKyEOk4UcChliYRUc= 2022-09-13T07:28:26.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TzWqAZm5K0eKyEOk4UcChliYRUc=', 'Content-Length': '0'} 2022-09-13T07:28:26.099 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ffb309e687d5ea31-006320311a-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.101 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:26.101 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:26.102 DEBUG:boto:Method: HEAD 2022-09-13T07:28:26.102 DEBUG:boto:Path: /yxvvuh-10/ 2022-09-13T07:28:26.103 DEBUG:boto:Data: 2022-09-13T07:28:26.103 DEBUG:boto:Headers: {} 2022-09-13T07:28:26.104 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:26.104 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.105 DEBUG:boto:Params: {} 2022-09-13T07:28:26.106 DEBUG:boto:Token: None 2022-09-13T07:28:26.106 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-10/ 2022-09-13T07:28:26.107 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YpQTGje+kQPQqzLJMXvtL9cREoI= 2022-09-13T07:28:26.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YpQTGje+kQPQqzLJMXvtL9cREoI=', 'Content-Length': '0'} 2022-09-13T07:28:26.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ae094e7511def0c-006320311a-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.111 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:26.112 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:26.113 DEBUG:boto:Method: PUT 2022-09-13T07:28:26.114 DEBUG:boto:Path: /yxvvuh-10/myobj 2022-09-13T07:28:26.114 DEBUG:boto:Data: 2022-09-13T07:28:26.115 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-13T07:28:26.115 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:26.116 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.116 DEBUG:boto:Params: {} 2022-09-13T07:28:26.117 DEBUG:boto:Token: None 2022-09-13T07:28:26.118 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-10/myobj 2022-09-13T07:28:26.118 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IaqaZiBzAajsljTn3iNOxOXU/G8= 2022-09-13T07:28:26.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IaqaZiBzAajsljTn3iNOxOXU/G8='} 2022-09-13T07:28:26.129 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed3d2ddce22b1e5d-006320311a-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.130 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:26.130 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:26.131 DEBUG:boto:Method: HEAD 2022-09-13T07:28:26.132 DEBUG:boto:Path: /yxvvuh-10/ 2022-09-13T07:28:26.133 DEBUG:boto:Data: 2022-09-13T07:28:26.133 DEBUG:boto:Headers: {} 2022-09-13T07:28:26.134 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:26.135 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.136 DEBUG:boto:Params: {} 2022-09-13T07:28:26.136 DEBUG:boto:Token: None 2022-09-13T07:28:26.137 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-10/ 2022-09-13T07:28:26.138 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YpQTGje+kQPQqzLJMXvtL9cREoI= 2022-09-13T07:28:26.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YpQTGje+kQPQqzLJMXvtL9cREoI=', 'Content-Length': '0'} 2022-09-13T07:28:26.141 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004af8abebebdcc9b7-006320311a-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.142 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:26.143 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:26.144 DEBUG:boto:Method: PUT 2022-09-13T07:28:26.145 DEBUG:boto:Path: /yxvvuh-10/_myobj 2022-09-13T07:28:26.146 DEBUG:boto:Data: 2022-09-13T07:28:26.147 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-13T07:28:26.148 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:26.149 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.149 DEBUG:boto:Params: {} 2022-09-13T07:28:26.150 DEBUG:boto:Token: None 2022-09-13T07:28:26.151 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-10/_myobj 2022-09-13T07:28:26.152 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:H7ENgBlnsxFIWox0UP+dXuAnfFc= 2022-09-13T07:28:26.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:H7ENgBlnsxFIWox0UP+dXuAnfFc='} 2022-09-13T07:28:26.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000712ccaa622a74250-006320311a-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.203 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:26.203 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:26.204 DEBUG:boto:Method: HEAD 2022-09-13T07:28:26.204 DEBUG:boto:Path: /yxvvuh-10/ 2022-09-13T07:28:26.205 DEBUG:boto:Data: 2022-09-13T07:28:26.205 DEBUG:boto:Headers: {} 2022-09-13T07:28:26.206 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:26.206 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.207 DEBUG:boto:Params: {} 2022-09-13T07:28:26.208 DEBUG:boto:Token: None 2022-09-13T07:28:26.209 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-10/ 2022-09-13T07:28:26.209 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YpQTGje+kQPQqzLJMXvtL9cREoI= 2022-09-13T07:28:26.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YpQTGje+kQPQqzLJMXvtL9cREoI=', 'Content-Length': '0'} 2022-09-13T07:28:26.212 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005952fe68ca2ea792-006320311a-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.214 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:26.214 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:26.215 DEBUG:boto:Method: PUT 2022-09-13T07:28:26.215 DEBUG:boto:Path: /yxvvuh-10/%3A 2022-09-13T07:28:26.216 DEBUG:boto:Data: 2022-09-13T07:28:26.216 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-13T07:28:26.217 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:26.217 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.218 DEBUG:boto:Params: {} 2022-09-13T07:28:26.218 DEBUG:boto:Token: None 2022-09-13T07:28:26.219 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-10/%3A 2022-09-13T07:28:26.220 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0sxZ+tnXHkM1we2enRHHuK9XFXs= 2022-09-13T07:28:26.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0sxZ+tnXHkM1we2enRHHuK9XFXs='} 2022-09-13T07:28:26.229 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009887fd762e054fc1-006320311a-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.231 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:26.231 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:26.232 DEBUG:boto:Method: HEAD 2022-09-13T07:28:26.233 DEBUG:boto:Path: /yxvvuh-10/ 2022-09-13T07:28:26.234 DEBUG:boto:Data: 2022-09-13T07:28:26.235 DEBUG:boto:Headers: {} 2022-09-13T07:28:26.236 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:26.237 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.238 DEBUG:boto:Params: {} 2022-09-13T07:28:26.239 DEBUG:boto:Token: None 2022-09-13T07:28:26.240 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-10/ 2022-09-13T07:28:26.241 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YpQTGje+kQPQqzLJMXvtL9cREoI= 2022-09-13T07:28:26.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YpQTGje+kQPQqzLJMXvtL9cREoI=', 'Content-Length': '0'} 2022-09-13T07:28:26.245 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd4887ca5477b5f9-006320311a-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.246 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:26.247 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:26.248 DEBUG:boto:Method: PUT 2022-09-13T07:28:26.249 DEBUG:boto:Path: /yxvvuh-10/%26 2022-09-13T07:28:26.250 DEBUG:boto:Data: 2022-09-13T07:28:26.251 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-13T07:28:26.252 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:26.253 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.254 DEBUG:boto:Params: {} 2022-09-13T07:28:26.255 DEBUG:boto:Token: None 2022-09-13T07:28:26.256 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-10/%26 2022-09-13T07:28:26.257 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Y7HZEIKUeZKtKZuq2SnxDhI2tFU= 2022-09-13T07:28:26.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Y7HZEIKUeZKtKZuq2SnxDhI2tFU='} 2022-09-13T07:28:26.265 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba6d69ce60c80d9b-006320311a-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.266 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:26.267 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:26.268 DEBUG:boto:Method: HEAD 2022-09-13T07:28:26.269 DEBUG:boto:Path: /yxvvuh-11/ 2022-09-13T07:28:26.270 DEBUG:boto:Data: 2022-09-13T07:28:26.271 DEBUG:boto:Headers: {} 2022-09-13T07:28:26.272 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:26.272 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.274 DEBUG:boto:Params: {} 2022-09-13T07:28:26.275 DEBUG:boto:Token: None 2022-09-13T07:28:26.276 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-11/ 2022-09-13T07:28:26.277 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DFgRg32dfp+5Bvu/DlaE2oozsRM= 2022-09-13T07:28:26.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DFgRg32dfp+5Bvu/DlaE2oozsRM=', 'Content-Length': '0'} 2022-09-13T07:28:26.281 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005cb1509b8782451f-006320311a-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.283 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:26.284 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:26.285 DEBUG:boto:Method: PUT 2022-09-13T07:28:26.285 DEBUG:boto:Path: /yxvvuh-11/myobj 2022-09-13T07:28:26.286 DEBUG:boto:Data: 2022-09-13T07:28:26.287 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-13T07:28:26.287 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:26.288 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.288 DEBUG:boto:Params: {} 2022-09-13T07:28:26.289 DEBUG:boto:Token: None 2022-09-13T07:28:26.289 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-11/myobj 2022-09-13T07:28:26.290 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZWZ23KLpMwxodFOaCjGs0mnAlmU= 2022-09-13T07:28:26.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZWZ23KLpMwxodFOaCjGs0mnAlmU='} 2022-09-13T07:28:26.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2b4ad461ce8b352-006320311a-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.303 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:26.303 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:26.304 DEBUG:boto:Method: HEAD 2022-09-13T07:28:26.304 DEBUG:boto:Path: /yxvvuh-11/ 2022-09-13T07:28:26.305 DEBUG:boto:Data: 2022-09-13T07:28:26.305 DEBUG:boto:Headers: {} 2022-09-13T07:28:26.306 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:26.306 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.307 DEBUG:boto:Params: {} 2022-09-13T07:28:26.307 DEBUG:boto:Token: None 2022-09-13T07:28:26.308 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-11/ 2022-09-13T07:28:26.309 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DFgRg32dfp+5Bvu/DlaE2oozsRM= 2022-09-13T07:28:26.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DFgRg32dfp+5Bvu/DlaE2oozsRM=', 'Content-Length': '0'} 2022-09-13T07:28:26.312 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0dd8b377e4b6dbe-006320311a-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.313 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:26.313 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:26.314 DEBUG:boto:Method: PUT 2022-09-13T07:28:26.315 DEBUG:boto:Path: /yxvvuh-11/_myobj 2022-09-13T07:28:26.315 DEBUG:boto:Data: 2022-09-13T07:28:26.316 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-13T07:28:26.316 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:26.317 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.317 DEBUG:boto:Params: {} 2022-09-13T07:28:26.318 DEBUG:boto:Token: None 2022-09-13T07:28:26.319 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-11/_myobj 2022-09-13T07:28:26.320 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nmIp4aXA70RJcb4V3f7dAX3xxrY= 2022-09-13T07:28:26.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nmIp4aXA70RJcb4V3f7dAX3xxrY='} 2022-09-13T07:28:26.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8ba97c7e19e027c-006320311a-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.363 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:26.364 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:26.364 DEBUG:boto:Method: HEAD 2022-09-13T07:28:26.365 DEBUG:boto:Path: /yxvvuh-11/ 2022-09-13T07:28:26.365 DEBUG:boto:Data: 2022-09-13T07:28:26.365 DEBUG:boto:Headers: {} 2022-09-13T07:28:26.366 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:26.366 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.367 DEBUG:boto:Params: {} 2022-09-13T07:28:26.367 DEBUG:boto:Token: None 2022-09-13T07:28:26.368 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-11/ 2022-09-13T07:28:26.368 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DFgRg32dfp+5Bvu/DlaE2oozsRM= 2022-09-13T07:28:26.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DFgRg32dfp+5Bvu/DlaE2oozsRM=', 'Content-Length': '0'} 2022-09-13T07:28:26.371 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004758267abb2d1eaf-006320311a-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.372 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:26.372 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:26.373 DEBUG:boto:Method: PUT 2022-09-13T07:28:26.373 DEBUG:boto:Path: /yxvvuh-11/%3A 2022-09-13T07:28:26.374 DEBUG:boto:Data: 2022-09-13T07:28:26.374 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-13T07:28:26.375 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:26.375 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.376 DEBUG:boto:Params: {} 2022-09-13T07:28:26.376 DEBUG:boto:Token: None 2022-09-13T07:28:26.377 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-11/%3A 2022-09-13T07:28:26.377 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VuWdb/bHOxHvv/ezdAvHvxz2faU= 2022-09-13T07:28:26.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VuWdb/bHOxHvv/ezdAvHvxz2faU='} 2022-09-13T07:28:26.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000378dbd4596d402de-006320311a-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.387 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:26.388 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:26.388 DEBUG:boto:Method: HEAD 2022-09-13T07:28:26.389 DEBUG:boto:Path: /yxvvuh-11/ 2022-09-13T07:28:26.389 DEBUG:boto:Data: 2022-09-13T07:28:26.389 DEBUG:boto:Headers: {} 2022-09-13T07:28:26.390 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:26.390 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.391 DEBUG:boto:Params: {} 2022-09-13T07:28:26.391 DEBUG:boto:Token: None 2022-09-13T07:28:26.392 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-11/ 2022-09-13T07:28:26.392 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DFgRg32dfp+5Bvu/DlaE2oozsRM= 2022-09-13T07:28:26.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DFgRg32dfp+5Bvu/DlaE2oozsRM=', 'Content-Length': '0'} 2022-09-13T07:28:26.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d41ad3d28a295c66-006320311a-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.396 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:26.397 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:26.397 DEBUG:boto:Method: PUT 2022-09-13T07:28:26.398 DEBUG:boto:Path: /yxvvuh-11/%26 2022-09-13T07:28:26.398 DEBUG:boto:Data: 2022-09-13T07:28:26.399 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-13T07:28:26.399 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:26.399 DEBUG:boto:Port: 8000 2022-09-13T07:28:26.400 DEBUG:boto:Params: {} 2022-09-13T07:28:26.400 DEBUG:boto:Token: None 2022-09-13T07:28:26.401 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-11/%26 2022-09-13T07:28:26.402 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7C6n3Cfs/pL+alc3THhslGy6mhM= 2022-09-13T07:28:26.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7C6n3Cfs/pL+alc3THhslGy6mhM='} 2022-09-13T07:28:26.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e7fcd4136fb40f6-006320311a-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.408 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:26.409 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:26.409 DEBUG:boto:Method: HEAD 2022-09-13T07:28:26.410 DEBUG:boto:Path: /yxvvuh-12/ 2022-09-13T07:28:26.410 DEBUG:boto:Data: 2022-09-13T07:28:26.411 DEBUG:boto:Headers: {} 2022-09-13T07:28:26.411 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:26.412 DEBUG:boto:Port: 8001 2022-09-13T07:28:26.412 DEBUG:boto:Params: {} 2022-09-13T07:28:26.413 DEBUG:boto:Token: None 2022-09-13T07:28:26.413 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-12/ 2022-09-13T07:28:26.414 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3yIe7WvfahqgQ5FsJHiuErQvHic= 2022-09-13T07:28:26.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3yIe7WvfahqgQ5FsJHiuErQvHic=', 'Content-Length': '0'} 2022-09-13T07:28:26.417 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9a38f08fb7c52eb-006320311a-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.417 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:26.418 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:26.418 DEBUG:boto:Method: PUT 2022-09-13T07:28:26.419 DEBUG:boto:Path: /yxvvuh-12/myobj 2022-09-13T07:28:26.419 DEBUG:boto:Data: 2022-09-13T07:28:26.420 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-13T07:28:26.420 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:26.421 DEBUG:boto:Port: 8001 2022-09-13T07:28:26.421 DEBUG:boto:Params: {} 2022-09-13T07:28:26.422 DEBUG:boto:Token: None 2022-09-13T07:28:26.422 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-12/myobj 2022-09-13T07:28:26.423 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:B29kmLm0w0Z4LYzyFU5xbF5JSRU= 2022-09-13T07:28:26.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:B29kmLm0w0Z4LYzyFU5xbF5JSRU='} 2022-09-13T07:28:26.435 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae991c02519a2081-006320311a-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.436 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:26.436 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:26.437 DEBUG:boto:Method: HEAD 2022-09-13T07:28:26.437 DEBUG:boto:Path: /yxvvuh-12/ 2022-09-13T07:28:26.438 DEBUG:boto:Data: 2022-09-13T07:28:26.438 DEBUG:boto:Headers: {} 2022-09-13T07:28:26.439 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:26.439 DEBUG:boto:Port: 8001 2022-09-13T07:28:26.439 DEBUG:boto:Params: {} 2022-09-13T07:28:26.440 DEBUG:boto:Token: None 2022-09-13T07:28:26.441 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-12/ 2022-09-13T07:28:26.441 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3yIe7WvfahqgQ5FsJHiuErQvHic= 2022-09-13T07:28:26.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3yIe7WvfahqgQ5FsJHiuErQvHic=', 'Content-Length': '0'} 2022-09-13T07:28:26.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9e99157c2e7ddeb-006320311a-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.445 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:26.445 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:26.446 DEBUG:boto:Method: PUT 2022-09-13T07:28:26.446 DEBUG:boto:Path: /yxvvuh-12/_myobj 2022-09-13T07:28:26.447 DEBUG:boto:Data: 2022-09-13T07:28:26.447 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-13T07:28:26.448 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:26.448 DEBUG:boto:Port: 8001 2022-09-13T07:28:26.449 DEBUG:boto:Params: {} 2022-09-13T07:28:26.449 DEBUG:boto:Token: None 2022-09-13T07:28:26.450 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-12/_myobj 2022-09-13T07:28:26.451 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j3vEByazNkl5OFF7Kfy+CjHHA9s= 2022-09-13T07:28:26.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j3vEByazNkl5OFF7Kfy+CjHHA9s='} 2022-09-13T07:28:26.467 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009353f7590dda6d63-006320311a-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.467 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:26.468 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:26.468 DEBUG:boto:Method: HEAD 2022-09-13T07:28:26.469 DEBUG:boto:Path: /yxvvuh-12/ 2022-09-13T07:28:26.469 DEBUG:boto:Data: 2022-09-13T07:28:26.470 DEBUG:boto:Headers: {} 2022-09-13T07:28:26.470 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:26.471 DEBUG:boto:Port: 8001 2022-09-13T07:28:26.471 DEBUG:boto:Params: {} 2022-09-13T07:28:26.472 DEBUG:boto:Token: None 2022-09-13T07:28:26.473 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-12/ 2022-09-13T07:28:26.473 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3yIe7WvfahqgQ5FsJHiuErQvHic= 2022-09-13T07:28:26.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3yIe7WvfahqgQ5FsJHiuErQvHic=', 'Content-Length': '0'} 2022-09-13T07:28:26.476 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a0d159e79d91a28-006320311a-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.477 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:26.478 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:26.478 DEBUG:boto:Method: PUT 2022-09-13T07:28:26.479 DEBUG:boto:Path: /yxvvuh-12/%3A 2022-09-13T07:28:26.479 DEBUG:boto:Data: 2022-09-13T07:28:26.480 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-13T07:28:26.480 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:26.481 DEBUG:boto:Port: 8001 2022-09-13T07:28:26.481 DEBUG:boto:Params: {} 2022-09-13T07:28:26.482 DEBUG:boto:Token: None 2022-09-13T07:28:26.483 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-12/%3A 2022-09-13T07:28:26.483 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IJlo6k8sdff2oFxdD7VwkOF+/2g= 2022-09-13T07:28:26.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IJlo6k8sdff2oFxdD7VwkOF+/2g='} 2022-09-13T07:28:26.495 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3935845aec6d028-006320311a-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.496 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:26.496 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:26.497 DEBUG:boto:Method: HEAD 2022-09-13T07:28:26.498 DEBUG:boto:Path: /yxvvuh-12/ 2022-09-13T07:28:26.498 DEBUG:boto:Data: 2022-09-13T07:28:26.499 DEBUG:boto:Headers: {} 2022-09-13T07:28:26.499 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:26.500 DEBUG:boto:Port: 8001 2022-09-13T07:28:26.500 DEBUG:boto:Params: {} 2022-09-13T07:28:26.501 DEBUG:boto:Token: None 2022-09-13T07:28:26.501 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-12/ 2022-09-13T07:28:26.502 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3yIe7WvfahqgQ5FsJHiuErQvHic= 2022-09-13T07:28:26.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3yIe7WvfahqgQ5FsJHiuErQvHic=', 'Content-Length': '0'} 2022-09-13T07:28:26.505 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5d67324ecec931d-006320311a-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.506 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:26.507 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:26.507 DEBUG:boto:Method: PUT 2022-09-13T07:28:26.508 DEBUG:boto:Path: /yxvvuh-12/%26 2022-09-13T07:28:26.508 DEBUG:boto:Data: 2022-09-13T07:28:26.509 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-13T07:28:26.509 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:26.510 DEBUG:boto:Port: 8001 2022-09-13T07:28:26.510 DEBUG:boto:Params: {} 2022-09-13T07:28:26.511 DEBUG:boto:Token: None 2022-09-13T07:28:26.512 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 13 Sep 2022 07:28:26 GMT /yxvvuh-12/%26 2022-09-13T07:28:26.512 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7f+4sryN7bN787Rq2o3m1nsPbwI= 2022-09-13T07:28:26.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:28:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7f+4sryN7bN787Rq2o3m1nsPbwI='} 2022-09-13T07:28:26.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3c5840789aac641-006320311a-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:26.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:26.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:26.560 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:26.617 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:26.618 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:26.655 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:28:26.680 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:28:26.681 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:26.681 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:26.682 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:28:26.682 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:26.682 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:26.683 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:26.683 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:28:26.684 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:26.684 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:26.685 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054106.087870_70.1", 2022-09-13T07:28:26.685 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:26.087870Z" 2022-09-13T07:28:26.686 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:26.686 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:26.687 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:28:26.688 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T07:28:26.688 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:26.688 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:28:26.689 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054106.087870_70.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:26.690 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:28:26.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:26.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:26.692 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:26.745 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:26.746 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:26.781 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:26.779+0000 7f69c7d87600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:26.782 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:26.780+0000 7f69c7d87600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:26.804 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:28:26.805 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:28:26.805 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:28:26.805 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:28:26.805 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:28:26.806 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:26.806 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:26.806 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:26.807 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:28:26.807 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:26.807 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:28:26.808 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:26.808 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:26.808 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:26.808 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:26.809 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:28:26.809 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:26.810 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:26.810 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:26.811 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:26.811 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:26.811 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:26.812 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:28:26.812 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:26.812 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:26.813 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:26.813 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:26.813 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:26.813 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:26.814 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:26.814 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:26.814 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:26.815 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:26.815 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:26.815 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:28:26.815 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:26.816 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:26.816 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:26.817 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:26.817 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:26.817 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:26.817 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:26.818 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:26.818 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:26.818 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:26.818 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:26.819 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:28:26.819 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:26.819 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:26.820 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:28:26.820 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:26.820 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:26.820 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:26.821 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:28:26.821 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:26.821 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:26.821 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:26.822 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:28:26.822 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:26.822 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:28:26.823 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:28:26.823 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:28:26.823 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:26.823 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:28:26Z" 2022-09-13T07:28:26.824 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:28:26.824 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:26Z" } 2022-09-13T07:28:26.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:28:26.826 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054106.087870_70.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:26.826 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:26.827 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663054106.087870_70.1 target= 2022-09-13T07:28:31.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:31.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:31.830 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:31.887 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:31.888 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:31.917 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:31.913+0000 7f824814d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:31.917 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:31.914+0000 7f824814d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:31.929 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:28:31.930 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:28:31.930 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:28:31.930 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:28:31.931 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:28:31.931 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:31.931 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:31.931 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:31.932 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:28:31.932 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:31.933 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:28:31.933 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:31.933 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:31.934 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:31.934 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:31.935 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:28:31.935 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:31.935 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:31.935 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:31.936 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:31.936 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:31.937 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:31.937 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:28:31.937 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:31.937 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:31.938 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:31.938 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:31.938 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:31.939 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:31.939 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:31.939 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:31.939 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:31.940 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:31.940 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:31.940 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:28:31.940 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:31.941 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:31.941 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054106.087870_70.1", 2022-09-13T07:28:31.941 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:31.941 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:31.942 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:31.942 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:26.087870Z", 2022-09-13T07:28:31.943 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:31.943 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:31.943 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:31.943 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:31.944 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:28:31.944 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:31.944 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:31.944 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:28:31.945 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:31.945 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:31.945 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:31.945 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:28:31.946 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:31.946 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:31.946 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:31.946 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:28:31.947 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:31.947 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:28:31.947 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:28:31.947 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:28:31.948 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:31.948 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:28:31Z" 2022-09-13T07:28:31.948 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:28:31.949 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054106.087870_70.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:26.087870Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:31Z" } 2022-09-13T07:28:31.950 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054106.087870_70.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:26.087870Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:28:31.951 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054106.087870_70.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:31.951 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054106.087870_70.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:31.952 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:28:31.952 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:28:31.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:31.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:31.954 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:32.013 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:32.014 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:32.043 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:32.040+0000 7f538fdcd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:32.044 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:32.041+0000 7f538fdcd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:32.057 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:28:32.057 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:28:32.058 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:28:32.058 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:28:32.058 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:28:32.059 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:32.059 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:32.060 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:32.060 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:28:32.061 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:32.061 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:28:32.062 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:32.063 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:32.063 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:32.064 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:32.064 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:28:32.065 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:32.065 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:32.066 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:32.066 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:32.067 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:32.067 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:32.068 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:28:32.068 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:32.069 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:32.069 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:32.070 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:32.070 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:32.071 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:32.071 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:28:32.072 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:32.072 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:32.073 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:32.073 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:32.074 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:28:32.074 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:32.075 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:32.076 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054106.087870_70.1", 2022-09-13T07:28:32.076 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:32.077 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:32.077 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:32.078 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:26.087870Z", 2022-09-13T07:28:32.078 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:32.079 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:32.079 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:32.080 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:32.080 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:28:32.081 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:28:32.081 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:28:32.082 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:28:32.082 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:28:32.083 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:28:32.083 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:28:32.084 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:28:32.084 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:28:32.085 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:32.085 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:32.086 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:28:32.086 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:32.087 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:28:32.087 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:28:32.088 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:28:32.088 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:32.089 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:28:32Z" 2022-09-13T07:28:32.089 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:28:32.091 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054106.087870_70.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:26.087870Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:32Z" } 2022-09-13T07:28:32.092 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054106.087870_70.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:26.087870Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:28:32.092 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054106.087870_70.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:32.093 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054106.087870_70.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:32.093 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:28:32.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:28:32.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:28:32.095 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:28:32.153 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:32.153 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:32.196 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:32.192+0000 7f47b96be600 1 bucket sync caught up with source: 2022-09-13T07:28:32.196 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.7.5, , 00000000001.2.5, , , , 00000000001.8.5, 00000000001.7.5, , ] 2022-09-13T07:28:32.197 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000001.7.5, , 00000000001.2.5, , , , 00000000001.8.5, 00000000001.7.5, , ] 2022-09-13T07:28:32.197 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:32.192+0000 7f47b96be600 0 bucket checkpoint complete 2022-09-13T07:28:32.208 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-10 zones={test-zone2, test-zone1} 2022-09-13T07:28:32.209 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.210 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.211 DEBUG:boto:Method: HEAD 2022-09-13T07:28:32.212 DEBUG:boto:Path: /yxvvuh-10/ 2022-09-13T07:28:32.212 DEBUG:boto:Data: 2022-09-13T07:28:32.213 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.214 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.215 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.216 DEBUG:boto:Params: {} 2022-09-13T07:28:32.217 DEBUG:boto:Token: None 2022-09-13T07:28:32.218 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/ 2022-09-13T07:28:32.219 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zh3W+fnd9W1FkO0hHOuN5AFEXRU= 2022-09-13T07:28:32.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zh3W+fnd9W1FkO0hHOuN5AFEXRU=', 'Content-Length': '0'} 2022-09-13T07:28:32.223 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1a3d27c4e13a2f4-0063203120-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.224 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.225 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.226 DEBUG:boto:Method: HEAD 2022-09-13T07:28:32.227 DEBUG:boto:Path: /yxvvuh-10/ 2022-09-13T07:28:32.228 DEBUG:boto:Data: 2022-09-13T07:28:32.228 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.229 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.230 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.231 DEBUG:boto:Params: {} 2022-09-13T07:28:32.232 DEBUG:boto:Token: None 2022-09-13T07:28:32.233 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/ 2022-09-13T07:28:32.234 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zh3W+fnd9W1FkO0hHOuN5AFEXRU= 2022-09-13T07:28:32.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zh3W+fnd9W1FkO0hHOuN5AFEXRU=', 'Content-Length': '0'} 2022-09-13T07:28:32.238 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009baf2f2925c849bb-0063203120-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.240 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:28:32.240 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.242 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.242 DEBUG:boto:path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:32.243 DEBUG:boto:auth_path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:32.243 DEBUG:boto:Method: GET 2022-09-13T07:28:32.244 DEBUG:boto:Path: /yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:32.245 DEBUG:boto:Data: 2022-09-13T07:28:32.245 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.246 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.246 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.247 DEBUG:boto:Params: {} 2022-09-13T07:28:32.248 DEBUG:boto:Token: None 2022-09-13T07:28:32.249 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/?versions 2022-09-13T07:28:32.250 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UPWpAyDUWPtF83WDVW0LyvWRRmA= 2022-09-13T07:28:32.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UPWpAyDUWPtF83WDVW0LyvWRRmA=', 'Content-Length': '0'} 2022-09-13T07:28:32.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091a02d278f6aa4d5-0063203120-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.254 DEBUG:boto:b'yxvvuh-10999false&nulltrue2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:32.258 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-13T07:28:32.259 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-13T07:28:32.260 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-13T07:28:32.260 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:28:32.261 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:28:32.261 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.262 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.262 DEBUG:boto:path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:32.263 DEBUG:boto:auth_path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:32.264 DEBUG:boto:Method: GET 2022-09-13T07:28:32.264 DEBUG:boto:Path: /yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:32.265 DEBUG:boto:Data: 2022-09-13T07:28:32.266 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.267 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.268 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.269 DEBUG:boto:Params: {} 2022-09-13T07:28:32.270 DEBUG:boto:Token: None 2022-09-13T07:28:32.270 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/?versions 2022-09-13T07:28:32.272 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UPWpAyDUWPtF83WDVW0LyvWRRmA= 2022-09-13T07:28:32.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UPWpAyDUWPtF83WDVW0LyvWRRmA=', 'Content-Length': '0'} 2022-09-13T07:28:32.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cb3e8285bd0417c-0063203120-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.278 DEBUG:boto:b'yxvvuh-10999false&nulltrue2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:32.279 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-13T07:28:32.279 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-13T07:28:32.280 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-13T07:28:32.281 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:28:32.281 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.282 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.283 DEBUG:boto:path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:32.284 DEBUG:boto:auth_path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:32.284 DEBUG:boto:Method: GET 2022-09-13T07:28:32.285 DEBUG:boto:Path: /yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:32.285 DEBUG:boto:Data: 2022-09-13T07:28:32.286 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.286 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.287 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.287 DEBUG:boto:Params: {} 2022-09-13T07:28:32.288 DEBUG:boto:Token: None 2022-09-13T07:28:32.288 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/?versions 2022-09-13T07:28:32.289 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UPWpAyDUWPtF83WDVW0LyvWRRmA= 2022-09-13T07:28:32.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UPWpAyDUWPtF83WDVW0LyvWRRmA=', 'Content-Length': '0'} 2022-09-13T07:28:32.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090a5872384029aac-0063203120-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.294 DEBUG:boto:b'yxvvuh-10999false&nulltrue2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:32.295 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.295 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.296 DEBUG:boto:path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:32.296 DEBUG:boto:auth_path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:32.297 DEBUG:boto:Method: GET 2022-09-13T07:28:32.297 DEBUG:boto:Path: /yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:32.298 DEBUG:boto:Data: 2022-09-13T07:28:32.299 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.299 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.301 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.301 DEBUG:boto:Params: {} 2022-09-13T07:28:32.302 DEBUG:boto:Token: None 2022-09-13T07:28:32.302 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/?versions 2022-09-13T07:28:32.303 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UPWpAyDUWPtF83WDVW0LyvWRRmA= 2022-09-13T07:28:32.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UPWpAyDUWPtF83WDVW0LyvWRRmA=', 'Content-Length': '0'} 2022-09-13T07:28:32.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014f633ff8bc71d81-0063203120-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.307 DEBUG:boto:b'yxvvuh-10999false&nulltrue2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:32.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:32.309 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:32.309 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:32.310 DEBUG:boto:path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.310 DEBUG:boto:auth_path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.311 DEBUG:boto:Method: GET 2022-09-13T07:28:32.311 DEBUG:boto:Path: /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.312 DEBUG:boto:Data: 2022-09-13T07:28:32.313 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.313 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.314 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.314 DEBUG:boto:Params: {} 2022-09-13T07:28:32.315 DEBUG:boto:Token: None 2022-09-13T07:28:32.315 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.316 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vJjNIVlte+y/twyiu6G+e/zLjBY= 2022-09-13T07:28:32.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vJjNIVlte+y/twyiu6G+e/zLjBY=', 'Content-Length': '0'} 2022-09-13T07:28:32.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d52c33b1bd3e729a-0063203120-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.362 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:32.363 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:32.364 DEBUG:boto:path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.365 DEBUG:boto:auth_path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.366 DEBUG:boto:Method: GET 2022-09-13T07:28:32.366 DEBUG:boto:Path: /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.367 DEBUG:boto:Data: 2022-09-13T07:28:32.368 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.369 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.369 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.370 DEBUG:boto:Params: {} 2022-09-13T07:28:32.371 DEBUG:boto:Token: None 2022-09-13T07:28:32.371 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.372 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vJjNIVlte+y/twyiu6G+e/zLjBY= 2022-09-13T07:28:32.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vJjNIVlte+y/twyiu6G+e/zLjBY=', 'Content-Length': '0'} 2022-09-13T07:28:32.375 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8da8325604977da-0063203120-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.376 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:32.376 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:32.377 DEBUG:boto:path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.377 DEBUG:boto:auth_path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.377 DEBUG:boto:Method: HEAD 2022-09-13T07:28:32.378 DEBUG:boto:Path: /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.378 DEBUG:boto:Data: 2022-09-13T07:28:32.379 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.379 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.380 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.381 DEBUG:boto:Params: {} 2022-09-13T07:28:32.381 DEBUG:boto:Token: None 2022-09-13T07:28:32.382 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.382 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hsec5eBrJaAVtTXl6BoxeABwxS4= 2022-09-13T07:28:32.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hsec5eBrJaAVtTXl6BoxeABwxS4=', 'Content-Length': '0'} 2022-09-13T07:28:32.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b351f3a1455ed64f-0063203120-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.386 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:32.386 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:32.387 DEBUG:boto:path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.388 DEBUG:boto:auth_path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.388 DEBUG:boto:Method: HEAD 2022-09-13T07:28:32.389 DEBUG:boto:Path: /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.389 DEBUG:boto:Data: 2022-09-13T07:28:32.390 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.390 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.391 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.391 DEBUG:boto:Params: {} 2022-09-13T07:28:32.392 DEBUG:boto:Token: None 2022-09-13T07:28:32.392 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.393 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hsec5eBrJaAVtTXl6BoxeABwxS4= 2022-09-13T07:28:32.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hsec5eBrJaAVtTXl6BoxeABwxS4=', 'Content-Length': '0'} 2022-09-13T07:28:32.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006575d532a02a2869-0063203120-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:32.397 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:32.398 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:32.399 DEBUG:boto:path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.399 DEBUG:boto:auth_path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.400 DEBUG:boto:Method: GET 2022-09-13T07:28:32.401 DEBUG:boto:Path: /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.401 DEBUG:boto:Data: 2022-09-13T07:28:32.402 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.403 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.404 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.405 DEBUG:boto:Params: {} 2022-09-13T07:28:32.406 DEBUG:boto:Token: None 2022-09-13T07:28:32.407 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.408 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vJjNIVlte+y/twyiu6G+e/zLjBY= 2022-09-13T07:28:32.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vJjNIVlte+y/twyiu6G+e/zLjBY=', 'Content-Length': '0'} 2022-09-13T07:28:32.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef4ca7aa65f87e61-0063203120-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.413 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:32.413 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:32.414 DEBUG:boto:path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.414 DEBUG:boto:auth_path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.415 DEBUG:boto:Method: GET 2022-09-13T07:28:32.416 DEBUG:boto:Path: /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.417 DEBUG:boto:Data: 2022-09-13T07:28:32.417 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.418 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.418 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.419 DEBUG:boto:Params: {} 2022-09-13T07:28:32.420 DEBUG:boto:Token: None 2022-09-13T07:28:32.421 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:32.421 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vJjNIVlte+y/twyiu6G+e/zLjBY= 2022-09-13T07:28:32.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vJjNIVlte+y/twyiu6G+e/zLjBY=', 'Content-Length': '0'} 2022-09-13T07:28:32.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d572ea60812a8cd3-0063203120-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.425 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.425 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.426 DEBUG:boto:path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:32.426 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:32.427 DEBUG:boto:Method: GET 2022-09-13T07:28:32.427 DEBUG:boto:Path: /yxvvuh-10/?prefix=%26 2022-09-13T07:28:32.428 DEBUG:boto:Data: 2022-09-13T07:28:32.428 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.429 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.429 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.430 DEBUG:boto:Params: {} 2022-09-13T07:28:32.431 DEBUG:boto:Token: None 2022-09-13T07:28:32.431 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/ 2022-09-13T07:28:32.432 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY= 2022-09-13T07:28:32.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY=', 'Content-Length': '0'} 2022-09-13T07:28:32.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eacd105d4eaa14dc-0063203120-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.435 DEBUG:boto:b'yxvvuh-10&1000false&2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:32.436 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.437 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.437 DEBUG:boto:path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:32.438 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:32.438 DEBUG:boto:Method: GET 2022-09-13T07:28:32.439 DEBUG:boto:Path: /yxvvuh-10/?prefix=%26 2022-09-13T07:28:32.439 DEBUG:boto:Data: 2022-09-13T07:28:32.440 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.440 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.441 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.441 DEBUG:boto:Params: {} 2022-09-13T07:28:32.442 DEBUG:boto:Token: None 2022-09-13T07:28:32.443 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/ 2022-09-13T07:28:32.443 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY= 2022-09-13T07:28:32.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY=', 'Content-Length': '0'} 2022-09-13T07:28:32.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ec5d39800774955-0063203120-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.447 DEBUG:boto:b'yxvvuh-10&1000false&2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:32.449 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:32.449 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:32.450 DEBUG:boto:Method: HEAD 2022-09-13T07:28:32.450 DEBUG:boto:Path: /yxvvuh-10/%26 2022-09-13T07:28:32.451 DEBUG:boto:Data: 2022-09-13T07:28:32.451 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.452 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.453 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.453 DEBUG:boto:Params: {} 2022-09-13T07:28:32.454 DEBUG:boto:Token: None 2022-09-13T07:28:32.455 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%26 2022-09-13T07:28:32.455 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7VV24kUaiA310+AYB+ii7ep9po0= 2022-09-13T07:28:32.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7VV24kUaiA310+AYB+ii7ep9po0=', 'Content-Length': '0'} 2022-09-13T07:28:32.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c63b13868814d433-0063203120-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.459 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:32.460 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:32.460 DEBUG:boto:Method: HEAD 2022-09-13T07:28:32.461 DEBUG:boto:Path: /yxvvuh-10/%26 2022-09-13T07:28:32.461 DEBUG:boto:Data: 2022-09-13T07:28:32.462 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.462 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.463 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.463 DEBUG:boto:Params: {} 2022-09-13T07:28:32.464 DEBUG:boto:Token: None 2022-09-13T07:28:32.464 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%26 2022-09-13T07:28:32.465 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7VV24kUaiA310+AYB+ii7ep9po0= 2022-09-13T07:28:32.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7VV24kUaiA310+AYB+ii7ep9po0=', 'Content-Length': '0'} 2022-09-13T07:28:32.468 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a54dd20c9b19443-0063203120-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:32.469 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:32.469 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:32.470 DEBUG:boto:Method: GET 2022-09-13T07:28:32.471 DEBUG:boto:Path: /yxvvuh-10/%26 2022-09-13T07:28:32.471 DEBUG:boto:Data: 2022-09-13T07:28:32.472 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.472 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.473 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.474 DEBUG:boto:Params: {} 2022-09-13T07:28:32.475 DEBUG:boto:Token: None 2022-09-13T07:28:32.475 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%26 2022-09-13T07:28:32.476 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O2hTIjD0VLV74t1m4Id5/VxvQLc= 2022-09-13T07:28:32.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O2hTIjD0VLV74t1m4Id5/VxvQLc=', 'Content-Length': '0'} 2022-09-13T07:28:32.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084d6ddcd599e580e-0063203120-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.522 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:32.523 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:32.524 DEBUG:boto:Method: GET 2022-09-13T07:28:32.524 DEBUG:boto:Path: /yxvvuh-10/%26 2022-09-13T07:28:32.525 DEBUG:boto:Data: 2022-09-13T07:28:32.526 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.526 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.527 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.527 DEBUG:boto:Params: {} 2022-09-13T07:28:32.528 DEBUG:boto:Token: None 2022-09-13T07:28:32.529 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%26 2022-09-13T07:28:32.529 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O2hTIjD0VLV74t1m4Id5/VxvQLc= 2022-09-13T07:28:32.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O2hTIjD0VLV74t1m4Id5/VxvQLc=', 'Content-Length': '0'} 2022-09-13T07:28:32.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3b0a988d7d4e21b-0063203120-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.534 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.535 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.535 DEBUG:boto:path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:32.536 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:32.536 DEBUG:boto:Method: GET 2022-09-13T07:28:32.537 DEBUG:boto:Path: /yxvvuh-10/?prefix=%26 2022-09-13T07:28:32.537 DEBUG:boto:Data: 2022-09-13T07:28:32.538 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.538 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.539 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.540 DEBUG:boto:Params: {} 2022-09-13T07:28:32.541 DEBUG:boto:Token: None 2022-09-13T07:28:32.541 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/ 2022-09-13T07:28:32.542 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY= 2022-09-13T07:28:32.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY=', 'Content-Length': '0'} 2022-09-13T07:28:32.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af524a7049e3d70b-0063203120-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.546 DEBUG:boto:b'yxvvuh-10&1000false&2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:32.548 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.548 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.549 DEBUG:boto:path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:32.550 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:32.551 DEBUG:boto:Method: GET 2022-09-13T07:28:32.551 DEBUG:boto:Path: /yxvvuh-10/?prefix=%26 2022-09-13T07:28:32.552 DEBUG:boto:Data: 2022-09-13T07:28:32.552 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.553 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.553 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.554 DEBUG:boto:Params: {} 2022-09-13T07:28:32.554 DEBUG:boto:Token: None 2022-09-13T07:28:32.555 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/ 2022-09-13T07:28:32.556 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY= 2022-09-13T07:28:32.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY=', 'Content-Length': '0'} 2022-09-13T07:28:32.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000906d5fd537aa1241-0063203120-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.560 DEBUG:boto:b'yxvvuh-10&1000false&2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:32.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:32.561 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:32.562 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:32.562 DEBUG:boto:path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.563 DEBUG:boto:auth_path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.563 DEBUG:boto:Method: GET 2022-09-13T07:28:32.564 DEBUG:boto:Path: /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.565 DEBUG:boto:Data: 2022-09-13T07:28:32.565 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.566 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.566 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.567 DEBUG:boto:Params: {} 2022-09-13T07:28:32.568 DEBUG:boto:Token: None 2022-09-13T07:28:32.569 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.569 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qLzfU3j3bWezY9vgyowcx3zX8rs= 2022-09-13T07:28:32.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qLzfU3j3bWezY9vgyowcx3zX8rs=', 'Content-Length': '0'} 2022-09-13T07:28:32.572 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b20133e591ea349d-0063203120-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.573 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:32.573 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:32.574 DEBUG:boto:path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.575 DEBUG:boto:auth_path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.575 DEBUG:boto:Method: GET 2022-09-13T07:28:32.576 DEBUG:boto:Path: /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.577 DEBUG:boto:Data: 2022-09-13T07:28:32.577 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.578 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.579 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.579 DEBUG:boto:Params: {} 2022-09-13T07:28:32.580 DEBUG:boto:Token: None 2022-09-13T07:28:32.581 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.582 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qLzfU3j3bWezY9vgyowcx3zX8rs= 2022-09-13T07:28:32.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qLzfU3j3bWezY9vgyowcx3zX8rs=', 'Content-Length': '0'} 2022-09-13T07:28:32.584 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a8b91ae97e713a7-0063203120-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.626 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:32.627 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:32.627 DEBUG:boto:path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.628 DEBUG:boto:auth_path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.628 DEBUG:boto:Method: HEAD 2022-09-13T07:28:32.629 DEBUG:boto:Path: /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.629 DEBUG:boto:Data: 2022-09-13T07:28:32.630 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.631 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.631 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.631 DEBUG:boto:Params: {} 2022-09-13T07:28:32.632 DEBUG:boto:Token: None 2022-09-13T07:28:32.633 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.633 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kaBQ3qxddZuCp4th1+2DSAwoWWg= 2022-09-13T07:28:32.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kaBQ3qxddZuCp4th1+2DSAwoWWg=', 'Content-Length': '0'} 2022-09-13T07:28:32.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc948386423f4b20-0063203120-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.637 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:32.637 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:32.638 DEBUG:boto:path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.639 DEBUG:boto:auth_path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.639 DEBUG:boto:Method: HEAD 2022-09-13T07:28:32.639 DEBUG:boto:Path: /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.640 DEBUG:boto:Data: 2022-09-13T07:28:32.640 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.641 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.641 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.642 DEBUG:boto:Params: {} 2022-09-13T07:28:32.643 DEBUG:boto:Token: None 2022-09-13T07:28:32.643 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.644 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kaBQ3qxddZuCp4th1+2DSAwoWWg= 2022-09-13T07:28:32.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kaBQ3qxddZuCp4th1+2DSAwoWWg=', 'Content-Length': '0'} 2022-09-13T07:28:32.646 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d441631dd0a26ae1-0063203120-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:32.647 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:32.648 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:32.648 DEBUG:boto:path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.649 DEBUG:boto:auth_path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.650 DEBUG:boto:Method: GET 2022-09-13T07:28:32.650 DEBUG:boto:Path: /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.651 DEBUG:boto:Data: 2022-09-13T07:28:32.651 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.652 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.652 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.653 DEBUG:boto:Params: {} 2022-09-13T07:28:32.653 DEBUG:boto:Token: None 2022-09-13T07:28:32.654 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.654 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qLzfU3j3bWezY9vgyowcx3zX8rs= 2022-09-13T07:28:32.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qLzfU3j3bWezY9vgyowcx3zX8rs=', 'Content-Length': '0'} 2022-09-13T07:28:32.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000835bd091f447527c-0063203120-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.658 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:32.658 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:32.659 DEBUG:boto:path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.659 DEBUG:boto:auth_path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.660 DEBUG:boto:Method: GET 2022-09-13T07:28:32.660 DEBUG:boto:Path: /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.661 DEBUG:boto:Data: 2022-09-13T07:28:32.661 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.662 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.662 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.663 DEBUG:boto:Params: {} 2022-09-13T07:28:32.663 DEBUG:boto:Token: None 2022-09-13T07:28:32.664 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:32.664 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qLzfU3j3bWezY9vgyowcx3zX8rs= 2022-09-13T07:28:32.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qLzfU3j3bWezY9vgyowcx3zX8rs=', 'Content-Length': '0'} 2022-09-13T07:28:32.667 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018214b75833d4add-0063203120-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.668 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.668 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.669 DEBUG:boto:path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:32.669 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:32.670 DEBUG:boto:Method: GET 2022-09-13T07:28:32.670 DEBUG:boto:Path: /yxvvuh-10/?prefix=%3A 2022-09-13T07:28:32.671 DEBUG:boto:Data: 2022-09-13T07:28:32.671 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.672 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.673 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.673 DEBUG:boto:Params: {} 2022-09-13T07:28:32.674 DEBUG:boto:Token: None 2022-09-13T07:28:32.674 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/ 2022-09-13T07:28:32.675 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY= 2022-09-13T07:28:32.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY=', 'Content-Length': '0'} 2022-09-13T07:28:32.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb03787a55aa4fed-0063203120-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.679 DEBUG:boto:b'yxvvuh-10:1000false:2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:32.679 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.680 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.681 DEBUG:boto:path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:32.681 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:32.682 DEBUG:boto:Method: GET 2022-09-13T07:28:32.682 DEBUG:boto:Path: /yxvvuh-10/?prefix=%3A 2022-09-13T07:28:32.683 DEBUG:boto:Data: 2022-09-13T07:28:32.683 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.684 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.684 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.685 DEBUG:boto:Params: {} 2022-09-13T07:28:32.686 DEBUG:boto:Token: None 2022-09-13T07:28:32.686 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/ 2022-09-13T07:28:32.687 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY= 2022-09-13T07:28:32.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY=', 'Content-Length': '0'} 2022-09-13T07:28:32.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c43ac6caccbab17-0063203120-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.691 DEBUG:boto:b'yxvvuh-10:1000false:2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:32.692 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:32.692 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:32.693 DEBUG:boto:Method: HEAD 2022-09-13T07:28:32.693 DEBUG:boto:Path: /yxvvuh-10/%3A 2022-09-13T07:28:32.694 DEBUG:boto:Data: 2022-09-13T07:28:32.694 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.695 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.695 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.696 DEBUG:boto:Params: {} 2022-09-13T07:28:32.697 DEBUG:boto:Token: None 2022-09-13T07:28:32.697 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%3A 2022-09-13T07:28:32.698 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YtGbQgMuJ4QlStqat3MTnUpBl+0= 2022-09-13T07:28:32.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YtGbQgMuJ4QlStqat3MTnUpBl+0=', 'Content-Length': '0'} 2022-09-13T07:28:32.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000754568e26abdf981-0063203120-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.701 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:32.702 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:32.702 DEBUG:boto:Method: HEAD 2022-09-13T07:28:32.703 DEBUG:boto:Path: /yxvvuh-10/%3A 2022-09-13T07:28:32.703 DEBUG:boto:Data: 2022-09-13T07:28:32.704 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.704 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.705 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.705 DEBUG:boto:Params: {} 2022-09-13T07:28:32.706 DEBUG:boto:Token: None 2022-09-13T07:28:32.706 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%3A 2022-09-13T07:28:32.707 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YtGbQgMuJ4QlStqat3MTnUpBl+0= 2022-09-13T07:28:32.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YtGbQgMuJ4QlStqat3MTnUpBl+0=', 'Content-Length': '0'} 2022-09-13T07:28:32.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5317a1d71cdd032-0063203120-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:32.711 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:32.712 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:32.712 DEBUG:boto:Method: GET 2022-09-13T07:28:32.713 DEBUG:boto:Path: /yxvvuh-10/%3A 2022-09-13T07:28:32.713 DEBUG:boto:Data: 2022-09-13T07:28:32.714 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.715 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.716 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.716 DEBUG:boto:Params: {} 2022-09-13T07:28:32.718 DEBUG:boto:Token: None 2022-09-13T07:28:32.719 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%3A 2022-09-13T07:28:32.719 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yX8WRVCCSde5jF/MBk7lf/lLVvc= 2022-09-13T07:28:32.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yX8WRVCCSde5jF/MBk7lf/lLVvc=', 'Content-Length': '0'} 2022-09-13T07:28:32.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a92e18abfdf5e071-0063203120-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.725 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:32.725 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:32.726 DEBUG:boto:Method: GET 2022-09-13T07:28:32.727 DEBUG:boto:Path: /yxvvuh-10/%3A 2022-09-13T07:28:32.728 DEBUG:boto:Data: 2022-09-13T07:28:32.729 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.730 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.731 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.732 DEBUG:boto:Params: {} 2022-09-13T07:28:32.733 DEBUG:boto:Token: None 2022-09-13T07:28:32.734 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/%3A 2022-09-13T07:28:32.735 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yX8WRVCCSde5jF/MBk7lf/lLVvc= 2022-09-13T07:28:32.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yX8WRVCCSde5jF/MBk7lf/lLVvc=', 'Content-Length': '0'} 2022-09-13T07:28:32.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c7b66d6614bdc5e-0063203120-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.786 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.787 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.788 DEBUG:boto:path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:32.789 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:32.790 DEBUG:boto:Method: GET 2022-09-13T07:28:32.791 DEBUG:boto:Path: /yxvvuh-10/?prefix=%3A 2022-09-13T07:28:32.792 DEBUG:boto:Data: 2022-09-13T07:28:32.793 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.793 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.794 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.794 DEBUG:boto:Params: {} 2022-09-13T07:28:32.795 DEBUG:boto:Token: None 2022-09-13T07:28:32.796 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/ 2022-09-13T07:28:32.796 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY= 2022-09-13T07:28:32.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY=', 'Content-Length': '0'} 2022-09-13T07:28:32.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088f418957f25f3bc-0063203120-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.800 DEBUG:boto:b'yxvvuh-10:1000false:2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:32.801 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.802 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.802 DEBUG:boto:path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:32.803 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:32.804 DEBUG:boto:Method: GET 2022-09-13T07:28:32.804 DEBUG:boto:Path: /yxvvuh-10/?prefix=%3A 2022-09-13T07:28:32.805 DEBUG:boto:Data: 2022-09-13T07:28:32.806 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.806 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.807 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.807 DEBUG:boto:Params: {} 2022-09-13T07:28:32.808 DEBUG:boto:Token: None 2022-09-13T07:28:32.809 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/ 2022-09-13T07:28:32.809 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY= 2022-09-13T07:28:32.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY=', 'Content-Length': '0'} 2022-09-13T07:28:32.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1f0ff4bc25073ae-0063203120-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.814 DEBUG:boto:b'yxvvuh-10:1000false:2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:32.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:32.815 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:32.816 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:32.817 DEBUG:boto:path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.818 DEBUG:boto:auth_path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.819 DEBUG:boto:Method: GET 2022-09-13T07:28:32.819 DEBUG:boto:Path: /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.820 DEBUG:boto:Data: 2022-09-13T07:28:32.821 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.822 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.823 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.824 DEBUG:boto:Params: {} 2022-09-13T07:28:32.824 DEBUG:boto:Token: None 2022-09-13T07:28:32.825 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.826 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Yg+oJKkLU7zaEoeCo+5DBgx2/w= 2022-09-13T07:28:32.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Yg+oJKkLU7zaEoeCo+5DBgx2/w=', 'Content-Length': '0'} 2022-09-13T07:28:32.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007798b2de35ad52bc-0063203120-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.875 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:32.877 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:32.878 DEBUG:boto:path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.879 DEBUG:boto:auth_path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.880 DEBUG:boto:Method: GET 2022-09-13T07:28:32.881 DEBUG:boto:Path: /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.882 DEBUG:boto:Data: 2022-09-13T07:28:32.883 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.884 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.885 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.886 DEBUG:boto:Params: {} 2022-09-13T07:28:32.887 DEBUG:boto:Token: None 2022-09-13T07:28:32.889 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.890 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Yg+oJKkLU7zaEoeCo+5DBgx2/w= 2022-09-13T07:28:32.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Yg+oJKkLU7zaEoeCo+5DBgx2/w=', 'Content-Length': '0'} 2022-09-13T07:28:32.893 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9ffc26fe470f479-0063203120-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.895 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:32.896 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:32.897 DEBUG:boto:path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.898 DEBUG:boto:auth_path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.898 DEBUG:boto:Method: HEAD 2022-09-13T07:28:32.900 DEBUG:boto:Path: /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.901 DEBUG:boto:Data: 2022-09-13T07:28:32.901 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.902 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.903 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.904 DEBUG:boto:Params: {} 2022-09-13T07:28:32.905 DEBUG:boto:Token: None 2022-09-13T07:28:32.907 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.908 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8V/cfWQHLON4NQ4yM7cy2gMexmM= 2022-09-13T07:28:32.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8V/cfWQHLON4NQ4yM7cy2gMexmM=', 'Content-Length': '0'} 2022-09-13T07:28:32.911 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cb2edd47b6e34aa-0063203120-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.912 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:32.913 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:32.914 DEBUG:boto:path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.915 DEBUG:boto:auth_path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.916 DEBUG:boto:Method: HEAD 2022-09-13T07:28:32.917 DEBUG:boto:Path: /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.918 DEBUG:boto:Data: 2022-09-13T07:28:32.919 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.920 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.921 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.921 DEBUG:boto:Params: {} 2022-09-13T07:28:32.922 DEBUG:boto:Token: None 2022-09-13T07:28:32.923 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.924 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8V/cfWQHLON4NQ4yM7cy2gMexmM= 2022-09-13T07:28:32.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8V/cfWQHLON4NQ4yM7cy2gMexmM=', 'Content-Length': '0'} 2022-09-13T07:28:32.926 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b7449e8d16fa41a-0063203120-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:32.928 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:32.928 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:32.929 DEBUG:boto:path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.929 DEBUG:boto:auth_path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.930 DEBUG:boto:Method: GET 2022-09-13T07:28:32.931 DEBUG:boto:Path: /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.931 DEBUG:boto:Data: 2022-09-13T07:28:32.932 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.932 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.933 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.933 DEBUG:boto:Params: {} 2022-09-13T07:28:32.934 DEBUG:boto:Token: None 2022-09-13T07:28:32.934 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.935 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Yg+oJKkLU7zaEoeCo+5DBgx2/w= 2022-09-13T07:28:32.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Yg+oJKkLU7zaEoeCo+5DBgx2/w=', 'Content-Length': '0'} 2022-09-13T07:28:32.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9a92d62bd23933f-0063203120-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.938 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:32.939 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:32.939 DEBUG:boto:path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.940 DEBUG:boto:auth_path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.941 DEBUG:boto:Method: GET 2022-09-13T07:28:32.941 DEBUG:boto:Path: /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.942 DEBUG:boto:Data: 2022-09-13T07:28:32.942 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.943 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.943 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.944 DEBUG:boto:Params: {} 2022-09-13T07:28:32.945 DEBUG:boto:Token: None 2022-09-13T07:28:32.946 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:32.947 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Yg+oJKkLU7zaEoeCo+5DBgx2/w= 2022-09-13T07:28:32.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Yg+oJKkLU7zaEoeCo+5DBgx2/w=', 'Content-Length': '0'} 2022-09-13T07:28:32.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019aa14457389c890-0063203120-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.952 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.953 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.954 DEBUG:boto:path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:32.955 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:32.956 DEBUG:boto:Method: GET 2022-09-13T07:28:32.958 DEBUG:boto:Path: /yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:32.958 DEBUG:boto:Data: 2022-09-13T07:28:32.959 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.960 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:32.960 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.961 DEBUG:boto:Params: {} 2022-09-13T07:28:32.961 DEBUG:boto:Token: None 2022-09-13T07:28:32.962 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/ 2022-09-13T07:28:32.963 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY= 2022-09-13T07:28:32.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY=', 'Content-Length': '0'} 2022-09-13T07:28:32.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7e5cb587f7e7f06-0063203120-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:32.970 DEBUG:boto:b'yxvvuh-10_myobj1000false_myobj2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:32.972 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:32.973 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:32.974 DEBUG:boto:path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:32.975 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:32.975 DEBUG:boto:Method: GET 2022-09-13T07:28:32.976 DEBUG:boto:Path: /yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:32.978 DEBUG:boto:Data: 2022-09-13T07:28:32.978 DEBUG:boto:Headers: {} 2022-09-13T07:28:32.979 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:32.980 DEBUG:boto:Port: 8000 2022-09-13T07:28:32.981 DEBUG:boto:Params: {} 2022-09-13T07:28:32.981 DEBUG:boto:Token: None 2022-09-13T07:28:32.982 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:32 GMT /yxvvuh-10/ 2022-09-13T07:28:32.984 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY= 2022-09-13T07:28:32.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fnL+IlPiJblW/wiXs4xwo+9JxPY=', 'Content-Length': '0'} 2022-09-13T07:28:32.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b179517961c57cda-0063203120-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.034 DEBUG:boto:b'yxvvuh-10_myobj1000false_myobj2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.036 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:33.037 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:33.037 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.038 DEBUG:boto:Path: /yxvvuh-10/_myobj 2022-09-13T07:28:33.039 DEBUG:boto:Data: 2022-09-13T07:28:33.040 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.041 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:33.042 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.043 DEBUG:boto:Params: {} 2022-09-13T07:28:33.044 DEBUG:boto:Token: None 2022-09-13T07:28:33.045 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/_myobj 2022-09-13T07:28:33.046 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mltXTfHM90bJZVZBqg/6FV7Bigg= 2022-09-13T07:28:33.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mltXTfHM90bJZVZBqg/6FV7Bigg=', 'Content-Length': '0'} 2022-09-13T07:28:33.050 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9ef4942d1ab9457-0063203121-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.051 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:33.051 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:33.052 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.052 DEBUG:boto:Path: /yxvvuh-10/_myobj 2022-09-13T07:28:33.053 DEBUG:boto:Data: 2022-09-13T07:28:33.054 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.055 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.056 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.056 DEBUG:boto:Params: {} 2022-09-13T07:28:33.057 DEBUG:boto:Token: None 2022-09-13T07:28:33.058 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/_myobj 2022-09-13T07:28:33.059 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mltXTfHM90bJZVZBqg/6FV7Bigg= 2022-09-13T07:28:33.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mltXTfHM90bJZVZBqg/6FV7Bigg=', 'Content-Length': '0'} 2022-09-13T07:28:33.062 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094f06e721606a744-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:33.065 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:33.066 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:33.067 DEBUG:boto:Method: GET 2022-09-13T07:28:33.067 DEBUG:boto:Path: /yxvvuh-10/_myobj 2022-09-13T07:28:33.068 DEBUG:boto:Data: 2022-09-13T07:28:33.068 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.069 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:33.069 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.070 DEBUG:boto:Params: {} 2022-09-13T07:28:33.070 DEBUG:boto:Token: None 2022-09-13T07:28:33.071 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/_myobj 2022-09-13T07:28:33.071 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BSFshonKog3xjTkmfvbSAungjQA= 2022-09-13T07:28:33.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BSFshonKog3xjTkmfvbSAungjQA=', 'Content-Length': '0'} 2022-09-13T07:28:33.074 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000479889d3c0a061ab-0063203121-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.075 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:33.076 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:33.076 DEBUG:boto:Method: GET 2022-09-13T07:28:33.077 DEBUG:boto:Path: /yxvvuh-10/_myobj 2022-09-13T07:28:33.077 DEBUG:boto:Data: 2022-09-13T07:28:33.078 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.078 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.079 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.079 DEBUG:boto:Params: {} 2022-09-13T07:28:33.080 DEBUG:boto:Token: None 2022-09-13T07:28:33.081 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/_myobj 2022-09-13T07:28:33.081 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BSFshonKog3xjTkmfvbSAungjQA= 2022-09-13T07:28:33.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BSFshonKog3xjTkmfvbSAungjQA=', 'Content-Length': '0'} 2022-09-13T07:28:33.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000574f5878d736de21-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.085 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.086 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.086 DEBUG:boto:path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:33.087 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:33.087 DEBUG:boto:Method: GET 2022-09-13T07:28:33.088 DEBUG:boto:Path: /yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:33.088 DEBUG:boto:Data: 2022-09-13T07:28:33.089 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.089 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:33.090 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.090 DEBUG:boto:Params: {} 2022-09-13T07:28:33.091 DEBUG:boto:Token: None 2022-09-13T07:28:33.092 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.092 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU= 2022-09-13T07:28:33.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU=', 'Content-Length': '0'} 2022-09-13T07:28:33.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e842aa294062d7b-0063203121-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.096 DEBUG:boto:b'yxvvuh-10_myobj1000false_myobj2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.097 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.098 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.098 DEBUG:boto:path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:33.099 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:33.099 DEBUG:boto:Method: GET 2022-09-13T07:28:33.100 DEBUG:boto:Path: /yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:33.100 DEBUG:boto:Data: 2022-09-13T07:28:33.101 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.101 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.102 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.102 DEBUG:boto:Params: {} 2022-09-13T07:28:33.103 DEBUG:boto:Token: None 2022-09-13T07:28:33.104 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.104 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU= 2022-09-13T07:28:33.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU=', 'Content-Length': '0'} 2022-09-13T07:28:33.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7e6feaf3d1be4ff-0063203121-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.108 DEBUG:boto:b'yxvvuh-10_myobj1000false_myobj2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:33.110 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:33.110 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:33.111 DEBUG:boto:path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.111 DEBUG:boto:auth_path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.112 DEBUG:boto:Method: GET 2022-09-13T07:28:33.113 DEBUG:boto:Path: /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.113 DEBUG:boto:Data: 2022-09-13T07:28:33.114 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.114 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:33.116 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.116 DEBUG:boto:Params: {} 2022-09-13T07:28:33.117 DEBUG:boto:Token: None 2022-09-13T07:28:33.118 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.119 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QWmMofnrFVg1vjFRvLRwVADwZrM= 2022-09-13T07:28:33.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QWmMofnrFVg1vjFRvLRwVADwZrM=', 'Content-Length': '0'} 2022-09-13T07:28:33.121 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065b5a901dc004f80-0063203121-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.162 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:33.163 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:33.163 DEBUG:boto:path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.164 DEBUG:boto:auth_path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.164 DEBUG:boto:Method: GET 2022-09-13T07:28:33.165 DEBUG:boto:Path: /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.165 DEBUG:boto:Data: 2022-09-13T07:28:33.166 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.166 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.167 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.167 DEBUG:boto:Params: {} 2022-09-13T07:28:33.168 DEBUG:boto:Token: None 2022-09-13T07:28:33.169 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.169 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QWmMofnrFVg1vjFRvLRwVADwZrM= 2022-09-13T07:28:33.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QWmMofnrFVg1vjFRvLRwVADwZrM=', 'Content-Length': '0'} 2022-09-13T07:28:33.172 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000276cf3b8265ab1-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.173 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:33.173 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:33.174 DEBUG:boto:path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.174 DEBUG:boto:auth_path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.175 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.175 DEBUG:boto:Path: /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.176 DEBUG:boto:Data: 2022-09-13T07:28:33.176 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.177 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:33.177 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.178 DEBUG:boto:Params: {} 2022-09-13T07:28:33.190 DEBUG:boto:Token: None 2022-09-13T07:28:33.191 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.191 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/dCGlXd4r/77XYeK4qfHvaQtcus= 2022-09-13T07:28:33.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/dCGlXd4r/77XYeK4qfHvaQtcus=', 'Content-Length': '0'} 2022-09-13T07:28:33.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008be79e9c772049c9-0063203121-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.195 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:33.195 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:33.196 DEBUG:boto:path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.196 DEBUG:boto:auth_path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.197 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.197 DEBUG:boto:Path: /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.197 DEBUG:boto:Data: 2022-09-13T07:28:33.198 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.198 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.199 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.199 DEBUG:boto:Params: {} 2022-09-13T07:28:33.200 DEBUG:boto:Token: None 2022-09-13T07:28:33.200 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.201 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/dCGlXd4r/77XYeK4qfHvaQtcus= 2022-09-13T07:28:33.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/dCGlXd4r/77XYeK4qfHvaQtcus=', 'Content-Length': '0'} 2022-09-13T07:28:33.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3f7d2a4b66fc66d-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:33.205 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:33.205 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:33.206 DEBUG:boto:path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.206 DEBUG:boto:auth_path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.207 DEBUG:boto:Method: GET 2022-09-13T07:28:33.207 DEBUG:boto:Path: /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.208 DEBUG:boto:Data: 2022-09-13T07:28:33.208 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.209 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:33.209 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.210 DEBUG:boto:Params: {} 2022-09-13T07:28:33.211 DEBUG:boto:Token: None 2022-09-13T07:28:33.211 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.212 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QWmMofnrFVg1vjFRvLRwVADwZrM= 2022-09-13T07:28:33.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QWmMofnrFVg1vjFRvLRwVADwZrM=', 'Content-Length': '0'} 2022-09-13T07:28:33.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b96d0470e2cbe70d-0063203121-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.215 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:33.216 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:33.216 DEBUG:boto:path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.217 DEBUG:boto:auth_path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.217 DEBUG:boto:Method: GET 2022-09-13T07:28:33.218 DEBUG:boto:Path: /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.219 DEBUG:boto:Data: 2022-09-13T07:28:33.219 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.220 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.220 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.221 DEBUG:boto:Params: {} 2022-09-13T07:28:33.221 DEBUG:boto:Token: None 2022-09-13T07:28:33.222 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:33.223 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QWmMofnrFVg1vjFRvLRwVADwZrM= 2022-09-13T07:28:33.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QWmMofnrFVg1vjFRvLRwVADwZrM=', 'Content-Length': '0'} 2022-09-13T07:28:33.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e2448f807d5045f-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.226 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.227 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.228 DEBUG:boto:path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:33.228 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:33.229 DEBUG:boto:Method: GET 2022-09-13T07:28:33.229 DEBUG:boto:Path: /yxvvuh-10/?prefix=myobj 2022-09-13T07:28:33.230 DEBUG:boto:Data: 2022-09-13T07:28:33.230 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.231 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:33.231 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.232 DEBUG:boto:Params: {} 2022-09-13T07:28:33.232 DEBUG:boto:Token: None 2022-09-13T07:28:33.233 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.234 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU= 2022-09-13T07:28:33.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU=', 'Content-Length': '0'} 2022-09-13T07:28:33.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009500f5778fe57f6c-0063203121-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.238 DEBUG:boto:b'yxvvuh-10myobj1000falsemyobj2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.239 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.239 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.240 DEBUG:boto:path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:33.240 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:33.241 DEBUG:boto:Method: GET 2022-09-13T07:28:33.241 DEBUG:boto:Path: /yxvvuh-10/?prefix=myobj 2022-09-13T07:28:33.242 DEBUG:boto:Data: 2022-09-13T07:28:33.243 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.243 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.244 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.244 DEBUG:boto:Params: {} 2022-09-13T07:28:33.245 DEBUG:boto:Token: None 2022-09-13T07:28:33.245 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.246 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU= 2022-09-13T07:28:33.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU=', 'Content-Length': '0'} 2022-09-13T07:28:33.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029185636d5277cb3-0063203121-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.250 DEBUG:boto:b'yxvvuh-10myobj1000falsemyobj2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.251 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:33.252 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:33.252 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.253 DEBUG:boto:Path: /yxvvuh-10/myobj 2022-09-13T07:28:33.253 DEBUG:boto:Data: 2022-09-13T07:28:33.254 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.254 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:33.255 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.255 DEBUG:boto:Params: {} 2022-09-13T07:28:33.256 DEBUG:boto:Token: None 2022-09-13T07:28:33.256 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/myobj 2022-09-13T07:28:33.257 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zOdmw7eOZ47OPE5J+eXjRYEfrXI= 2022-09-13T07:28:33.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zOdmw7eOZ47OPE5J+eXjRYEfrXI=', 'Content-Length': '0'} 2022-09-13T07:28:33.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0c248a1835665ae-0063203121-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.260 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:33.261 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:33.261 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.262 DEBUG:boto:Path: /yxvvuh-10/myobj 2022-09-13T07:28:33.262 DEBUG:boto:Data: 2022-09-13T07:28:33.263 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.263 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.264 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.264 DEBUG:boto:Params: {} 2022-09-13T07:28:33.265 DEBUG:boto:Token: None 2022-09-13T07:28:33.265 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/myobj 2022-09-13T07:28:33.266 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zOdmw7eOZ47OPE5J+eXjRYEfrXI= 2022-09-13T07:28:33.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zOdmw7eOZ47OPE5J+eXjRYEfrXI=', 'Content-Length': '0'} 2022-09-13T07:28:33.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000892f817b76fff875-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:33.270 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:33.270 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:33.271 DEBUG:boto:Method: GET 2022-09-13T07:28:33.272 DEBUG:boto:Path: /yxvvuh-10/myobj 2022-09-13T07:28:33.272 DEBUG:boto:Data: 2022-09-13T07:28:33.273 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.273 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:33.274 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.275 DEBUG:boto:Params: {} 2022-09-13T07:28:33.276 DEBUG:boto:Token: None 2022-09-13T07:28:33.276 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/myobj 2022-09-13T07:28:33.277 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F2iVRUgc51napI/xv9FLX+keTKY= 2022-09-13T07:28:33.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F2iVRUgc51napI/xv9FLX+keTKY=', 'Content-Length': '0'} 2022-09-13T07:28:33.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024824e863bd9ed7a-0063203121-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.281 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:33.281 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:33.282 DEBUG:boto:Method: GET 2022-09-13T07:28:33.282 DEBUG:boto:Path: /yxvvuh-10/myobj 2022-09-13T07:28:33.283 DEBUG:boto:Data: 2022-09-13T07:28:33.283 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.284 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.285 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.285 DEBUG:boto:Params: {} 2022-09-13T07:28:33.286 DEBUG:boto:Token: None 2022-09-13T07:28:33.286 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/myobj 2022-09-13T07:28:33.287 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F2iVRUgc51napI/xv9FLX+keTKY= 2022-09-13T07:28:33.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F2iVRUgc51napI/xv9FLX+keTKY=', 'Content-Length': '0'} 2022-09-13T07:28:33.290 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008007c1f57a80beaf-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.291 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.292 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.293 DEBUG:boto:path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:33.294 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:33.295 DEBUG:boto:Method: GET 2022-09-13T07:28:33.296 DEBUG:boto:Path: /yxvvuh-10/?prefix=myobj 2022-09-13T07:28:33.297 DEBUG:boto:Data: 2022-09-13T07:28:33.299 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.300 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:33.301 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.302 DEBUG:boto:Params: {} 2022-09-13T07:28:33.303 DEBUG:boto:Token: None 2022-09-13T07:28:33.304 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.305 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU= 2022-09-13T07:28:33.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU=', 'Content-Length': '0'} 2022-09-13T07:28:33.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c787392bdf5b2639-0063203121-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.311 DEBUG:boto:b'yxvvuh-10myobj1000falsemyobj2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.312 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.313 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.314 DEBUG:boto:path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:33.314 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:33.315 DEBUG:boto:Method: GET 2022-09-13T07:28:33.315 DEBUG:boto:Path: /yxvvuh-10/?prefix=myobj 2022-09-13T07:28:33.316 DEBUG:boto:Data: 2022-09-13T07:28:33.316 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.317 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.317 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.318 DEBUG:boto:Params: {} 2022-09-13T07:28:33.319 DEBUG:boto:Token: None 2022-09-13T07:28:33.319 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.320 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU= 2022-09-13T07:28:33.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU=', 'Content-Length': '0'} 2022-09-13T07:28:33.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036c298ca68708580-0063203121-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.324 DEBUG:boto:b'yxvvuh-10myobj1000falsemyobj2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.325 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-10 zones={test-zone2, test-zone1} 2022-09-13T07:28:33.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:28:33.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:28:33.326 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:28:33.391 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:33.392 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:33.430 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:33.427+0000 7f7abac11600 1 bucket sync caught up with source: 2022-09-13T07:28:33.431 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.7.5, , 00000000001.2.5, , , , 00000000001.8.5, 00000000001.7.5, , ] 2022-09-13T07:28:33.431 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000001.7.5, , 00000000001.2.5, , , , 00000000001.8.5, 00000000001.7.5, , ] 2022-09-13T07:28:33.431 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:33.427+0000 7f7abac11600 0 bucket checkpoint complete 2022-09-13T07:28:33.443 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-10 zones={test-zone3, test-zone1} 2022-09-13T07:28:33.444 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.444 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.445 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.445 DEBUG:boto:Path: /yxvvuh-10/ 2022-09-13T07:28:33.446 DEBUG:boto:Data: 2022-09-13T07:28:33.446 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.447 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.447 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.448 DEBUG:boto:Params: {} 2022-09-13T07:28:33.448 DEBUG:boto:Token: None 2022-09-13T07:28:33.449 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.449 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5kZaTJTs4mZniAqgHkgux2b8QPE= 2022-09-13T07:28:33.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5kZaTJTs4mZniAqgHkgux2b8QPE=', 'Content-Length': '0'} 2022-09-13T07:28:33.453 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc65855e089b0951-0063203121-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.454 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.455 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.455 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.456 DEBUG:boto:Path: /yxvvuh-10/ 2022-09-13T07:28:33.456 DEBUG:boto:Data: 2022-09-13T07:28:33.457 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.457 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.458 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.458 DEBUG:boto:Params: {} 2022-09-13T07:28:33.459 DEBUG:boto:Token: None 2022-09-13T07:28:33.460 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.460 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5kZaTJTs4mZniAqgHkgux2b8QPE= 2022-09-13T07:28:33.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5kZaTJTs4mZniAqgHkgux2b8QPE=', 'Content-Length': '0'} 2022-09-13T07:28:33.464 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c66556cf34ccd78-0063203121-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.464 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:28:33.465 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.465 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.466 DEBUG:boto:path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:33.466 DEBUG:boto:auth_path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:33.467 DEBUG:boto:Method: GET 2022-09-13T07:28:33.468 DEBUG:boto:Path: /yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:33.468 DEBUG:boto:Data: 2022-09-13T07:28:33.469 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.470 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.470 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.471 DEBUG:boto:Params: {} 2022-09-13T07:28:33.471 DEBUG:boto:Token: None 2022-09-13T07:28:33.472 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/?versions 2022-09-13T07:28:33.473 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:P/FTefqZuYmNUA19oGK6fCKRYrM= 2022-09-13T07:28:33.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:P/FTefqZuYmNUA19oGK6fCKRYrM=', 'Content-Length': '0'} 2022-09-13T07:28:33.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035cbec1d66c61381-0063203121-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.476 DEBUG:boto:b'yxvvuh-10999false&nulltrue2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.477 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-13T07:28:33.478 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-13T07:28:33.478 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-13T07:28:33.479 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:28:33.479 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:28:33.480 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.480 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.481 DEBUG:boto:path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:33.481 DEBUG:boto:auth_path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:33.482 DEBUG:boto:Method: GET 2022-09-13T07:28:33.482 DEBUG:boto:Path: /yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:33.483 DEBUG:boto:Data: 2022-09-13T07:28:33.483 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.484 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.484 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.485 DEBUG:boto:Params: {} 2022-09-13T07:28:33.485 DEBUG:boto:Token: None 2022-09-13T07:28:33.486 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/?versions 2022-09-13T07:28:33.486 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:P/FTefqZuYmNUA19oGK6fCKRYrM= 2022-09-13T07:28:33.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:P/FTefqZuYmNUA19oGK6fCKRYrM=', 'Content-Length': '0'} 2022-09-13T07:28:33.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ede4bfcad24cf72-0063203121-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.490 DEBUG:boto:b'yxvvuh-10999false&nulltrue2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.491 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-13T07:28:33.491 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-13T07:28:33.492 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-13T07:28:33.492 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:28:33.493 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.494 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.494 DEBUG:boto:path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:33.494 DEBUG:boto:auth_path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:33.495 DEBUG:boto:Method: GET 2022-09-13T07:28:33.495 DEBUG:boto:Path: /yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:33.496 DEBUG:boto:Data: 2022-09-13T07:28:33.496 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.497 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.498 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.498 DEBUG:boto:Params: {} 2022-09-13T07:28:33.499 DEBUG:boto:Token: None 2022-09-13T07:28:33.499 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/?versions 2022-09-13T07:28:33.500 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:P/FTefqZuYmNUA19oGK6fCKRYrM= 2022-09-13T07:28:33.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:P/FTefqZuYmNUA19oGK6fCKRYrM=', 'Content-Length': '0'} 2022-09-13T07:28:33.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb5ed8791234418a-0063203121-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.504 DEBUG:boto:b'yxvvuh-10999false&nulltrue2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.505 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.506 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.506 DEBUG:boto:path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:33.506 DEBUG:boto:auth_path=/yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:33.507 DEBUG:boto:Method: GET 2022-09-13T07:28:33.507 DEBUG:boto:Path: /yxvvuh-10/?versions&max-keys=999 2022-09-13T07:28:33.508 DEBUG:boto:Data: 2022-09-13T07:28:33.508 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.509 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.510 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.510 DEBUG:boto:Params: {} 2022-09-13T07:28:33.511 DEBUG:boto:Token: None 2022-09-13T07:28:33.511 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/?versions 2022-09-13T07:28:33.512 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:P/FTefqZuYmNUA19oGK6fCKRYrM= 2022-09-13T07:28:33.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:P/FTefqZuYmNUA19oGK6fCKRYrM=', 'Content-Length': '0'} 2022-09-13T07:28:33.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f64881bdcb5d89ae-0063203121-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.516 DEBUG:boto:b'yxvvuh-10999false&nulltrue2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:33.517 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:33.518 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:33.518 DEBUG:boto:path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.519 DEBUG:boto:auth_path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.519 DEBUG:boto:Method: GET 2022-09-13T07:28:33.520 DEBUG:boto:Path: /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.520 DEBUG:boto:Data: 2022-09-13T07:28:33.521 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.521 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.522 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.522 DEBUG:boto:Params: {} 2022-09-13T07:28:33.523 DEBUG:boto:Token: None 2022-09-13T07:28:33.524 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.524 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fBJvNEjO8iWD9ANGau7k222+xnQ= 2022-09-13T07:28:33.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fBJvNEjO8iWD9ANGau7k222+xnQ=', 'Content-Length': '0'} 2022-09-13T07:28:33.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b39fa4189d20be04-0063203121-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.528 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:33.528 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:33.529 DEBUG:boto:path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.529 DEBUG:boto:auth_path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.530 DEBUG:boto:Method: GET 2022-09-13T07:28:33.530 DEBUG:boto:Path: /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.531 DEBUG:boto:Data: 2022-09-13T07:28:33.531 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.532 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.532 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.533 DEBUG:boto:Params: {} 2022-09-13T07:28:33.533 DEBUG:boto:Token: None 2022-09-13T07:28:33.534 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.535 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fBJvNEjO8iWD9ANGau7k222+xnQ= 2022-09-13T07:28:33.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fBJvNEjO8iWD9ANGau7k222+xnQ=', 'Content-Length': '0'} 2022-09-13T07:28:33.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9071a8b7ea5c964-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.578 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:33.579 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:33.579 DEBUG:boto:path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.580 DEBUG:boto:auth_path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.580 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.581 DEBUG:boto:Path: /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.581 DEBUG:boto:Data: 2022-09-13T07:28:33.582 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.582 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.583 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.583 DEBUG:boto:Params: {} 2022-09-13T07:28:33.584 DEBUG:boto:Token: None 2022-09-13T07:28:33.586 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.587 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O+ffj+cR8pD9rus+sjP4oezlzLI= 2022-09-13T07:28:33.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O+ffj+cR8pD9rus+sjP4oezlzLI=', 'Content-Length': '0'} 2022-09-13T07:28:33.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009853e6c077579cef-0063203121-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.590 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:33.590 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:33.591 DEBUG:boto:path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.591 DEBUG:boto:auth_path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.592 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.592 DEBUG:boto:Path: /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.593 DEBUG:boto:Data: 2022-09-13T07:28:33.593 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.594 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.595 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.595 DEBUG:boto:Params: {} 2022-09-13T07:28:33.596 DEBUG:boto:Token: None 2022-09-13T07:28:33.596 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.597 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O+ffj+cR8pD9rus+sjP4oezlzLI= 2022-09-13T07:28:33.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O+ffj+cR8pD9rus+sjP4oezlzLI=', 'Content-Length': '0'} 2022-09-13T07:28:33.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040e43d7b0d70ffb6-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:33.601 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:33.601 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:33.602 DEBUG:boto:path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.603 DEBUG:boto:auth_path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.603 DEBUG:boto:Method: GET 2022-09-13T07:28:33.604 DEBUG:boto:Path: /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.605 DEBUG:boto:Data: 2022-09-13T07:28:33.605 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.606 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.606 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.607 DEBUG:boto:Params: {} 2022-09-13T07:28:33.607 DEBUG:boto:Token: None 2022-09-13T07:28:33.608 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.609 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fBJvNEjO8iWD9ANGau7k222+xnQ= 2022-09-13T07:28:33.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fBJvNEjO8iWD9ANGau7k222+xnQ=', 'Content-Length': '0'} 2022-09-13T07:28:33.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001daaf93310f1c378-0063203121-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.612 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:33.613 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:33.613 DEBUG:boto:path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.614 DEBUG:boto:auth_path=/yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.614 DEBUG:boto:Method: GET 2022-09-13T07:28:33.615 DEBUG:boto:Path: /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.615 DEBUG:boto:Data: 2022-09-13T07:28:33.616 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.616 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.617 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.617 DEBUG:boto:Params: {} 2022-09-13T07:28:33.618 DEBUG:boto:Token: None 2022-09-13T07:28:33.619 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%26?versionId=null 2022-09-13T07:28:33.619 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fBJvNEjO8iWD9ANGau7k222+xnQ= 2022-09-13T07:28:33.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fBJvNEjO8iWD9ANGau7k222+xnQ=', 'Content-Length': '0'} 2022-09-13T07:28:33.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd547cb2963ec633-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.623 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.624 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.625 DEBUG:boto:path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:33.625 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:33.626 DEBUG:boto:Method: GET 2022-09-13T07:28:33.626 DEBUG:boto:Path: /yxvvuh-10/?prefix=%26 2022-09-13T07:28:33.627 DEBUG:boto:Data: 2022-09-13T07:28:33.627 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.628 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.629 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.629 DEBUG:boto:Params: {} 2022-09-13T07:28:33.630 DEBUG:boto:Token: None 2022-09-13T07:28:33.630 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.631 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU= 2022-09-13T07:28:33.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU=', 'Content-Length': '0'} 2022-09-13T07:28:33.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000019afc7bc600717e-0063203121-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.636 DEBUG:boto:b'yxvvuh-10&1000false&2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.636 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.637 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.638 DEBUG:boto:path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:33.639 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:33.639 DEBUG:boto:Method: GET 2022-09-13T07:28:33.640 DEBUG:boto:Path: /yxvvuh-10/?prefix=%26 2022-09-13T07:28:33.640 DEBUG:boto:Data: 2022-09-13T07:28:33.641 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.641 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.642 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.642 DEBUG:boto:Params: {} 2022-09-13T07:28:33.643 DEBUG:boto:Token: None 2022-09-13T07:28:33.644 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.645 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU= 2022-09-13T07:28:33.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU=', 'Content-Length': '0'} 2022-09-13T07:28:33.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3a58f2d062325a6-0063203121-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.650 DEBUG:boto:b'yxvvuh-10&1000false&2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.651 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:33.652 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:33.653 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.654 DEBUG:boto:Path: /yxvvuh-10/%26 2022-09-13T07:28:33.654 DEBUG:boto:Data: 2022-09-13T07:28:33.655 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.655 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.656 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.657 DEBUG:boto:Params: {} 2022-09-13T07:28:33.657 DEBUG:boto:Token: None 2022-09-13T07:28:33.658 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%26 2022-09-13T07:28:33.658 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SWR0gFoXxOfV/FdaaImBtipgAEc= 2022-09-13T07:28:33.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SWR0gFoXxOfV/FdaaImBtipgAEc=', 'Content-Length': '0'} 2022-09-13T07:28:33.661 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc0ed28b3209b109-0063203121-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.661 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:33.662 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:33.663 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.663 DEBUG:boto:Path: /yxvvuh-10/%26 2022-09-13T07:28:33.664 DEBUG:boto:Data: 2022-09-13T07:28:33.664 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.665 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.665 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.666 DEBUG:boto:Params: {} 2022-09-13T07:28:33.667 DEBUG:boto:Token: None 2022-09-13T07:28:33.667 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%26 2022-09-13T07:28:33.668 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SWR0gFoXxOfV/FdaaImBtipgAEc= 2022-09-13T07:28:33.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SWR0gFoXxOfV/FdaaImBtipgAEc=', 'Content-Length': '0'} 2022-09-13T07:28:33.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000081346413285dbb8-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:33.672 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:33.673 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:33.673 DEBUG:boto:Method: GET 2022-09-13T07:28:33.674 DEBUG:boto:Path: /yxvvuh-10/%26 2022-09-13T07:28:33.674 DEBUG:boto:Data: 2022-09-13T07:28:33.675 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.675 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.676 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.676 DEBUG:boto:Params: {} 2022-09-13T07:28:33.677 DEBUG:boto:Token: None 2022-09-13T07:28:33.677 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%26 2022-09-13T07:28:33.678 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GVdzwlVzj7UOKyT6Qj3IoUxcSW8= 2022-09-13T07:28:33.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GVdzwlVzj7UOKyT6Qj3IoUxcSW8=', 'Content-Length': '0'} 2022-09-13T07:28:33.681 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007850301acfcd7c75-0063203121-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.681 DEBUG:boto:path=/yxvvuh-10/%26 2022-09-13T07:28:33.682 DEBUG:boto:auth_path=/yxvvuh-10/%26 2022-09-13T07:28:33.683 DEBUG:boto:Method: GET 2022-09-13T07:28:33.683 DEBUG:boto:Path: /yxvvuh-10/%26 2022-09-13T07:28:33.684 DEBUG:boto:Data: 2022-09-13T07:28:33.684 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.685 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.685 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.686 DEBUG:boto:Params: {} 2022-09-13T07:28:33.686 DEBUG:boto:Token: None 2022-09-13T07:28:33.687 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%26 2022-09-13T07:28:33.687 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GVdzwlVzj7UOKyT6Qj3IoUxcSW8= 2022-09-13T07:28:33.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GVdzwlVzj7UOKyT6Qj3IoUxcSW8=', 'Content-Length': '0'} 2022-09-13T07:28:33.690 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085395f255e47c3e3-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.734 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.735 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.735 DEBUG:boto:path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:33.736 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:33.736 DEBUG:boto:Method: GET 2022-09-13T07:28:33.737 DEBUG:boto:Path: /yxvvuh-10/?prefix=%26 2022-09-13T07:28:33.738 DEBUG:boto:Data: 2022-09-13T07:28:33.738 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.739 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.740 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.741 DEBUG:boto:Params: {} 2022-09-13T07:28:33.741 DEBUG:boto:Token: None 2022-09-13T07:28:33.742 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.743 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU= 2022-09-13T07:28:33.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU=', 'Content-Length': '0'} 2022-09-13T07:28:33.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006564ae5ad45465e7-0063203121-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.746 DEBUG:boto:b'yxvvuh-10&1000false&2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.747 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.748 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.749 DEBUG:boto:path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:33.749 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%26 2022-09-13T07:28:33.750 DEBUG:boto:Method: GET 2022-09-13T07:28:33.750 DEBUG:boto:Path: /yxvvuh-10/?prefix=%26 2022-09-13T07:28:33.751 DEBUG:boto:Data: 2022-09-13T07:28:33.751 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.752 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.753 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.753 DEBUG:boto:Params: {} 2022-09-13T07:28:33.754 DEBUG:boto:Token: None 2022-09-13T07:28:33.754 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.755 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU= 2022-09-13T07:28:33.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU=', 'Content-Length': '0'} 2022-09-13T07:28:33.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005867d271aeb48c3b-0063203121-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.760 DEBUG:boto:b'yxvvuh-10&1000false&2022-09-13T07:28:26.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:33.761 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:33.762 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:33.762 DEBUG:boto:path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.763 DEBUG:boto:auth_path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.763 DEBUG:boto:Method: GET 2022-09-13T07:28:33.764 DEBUG:boto:Path: /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.764 DEBUG:boto:Data: 2022-09-13T07:28:33.765 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.766 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.766 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.767 DEBUG:boto:Params: {} 2022-09-13T07:28:33.768 DEBUG:boto:Token: None 2022-09-13T07:28:33.768 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.769 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IsT1/rQ0c1lRJfVJ1D32AyT6vv4= 2022-09-13T07:28:33.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IsT1/rQ0c1lRJfVJ1D32AyT6vv4=', 'Content-Length': '0'} 2022-09-13T07:28:33.772 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054802b17031a96eb-0063203121-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.772 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:33.773 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:33.774 DEBUG:boto:path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.775 DEBUG:boto:auth_path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.775 DEBUG:boto:Method: GET 2022-09-13T07:28:33.776 DEBUG:boto:Path: /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.776 DEBUG:boto:Data: 2022-09-13T07:28:33.777 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.778 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.778 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.779 DEBUG:boto:Params: {} 2022-09-13T07:28:33.779 DEBUG:boto:Token: None 2022-09-13T07:28:33.780 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.780 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IsT1/rQ0c1lRJfVJ1D32AyT6vv4= 2022-09-13T07:28:33.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IsT1/rQ0c1lRJfVJ1D32AyT6vv4=', 'Content-Length': '0'} 2022-09-13T07:28:33.783 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4d1a86d2dcdd0e1-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.784 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:33.785 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:33.785 DEBUG:boto:path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.786 DEBUG:boto:auth_path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.786 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.787 DEBUG:boto:Path: /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.787 DEBUG:boto:Data: 2022-09-13T07:28:33.788 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.788 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.789 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.789 DEBUG:boto:Params: {} 2022-09-13T07:28:33.790 DEBUG:boto:Token: None 2022-09-13T07:28:33.791 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.791 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bVH0B/dG3+DwTDW6wk0fFtJPTQQ= 2022-09-13T07:28:33.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bVH0B/dG3+DwTDW6wk0fFtJPTQQ=', 'Content-Length': '0'} 2022-09-13T07:28:33.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c78ba786251dec6-0063203121-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.795 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:33.795 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:33.796 DEBUG:boto:path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.796 DEBUG:boto:auth_path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.797 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.797 DEBUG:boto:Path: /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.798 DEBUG:boto:Data: 2022-09-13T07:28:33.798 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.799 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.799 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.800 DEBUG:boto:Params: {} 2022-09-13T07:28:33.801 DEBUG:boto:Token: None 2022-09-13T07:28:33.801 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.802 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bVH0B/dG3+DwTDW6wk0fFtJPTQQ= 2022-09-13T07:28:33.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bVH0B/dG3+DwTDW6wk0fFtJPTQQ=', 'Content-Length': '0'} 2022-09-13T07:28:33.804 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d0548ac37b7b731-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:33.806 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:33.806 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:33.807 DEBUG:boto:path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.807 DEBUG:boto:auth_path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.808 DEBUG:boto:Method: GET 2022-09-13T07:28:33.809 DEBUG:boto:Path: /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.809 DEBUG:boto:Data: 2022-09-13T07:28:33.810 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.811 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.811 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.812 DEBUG:boto:Params: {} 2022-09-13T07:28:33.813 DEBUG:boto:Token: None 2022-09-13T07:28:33.813 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.814 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IsT1/rQ0c1lRJfVJ1D32AyT6vv4= 2022-09-13T07:28:33.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IsT1/rQ0c1lRJfVJ1D32AyT6vv4=', 'Content-Length': '0'} 2022-09-13T07:28:33.816 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000116a65c67ad6d5b-0063203121-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.862 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:33.863 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:33.863 DEBUG:boto:path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.864 DEBUG:boto:auth_path=/yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.864 DEBUG:boto:Method: GET 2022-09-13T07:28:33.865 DEBUG:boto:Path: /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.865 DEBUG:boto:Data: 2022-09-13T07:28:33.866 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.866 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.867 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.867 DEBUG:boto:Params: {} 2022-09-13T07:28:33.868 DEBUG:boto:Token: None 2022-09-13T07:28:33.869 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%3A?versionId=null 2022-09-13T07:28:33.869 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IsT1/rQ0c1lRJfVJ1D32AyT6vv4= 2022-09-13T07:28:33.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IsT1/rQ0c1lRJfVJ1D32AyT6vv4=', 'Content-Length': '0'} 2022-09-13T07:28:33.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df57840dd7e2cba0-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.873 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.874 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.874 DEBUG:boto:path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:33.875 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:33.875 DEBUG:boto:Method: GET 2022-09-13T07:28:33.876 DEBUG:boto:Path: /yxvvuh-10/?prefix=%3A 2022-09-13T07:28:33.876 DEBUG:boto:Data: 2022-09-13T07:28:33.877 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.878 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.878 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.879 DEBUG:boto:Params: {} 2022-09-13T07:28:33.879 DEBUG:boto:Token: None 2022-09-13T07:28:33.880 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.881 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU= 2022-09-13T07:28:33.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU=', 'Content-Length': '0'} 2022-09-13T07:28:33.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6d776bafc0cbb61-0063203121-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.885 DEBUG:boto:b'yxvvuh-10:1000false:2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.885 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.886 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.887 DEBUG:boto:path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:33.887 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:33.888 DEBUG:boto:Method: GET 2022-09-13T07:28:33.888 DEBUG:boto:Path: /yxvvuh-10/?prefix=%3A 2022-09-13T07:28:33.889 DEBUG:boto:Data: 2022-09-13T07:28:33.889 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.890 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.891 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.891 DEBUG:boto:Params: {} 2022-09-13T07:28:33.892 DEBUG:boto:Token: None 2022-09-13T07:28:33.892 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.893 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU= 2022-09-13T07:28:33.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU=', 'Content-Length': '0'} 2022-09-13T07:28:33.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa4416afaf4e71b8-0063203121-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.897 DEBUG:boto:b'yxvvuh-10:1000false:2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.898 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:33.899 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:33.899 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.900 DEBUG:boto:Path: /yxvvuh-10/%3A 2022-09-13T07:28:33.900 DEBUG:boto:Data: 2022-09-13T07:28:33.901 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.901 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.902 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.902 DEBUG:boto:Params: {} 2022-09-13T07:28:33.903 DEBUG:boto:Token: None 2022-09-13T07:28:33.904 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%3A 2022-09-13T07:28:33.904 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:d5fXbm6heI2NT8i4kITcQe8l5tA= 2022-09-13T07:28:33.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:d5fXbm6heI2NT8i4kITcQe8l5tA=', 'Content-Length': '0'} 2022-09-13T07:28:33.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c56e64180f2491f0-0063203121-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.908 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:33.909 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:33.909 DEBUG:boto:Method: HEAD 2022-09-13T07:28:33.910 DEBUG:boto:Path: /yxvvuh-10/%3A 2022-09-13T07:28:33.910 DEBUG:boto:Data: 2022-09-13T07:28:33.911 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.912 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.912 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.913 DEBUG:boto:Params: {} 2022-09-13T07:28:33.914 DEBUG:boto:Token: None 2022-09-13T07:28:33.914 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%3A 2022-09-13T07:28:33.915 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:d5fXbm6heI2NT8i4kITcQe8l5tA= 2022-09-13T07:28:33.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:d5fXbm6heI2NT8i4kITcQe8l5tA=', 'Content-Length': '0'} 2022-09-13T07:28:33.918 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020c281f780bbdcf1-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:33.919 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:33.919 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:33.920 DEBUG:boto:Method: GET 2022-09-13T07:28:33.921 DEBUG:boto:Path: /yxvvuh-10/%3A 2022-09-13T07:28:33.921 DEBUG:boto:Data: 2022-09-13T07:28:33.922 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.923 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.923 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.924 DEBUG:boto:Params: {} 2022-09-13T07:28:33.924 DEBUG:boto:Token: None 2022-09-13T07:28:33.925 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%3A 2022-09-13T07:28:33.925 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KMC2k0KnyFrJu1t7OHGK0lOXJBY= 2022-09-13T07:28:33.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KMC2k0KnyFrJu1t7OHGK0lOXJBY=', 'Content-Length': '0'} 2022-09-13T07:28:33.928 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e311e1c5c9baa5b-0063203121-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.929 DEBUG:boto:path=/yxvvuh-10/%3A 2022-09-13T07:28:33.930 DEBUG:boto:auth_path=/yxvvuh-10/%3A 2022-09-13T07:28:33.930 DEBUG:boto:Method: GET 2022-09-13T07:28:33.931 DEBUG:boto:Path: /yxvvuh-10/%3A 2022-09-13T07:28:33.932 DEBUG:boto:Data: 2022-09-13T07:28:33.932 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.933 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.933 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.934 DEBUG:boto:Params: {} 2022-09-13T07:28:33.935 DEBUG:boto:Token: None 2022-09-13T07:28:33.935 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/%3A 2022-09-13T07:28:33.936 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KMC2k0KnyFrJu1t7OHGK0lOXJBY= 2022-09-13T07:28:33.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KMC2k0KnyFrJu1t7OHGK0lOXJBY=', 'Content-Length': '0'} 2022-09-13T07:28:33.938 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086dafafc09bdd6b6-0063203121-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.939 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.939 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.940 DEBUG:boto:path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:33.941 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:33.941 DEBUG:boto:Method: GET 2022-09-13T07:28:33.942 DEBUG:boto:Path: /yxvvuh-10/?prefix=%3A 2022-09-13T07:28:33.942 DEBUG:boto:Data: 2022-09-13T07:28:33.943 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.943 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:33.944 DEBUG:boto:Port: 8001 2022-09-13T07:28:33.945 DEBUG:boto:Params: {} 2022-09-13T07:28:33.945 DEBUG:boto:Token: None 2022-09-13T07:28:33.946 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.947 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU= 2022-09-13T07:28:33.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU=', 'Content-Length': '0'} 2022-09-13T07:28:33.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000096ba32acd9c3518-0063203121-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:33.951 DEBUG:boto:b'yxvvuh-10:1000false:2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:33.952 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:33.952 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:33.953 DEBUG:boto:path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:33.953 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=%3A 2022-09-13T07:28:33.954 DEBUG:boto:Method: GET 2022-09-13T07:28:33.955 DEBUG:boto:Path: /yxvvuh-10/?prefix=%3A 2022-09-13T07:28:33.956 DEBUG:boto:Data: 2022-09-13T07:28:33.956 DEBUG:boto:Headers: {} 2022-09-13T07:28:33.957 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:33.957 DEBUG:boto:Port: 8000 2022-09-13T07:28:33.958 DEBUG:boto:Params: {} 2022-09-13T07:28:33.958 DEBUG:boto:Token: None 2022-09-13T07:28:33.959 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:33 GMT /yxvvuh-10/ 2022-09-13T07:28:33.959 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU= 2022-09-13T07:28:33.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:imakbSnSEv+rl+VgYYzstTOjWrU=', 'Content-Length': '0'} 2022-09-13T07:28:33.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f5d704761787395-0063203121-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.006 DEBUG:boto:b'yxvvuh-10:1000false:2022-09-13T07:28:26.220Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:34.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:34.008 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.008 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.009 DEBUG:boto:path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.010 DEBUG:boto:auth_path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.010 DEBUG:boto:Method: GET 2022-09-13T07:28:34.011 DEBUG:boto:Path: /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.011 DEBUG:boto:Data: 2022-09-13T07:28:34.012 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.014 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:34.015 DEBUG:boto:Port: 8001 2022-09-13T07:28:34.015 DEBUG:boto:Params: {} 2022-09-13T07:28:34.016 DEBUG:boto:Token: None 2022-09-13T07:28:34.017 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.017 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i/I/cps6bs8Me/mDK1nX0XRiz0g= 2022-09-13T07:28:34.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i/I/cps6bs8Me/mDK1nX0XRiz0g=', 'Content-Length': '0'} 2022-09-13T07:28:34.024 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044b7e9edf9dfb5e1-0063203122-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.025 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.026 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.027 DEBUG:boto:path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.027 DEBUG:boto:auth_path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.028 DEBUG:boto:Method: GET 2022-09-13T07:28:34.029 DEBUG:boto:Path: /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.030 DEBUG:boto:Data: 2022-09-13T07:28:34.031 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.032 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:34.032 DEBUG:boto:Port: 8000 2022-09-13T07:28:34.033 DEBUG:boto:Params: {} 2022-09-13T07:28:34.033 DEBUG:boto:Token: None 2022-09-13T07:28:34.034 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.035 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i/I/cps6bs8Me/mDK1nX0XRiz0g= 2022-09-13T07:28:34.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i/I/cps6bs8Me/mDK1nX0XRiz0g=', 'Content-Length': '0'} 2022-09-13T07:28:34.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007782e26205af5928-0063203122-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.038 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.039 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.039 DEBUG:boto:path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.040 DEBUG:boto:auth_path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.040 DEBUG:boto:Method: HEAD 2022-09-13T07:28:34.041 DEBUG:boto:Path: /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.041 DEBUG:boto:Data: 2022-09-13T07:28:34.042 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.042 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:34.043 DEBUG:boto:Port: 8001 2022-09-13T07:28:34.043 DEBUG:boto:Params: {} 2022-09-13T07:28:34.044 DEBUG:boto:Token: None 2022-09-13T07:28:34.045 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.045 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:JOQwJ954SHgn1DTBbW22VtYLoow= 2022-09-13T07:28:34.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JOQwJ954SHgn1DTBbW22VtYLoow=', 'Content-Length': '0'} 2022-09-13T07:28:34.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b7150892af96b1c-0063203122-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.050 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.050 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.051 DEBUG:boto:path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.052 DEBUG:boto:auth_path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.052 DEBUG:boto:Method: HEAD 2022-09-13T07:28:34.053 DEBUG:boto:Path: /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.054 DEBUG:boto:Data: 2022-09-13T07:28:34.054 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.055 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:34.055 DEBUG:boto:Port: 8000 2022-09-13T07:28:34.055 DEBUG:boto:Params: {} 2022-09-13T07:28:34.056 DEBUG:boto:Token: None 2022-09-13T07:28:34.057 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.058 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:JOQwJ954SHgn1DTBbW22VtYLoow= 2022-09-13T07:28:34.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JOQwJ954SHgn1DTBbW22VtYLoow=', 'Content-Length': '0'} 2022-09-13T07:28:34.060 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000458fa4301c8b53d2-0063203122-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:34.062 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.063 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.063 DEBUG:boto:path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.064 DEBUG:boto:auth_path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.065 DEBUG:boto:Method: GET 2022-09-13T07:28:34.066 DEBUG:boto:Path: /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.066 DEBUG:boto:Data: 2022-09-13T07:28:34.067 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.068 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:34.068 DEBUG:boto:Port: 8001 2022-09-13T07:28:34.069 DEBUG:boto:Params: {} 2022-09-13T07:28:34.069 DEBUG:boto:Token: None 2022-09-13T07:28:34.070 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.071 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i/I/cps6bs8Me/mDK1nX0XRiz0g= 2022-09-13T07:28:34.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i/I/cps6bs8Me/mDK1nX0XRiz0g=', 'Content-Length': '0'} 2022-09-13T07:28:34.075 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b66dda9a72a10879-0063203122-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.076 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.077 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.077 DEBUG:boto:path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.078 DEBUG:boto:auth_path=/yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.078 DEBUG:boto:Method: GET 2022-09-13T07:28:34.079 DEBUG:boto:Path: /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.079 DEBUG:boto:Data: 2022-09-13T07:28:34.080 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.080 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:34.081 DEBUG:boto:Port: 8000 2022-09-13T07:28:34.082 DEBUG:boto:Params: {} 2022-09-13T07:28:34.082 DEBUG:boto:Token: None 2022-09-13T07:28:34.083 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/_myobj?versionId=null 2022-09-13T07:28:34.084 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i/I/cps6bs8Me/mDK1nX0XRiz0g= 2022-09-13T07:28:34.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i/I/cps6bs8Me/mDK1nX0XRiz0g=', 'Content-Length': '0'} 2022-09-13T07:28:34.087 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f7c7e1a0e85ce2c-0063203122-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.089 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:34.090 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:34.090 DEBUG:boto:path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:34.091 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:34.091 DEBUG:boto:Method: GET 2022-09-13T07:28:34.092 DEBUG:boto:Path: /yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:34.092 DEBUG:boto:Data: 2022-09-13T07:28:34.093 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.093 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:34.094 DEBUG:boto:Port: 8001 2022-09-13T07:28:34.094 DEBUG:boto:Params: {} 2022-09-13T07:28:34.095 DEBUG:boto:Token: None 2022-09-13T07:28:34.096 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/ 2022-09-13T07:28:34.096 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ= 2022-09-13T07:28:34.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ=', 'Content-Length': '0'} 2022-09-13T07:28:34.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b61ad2e9ddb17d9f-0063203122-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.100 DEBUG:boto:b'yxvvuh-10_myobj1000false_myobj2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:34.101 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:34.101 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:34.102 DEBUG:boto:path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:34.102 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:34.103 DEBUG:boto:Method: GET 2022-09-13T07:28:34.103 DEBUG:boto:Path: /yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:34.104 DEBUG:boto:Data: 2022-09-13T07:28:34.105 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.105 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:34.106 DEBUG:boto:Port: 8000 2022-09-13T07:28:34.106 DEBUG:boto:Params: {} 2022-09-13T07:28:34.107 DEBUG:boto:Token: None 2022-09-13T07:28:34.108 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/ 2022-09-13T07:28:34.108 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ= 2022-09-13T07:28:34.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ=', 'Content-Length': '0'} 2022-09-13T07:28:34.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8f1025be9e32132-0063203122-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.113 DEBUG:boto:b'yxvvuh-10_myobj1000false_myobj2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:34.114 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.115 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.116 DEBUG:boto:Method: HEAD 2022-09-13T07:28:34.116 DEBUG:boto:Path: /yxvvuh-10/_myobj 2022-09-13T07:28:34.117 DEBUG:boto:Data: 2022-09-13T07:28:34.117 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.118 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:34.119 DEBUG:boto:Port: 8001 2022-09-13T07:28:34.120 DEBUG:boto:Params: {} 2022-09-13T07:28:34.121 DEBUG:boto:Token: None 2022-09-13T07:28:34.123 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/_myobj 2022-09-13T07:28:34.124 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fgIMZGA1nLlrIbT8EcHmidwr4w8= 2022-09-13T07:28:34.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fgIMZGA1nLlrIbT8EcHmidwr4w8=', 'Content-Length': '0'} 2022-09-13T07:28:34.127 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f04b5d7406bf363-0063203122-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.128 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.129 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.129 DEBUG:boto:Method: HEAD 2022-09-13T07:28:34.130 DEBUG:boto:Path: /yxvvuh-10/_myobj 2022-09-13T07:28:34.130 DEBUG:boto:Data: 2022-09-13T07:28:34.131 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.131 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:34.132 DEBUG:boto:Port: 8000 2022-09-13T07:28:34.132 DEBUG:boto:Params: {} 2022-09-13T07:28:34.133 DEBUG:boto:Token: None 2022-09-13T07:28:34.133 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/_myobj 2022-09-13T07:28:34.134 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fgIMZGA1nLlrIbT8EcHmidwr4w8= 2022-09-13T07:28:34.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fgIMZGA1nLlrIbT8EcHmidwr4w8=', 'Content-Length': '0'} 2022-09-13T07:28:34.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcebf21499182c2b-0063203122-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:34.138 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.138 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.139 DEBUG:boto:Method: GET 2022-09-13T07:28:34.139 DEBUG:boto:Path: /yxvvuh-10/_myobj 2022-09-13T07:28:34.140 DEBUG:boto:Data: 2022-09-13T07:28:34.140 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.141 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:34.142 DEBUG:boto:Port: 8001 2022-09-13T07:28:34.142 DEBUG:boto:Params: {} 2022-09-13T07:28:34.143 DEBUG:boto:Token: None 2022-09-13T07:28:34.143 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/_myobj 2022-09-13T07:28:34.144 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kWY4hgZbu0TTUZBUIhwpFQro1mc= 2022-09-13T07:28:34.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kWY4hgZbu0TTUZBUIhwpFQro1mc=', 'Content-Length': '0'} 2022-09-13T07:28:34.147 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e07db0133577363-0063203122-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.148 DEBUG:boto:path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.148 DEBUG:boto:auth_path=/yxvvuh-10/_myobj 2022-09-13T07:28:34.149 DEBUG:boto:Method: GET 2022-09-13T07:28:34.149 DEBUG:boto:Path: /yxvvuh-10/_myobj 2022-09-13T07:28:34.150 DEBUG:boto:Data: 2022-09-13T07:28:34.150 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.151 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:34.151 DEBUG:boto:Port: 8000 2022-09-13T07:28:34.152 DEBUG:boto:Params: {} 2022-09-13T07:28:34.152 DEBUG:boto:Token: None 2022-09-13T07:28:34.153 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/_myobj 2022-09-13T07:28:34.154 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kWY4hgZbu0TTUZBUIhwpFQro1mc= 2022-09-13T07:28:34.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kWY4hgZbu0TTUZBUIhwpFQro1mc=', 'Content-Length': '0'} 2022-09-13T07:28:34.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a89478a2b4b4ce0-0063203122-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.158 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:34.158 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:34.159 DEBUG:boto:path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:34.160 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:34.160 DEBUG:boto:Method: GET 2022-09-13T07:28:34.161 DEBUG:boto:Path: /yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:34.161 DEBUG:boto:Data: 2022-09-13T07:28:34.162 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.162 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:34.163 DEBUG:boto:Port: 8001 2022-09-13T07:28:34.163 DEBUG:boto:Params: {} 2022-09-13T07:28:34.164 DEBUG:boto:Token: None 2022-09-13T07:28:34.165 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/ 2022-09-13T07:28:34.167 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ= 2022-09-13T07:28:34.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ=', 'Content-Length': '0'} 2022-09-13T07:28:34.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035d003ea8022dbc7-0063203122-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.173 DEBUG:boto:b'yxvvuh-10_myobj1000false_myobj2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:34.175 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:34.175 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:34.176 DEBUG:boto:path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:34.177 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:34.178 DEBUG:boto:Method: GET 2022-09-13T07:28:34.179 DEBUG:boto:Path: /yxvvuh-10/?prefix=_myobj 2022-09-13T07:28:34.180 DEBUG:boto:Data: 2022-09-13T07:28:34.181 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.182 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:34.182 DEBUG:boto:Port: 8000 2022-09-13T07:28:34.183 DEBUG:boto:Params: {} 2022-09-13T07:28:34.184 DEBUG:boto:Token: None 2022-09-13T07:28:34.186 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/ 2022-09-13T07:28:34.187 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ= 2022-09-13T07:28:34.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ=', 'Content-Length': '0'} 2022-09-13T07:28:34.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f051a9aac1715512-0063203122-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.192 DEBUG:boto:b'yxvvuh-10_myobj1000false_myobj2022-09-13T07:28:26.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:34.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:34.194 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:34.195 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:34.196 DEBUG:boto:path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.197 DEBUG:boto:auth_path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.198 DEBUG:boto:Method: GET 2022-09-13T07:28:34.199 DEBUG:boto:Path: /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.199 DEBUG:boto:Data: 2022-09-13T07:28:34.200 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.201 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:34.202 DEBUG:boto:Port: 8001 2022-09-13T07:28:34.202 DEBUG:boto:Params: {} 2022-09-13T07:28:34.203 DEBUG:boto:Token: None 2022-09-13T07:28:34.204 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.206 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tB1Bfho6WD+uw7KpLDfdmJ9jdzM= 2022-09-13T07:28:34.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tB1Bfho6WD+uw7KpLDfdmJ9jdzM=', 'Content-Length': '0'} 2022-09-13T07:28:34.210 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abdbd622ac224af1-0063203122-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.250 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:34.251 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:34.252 DEBUG:boto:path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.253 DEBUG:boto:auth_path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.254 DEBUG:boto:Method: GET 2022-09-13T07:28:34.255 DEBUG:boto:Path: /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.256 DEBUG:boto:Data: 2022-09-13T07:28:34.257 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.257 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:34.258 DEBUG:boto:Port: 8000 2022-09-13T07:28:34.259 DEBUG:boto:Params: {} 2022-09-13T07:28:34.260 DEBUG:boto:Token: None 2022-09-13T07:28:34.261 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.262 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tB1Bfho6WD+uw7KpLDfdmJ9jdzM= 2022-09-13T07:28:34.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tB1Bfho6WD+uw7KpLDfdmJ9jdzM=', 'Content-Length': '0'} 2022-09-13T07:28:34.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc0fe9aca130332a-0063203122-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.266 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:34.266 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:34.267 DEBUG:boto:path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.267 DEBUG:boto:auth_path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.268 DEBUG:boto:Method: HEAD 2022-09-13T07:28:34.268 DEBUG:boto:Path: /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.269 DEBUG:boto:Data: 2022-09-13T07:28:34.269 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.270 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:34.270 DEBUG:boto:Port: 8001 2022-09-13T07:28:34.271 DEBUG:boto:Params: {} 2022-09-13T07:28:34.272 DEBUG:boto:Token: None 2022-09-13T07:28:34.272 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.273 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0AJIVDQ+LzqYIrCp7bjJ3MDxzjA= 2022-09-13T07:28:34.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0AJIVDQ+LzqYIrCp7bjJ3MDxzjA=', 'Content-Length': '0'} 2022-09-13T07:28:34.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044bf922ce72e5379-0063203122-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.277 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:34.277 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:34.278 DEBUG:boto:path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.278 DEBUG:boto:auth_path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.279 DEBUG:boto:Method: HEAD 2022-09-13T07:28:34.280 DEBUG:boto:Path: /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.281 DEBUG:boto:Data: 2022-09-13T07:28:34.282 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.283 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:34.283 DEBUG:boto:Port: 8000 2022-09-13T07:28:34.284 DEBUG:boto:Params: {} 2022-09-13T07:28:34.285 DEBUG:boto:Token: None 2022-09-13T07:28:34.286 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.287 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0AJIVDQ+LzqYIrCp7bjJ3MDxzjA= 2022-09-13T07:28:34.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0AJIVDQ+LzqYIrCp7bjJ3MDxzjA=', 'Content-Length': '0'} 2022-09-13T07:28:34.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020081984e766d1f3-0063203122-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:34.297 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:34.297 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:34.298 DEBUG:boto:path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.298 DEBUG:boto:auth_path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.299 DEBUG:boto:Method: GET 2022-09-13T07:28:34.300 DEBUG:boto:Path: /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.300 DEBUG:boto:Data: 2022-09-13T07:28:34.301 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.301 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:34.302 DEBUG:boto:Port: 8001 2022-09-13T07:28:34.303 DEBUG:boto:Params: {} 2022-09-13T07:28:34.304 DEBUG:boto:Token: None 2022-09-13T07:28:34.305 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.305 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tB1Bfho6WD+uw7KpLDfdmJ9jdzM= 2022-09-13T07:28:34.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tB1Bfho6WD+uw7KpLDfdmJ9jdzM=', 'Content-Length': '0'} 2022-09-13T07:28:34.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa70d2548040c5f1-0063203122-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.309 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:34.309 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:34.310 DEBUG:boto:path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.310 DEBUG:boto:auth_path=/yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.311 DEBUG:boto:Method: GET 2022-09-13T07:28:34.311 DEBUG:boto:Path: /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.312 DEBUG:boto:Data: 2022-09-13T07:28:34.312 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.313 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:34.313 DEBUG:boto:Port: 8000 2022-09-13T07:28:34.314 DEBUG:boto:Params: {} 2022-09-13T07:28:34.315 DEBUG:boto:Token: None 2022-09-13T07:28:34.315 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/myobj?versionId=null 2022-09-13T07:28:34.316 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tB1Bfho6WD+uw7KpLDfdmJ9jdzM= 2022-09-13T07:28:34.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tB1Bfho6WD+uw7KpLDfdmJ9jdzM=', 'Content-Length': '0'} 2022-09-13T07:28:34.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f37ec2a4404b784-0063203122-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.319 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:34.320 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:34.321 DEBUG:boto:path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:34.321 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:34.322 DEBUG:boto:Method: GET 2022-09-13T07:28:34.323 DEBUG:boto:Path: /yxvvuh-10/?prefix=myobj 2022-09-13T07:28:34.324 DEBUG:boto:Data: 2022-09-13T07:28:34.325 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.326 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:34.327 DEBUG:boto:Port: 8001 2022-09-13T07:28:34.327 DEBUG:boto:Params: {} 2022-09-13T07:28:34.328 DEBUG:boto:Token: None 2022-09-13T07:28:34.329 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/ 2022-09-13T07:28:34.330 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ= 2022-09-13T07:28:34.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ=', 'Content-Length': '0'} 2022-09-13T07:28:34.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eeba231441cfaa5-0063203122-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.335 DEBUG:boto:b'yxvvuh-10myobj1000falsemyobj2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:34.337 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:34.337 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:34.338 DEBUG:boto:path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:34.338 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:34.339 DEBUG:boto:Method: GET 2022-09-13T07:28:34.340 DEBUG:boto:Path: /yxvvuh-10/?prefix=myobj 2022-09-13T07:28:34.341 DEBUG:boto:Data: 2022-09-13T07:28:34.342 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.343 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:34.344 DEBUG:boto:Port: 8000 2022-09-13T07:28:34.344 DEBUG:boto:Params: {} 2022-09-13T07:28:34.345 DEBUG:boto:Token: None 2022-09-13T07:28:34.346 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/ 2022-09-13T07:28:34.347 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ= 2022-09-13T07:28:34.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ=', 'Content-Length': '0'} 2022-09-13T07:28:34.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017262da13a632e94-0063203122-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.394 DEBUG:boto:b'yxvvuh-10myobj1000falsemyobj2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:34.395 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:34.395 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:34.396 DEBUG:boto:Method: HEAD 2022-09-13T07:28:34.397 DEBUG:boto:Path: /yxvvuh-10/myobj 2022-09-13T07:28:34.397 DEBUG:boto:Data: 2022-09-13T07:28:34.398 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.398 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:34.399 DEBUG:boto:Port: 8001 2022-09-13T07:28:34.399 DEBUG:boto:Params: {} 2022-09-13T07:28:34.400 DEBUG:boto:Token: None 2022-09-13T07:28:34.400 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/myobj 2022-09-13T07:28:34.401 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YMfUafZyWeP2GRVVVQXJm38KhqY= 2022-09-13T07:28:34.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YMfUafZyWeP2GRVVVQXJm38KhqY=', 'Content-Length': '0'} 2022-09-13T07:28:34.404 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e3c437d13f62209-0063203122-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.404 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:34.405 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:34.405 DEBUG:boto:Method: HEAD 2022-09-13T07:28:34.406 DEBUG:boto:Path: /yxvvuh-10/myobj 2022-09-13T07:28:34.406 DEBUG:boto:Data: 2022-09-13T07:28:34.407 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.408 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:34.408 DEBUG:boto:Port: 8000 2022-09-13T07:28:34.409 DEBUG:boto:Params: {} 2022-09-13T07:28:34.409 DEBUG:boto:Token: None 2022-09-13T07:28:34.410 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/myobj 2022-09-13T07:28:34.411 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YMfUafZyWeP2GRVVVQXJm38KhqY= 2022-09-13T07:28:34.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YMfUafZyWeP2GRVVVQXJm38KhqY=', 'Content-Length': '0'} 2022-09-13T07:28:34.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9737ebb7540043c-0063203122-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:34.415 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:34.416 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:34.416 DEBUG:boto:Method: GET 2022-09-13T07:28:34.417 DEBUG:boto:Path: /yxvvuh-10/myobj 2022-09-13T07:28:34.417 DEBUG:boto:Data: 2022-09-13T07:28:34.418 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.418 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:34.419 DEBUG:boto:Port: 8001 2022-09-13T07:28:34.419 DEBUG:boto:Params: {} 2022-09-13T07:28:34.420 DEBUG:boto:Token: None 2022-09-13T07:28:34.421 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/myobj 2022-09-13T07:28:34.422 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x4uu+7VOKXEaxSA7579N+USf40M= 2022-09-13T07:28:34.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x4uu+7VOKXEaxSA7579N+USf40M=', 'Content-Length': '0'} 2022-09-13T07:28:34.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015d6fdf15418545c-0063203122-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.427 DEBUG:boto:path=/yxvvuh-10/myobj 2022-09-13T07:28:34.427 DEBUG:boto:auth_path=/yxvvuh-10/myobj 2022-09-13T07:28:34.428 DEBUG:boto:Method: GET 2022-09-13T07:28:34.428 DEBUG:boto:Path: /yxvvuh-10/myobj 2022-09-13T07:28:34.429 DEBUG:boto:Data: 2022-09-13T07:28:34.429 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.430 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:34.430 DEBUG:boto:Port: 8000 2022-09-13T07:28:34.431 DEBUG:boto:Params: {} 2022-09-13T07:28:34.431 DEBUG:boto:Token: None 2022-09-13T07:28:34.432 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/myobj 2022-09-13T07:28:34.433 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x4uu+7VOKXEaxSA7579N+USf40M= 2022-09-13T07:28:34.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x4uu+7VOKXEaxSA7579N+USf40M=', 'Content-Length': '0'} 2022-09-13T07:28:34.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6c6ae65e0af1218-0063203122-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.436 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:34.437 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:34.437 DEBUG:boto:path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:34.438 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:34.439 DEBUG:boto:Method: GET 2022-09-13T07:28:34.439 DEBUG:boto:Path: /yxvvuh-10/?prefix=myobj 2022-09-13T07:28:34.440 DEBUG:boto:Data: 2022-09-13T07:28:34.440 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.441 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:34.441 DEBUG:boto:Port: 8001 2022-09-13T07:28:34.442 DEBUG:boto:Params: {} 2022-09-13T07:28:34.442 DEBUG:boto:Token: None 2022-09-13T07:28:34.443 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/ 2022-09-13T07:28:34.443 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ= 2022-09-13T07:28:34.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ=', 'Content-Length': '0'} 2022-09-13T07:28:34.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9431b8ee483f626-0063203122-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.447 DEBUG:boto:b'yxvvuh-10myobj1000falsemyobj2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:34.448 DEBUG:boto:path=/yxvvuh-10/ 2022-09-13T07:28:34.449 DEBUG:boto:auth_path=/yxvvuh-10/ 2022-09-13T07:28:34.449 DEBUG:boto:path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:34.450 DEBUG:boto:auth_path=/yxvvuh-10/?prefix=myobj 2022-09-13T07:28:34.450 DEBUG:boto:Method: GET 2022-09-13T07:28:34.451 DEBUG:boto:Path: /yxvvuh-10/?prefix=myobj 2022-09-13T07:28:34.452 DEBUG:boto:Data: 2022-09-13T07:28:34.452 DEBUG:boto:Headers: {} 2022-09-13T07:28:34.453 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:34.453 DEBUG:boto:Port: 8000 2022-09-13T07:28:34.454 DEBUG:boto:Params: {} 2022-09-13T07:28:34.454 DEBUG:boto:Token: None 2022-09-13T07:28:34.455 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:34 GMT /yxvvuh-10/ 2022-09-13T07:28:34.456 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ= 2022-09-13T07:28:34.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pkuctvTVi1dqdckDdoiNcZ49RGQ=', 'Content-Length': '0'} 2022-09-13T07:28:34.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ce0ce0222689c8f-0063203122-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:34.461 DEBUG:boto:b'yxvvuh-10myobj1000falsemyobj2022-09-13T07:28:26.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:34.462 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-10 zones={test-zone3, test-zone1} 2022-09-13T07:28:34.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:28:34.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:28:34.465 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:28:34.528 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:34.528 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:34.576 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:34.572+0000 7fc879ddb600 1 waiting to reach incremental sync.. 2022-09-13T07:28:39.577 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:39.573+0000 7fc879ddb600 1 waiting to reach incremental sync.. 2022-09-13T07:28:44.577 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:44.573+0000 7fc879ddb600 1 waiting to reach incremental sync.. 2022-09-13T07:28:49.579 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:49.575+0000 7fc879ddb600 1 bucket sync caught up with source: 2022-09-13T07:28:49.579 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.18.5, , 00000000001.7.5, , , , 00000000001.5.5, 00000000001.16.5, , ] 2022-09-13T07:28:49.580 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000001.18.5, , 00000000001.7.5, , , , 00000000001.5.5, 00000000001.16.5, , ] 2022-09-13T07:28:49.580 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:49.575+0000 7fc879ddb600 0 bucket checkpoint complete 2022-09-13T07:28:49.596 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-11 zones={test-zone1, test-zone2} 2022-09-13T07:28:49.596 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:49.597 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:49.597 DEBUG:boto:Method: HEAD 2022-09-13T07:28:49.598 DEBUG:boto:Path: /yxvvuh-11/ 2022-09-13T07:28:49.598 DEBUG:boto:Data: 2022-09-13T07:28:49.599 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.599 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:49.600 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.600 DEBUG:boto:Params: {} 2022-09-13T07:28:49.601 DEBUG:boto:Token: None 2022-09-13T07:28:49.602 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/ 2022-09-13T07:28:49.602 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZuAjBxNL+qzn76JN/q7uS7zBxQU= 2022-09-13T07:28:49.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZuAjBxNL+qzn76JN/q7uS7zBxQU=', 'Content-Length': '0'} 2022-09-13T07:28:49.606 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f20e1f74b2317f98-0063203131-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.607 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:49.607 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:49.608 DEBUG:boto:Method: HEAD 2022-09-13T07:28:49.608 DEBUG:boto:Path: /yxvvuh-11/ 2022-09-13T07:28:49.609 DEBUG:boto:Data: 2022-09-13T07:28:49.609 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.610 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:49.610 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.611 DEBUG:boto:Params: {} 2022-09-13T07:28:49.611 DEBUG:boto:Token: None 2022-09-13T07:28:49.612 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/ 2022-09-13T07:28:49.613 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZuAjBxNL+qzn76JN/q7uS7zBxQU= 2022-09-13T07:28:49.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZuAjBxNL+qzn76JN/q7uS7zBxQU=', 'Content-Length': '0'} 2022-09-13T07:28:49.616 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c0577b325d63540-0063203131-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.617 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:28:49.617 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:49.618 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:49.618 DEBUG:boto:path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:49.619 DEBUG:boto:auth_path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:49.619 DEBUG:boto:Method: GET 2022-09-13T07:28:49.620 DEBUG:boto:Path: /yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:49.620 DEBUG:boto:Data: 2022-09-13T07:28:49.621 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.621 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:49.622 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.622 DEBUG:boto:Params: {} 2022-09-13T07:28:49.623 DEBUG:boto:Token: None 2022-09-13T07:28:49.624 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/?versions 2022-09-13T07:28:49.624 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:m5X0cLxIV79HG1jFPv8nrd+H09A= 2022-09-13T07:28:49.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:m5X0cLxIV79HG1jFPv8nrd+H09A=', 'Content-Length': '0'} 2022-09-13T07:28:49.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041c9ef6659e1af8f-0063203131-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.628 DEBUG:boto:b'yxvvuh-11999false&nulltrue2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:49.629 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-13T07:28:49.629 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-13T07:28:49.630 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-13T07:28:49.630 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:28:49.631 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:28:49.632 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:49.632 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:49.633 DEBUG:boto:path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:49.634 DEBUG:boto:auth_path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:49.634 DEBUG:boto:Method: GET 2022-09-13T07:28:49.635 DEBUG:boto:Path: /yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:49.636 DEBUG:boto:Data: 2022-09-13T07:28:49.636 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.637 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:49.637 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.638 DEBUG:boto:Params: {} 2022-09-13T07:28:49.638 DEBUG:boto:Token: None 2022-09-13T07:28:49.639 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/?versions 2022-09-13T07:28:49.640 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:m5X0cLxIV79HG1jFPv8nrd+H09A= 2022-09-13T07:28:49.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:m5X0cLxIV79HG1jFPv8nrd+H09A=', 'Content-Length': '0'} 2022-09-13T07:28:49.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4dc945f47b2dae0-0063203131-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.644 DEBUG:boto:b'yxvvuh-11999false&nulltrue2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:49.645 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-13T07:28:49.645 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-13T07:28:49.646 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-13T07:28:49.647 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:28:49.648 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:49.649 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:49.650 DEBUG:boto:path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:49.650 DEBUG:boto:auth_path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:49.651 DEBUG:boto:Method: GET 2022-09-13T07:28:49.652 DEBUG:boto:Path: /yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:49.652 DEBUG:boto:Data: 2022-09-13T07:28:49.653 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.653 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:49.654 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.654 DEBUG:boto:Params: {} 2022-09-13T07:28:49.655 DEBUG:boto:Token: None 2022-09-13T07:28:49.656 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/?versions 2022-09-13T07:28:49.657 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:m5X0cLxIV79HG1jFPv8nrd+H09A= 2022-09-13T07:28:49.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:m5X0cLxIV79HG1jFPv8nrd+H09A=', 'Content-Length': '0'} 2022-09-13T07:28:49.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a2d58a42578de16-0063203131-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.661 DEBUG:boto:b'yxvvuh-11999false&nulltrue2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:49.662 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:49.662 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:49.663 DEBUG:boto:path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:49.663 DEBUG:boto:auth_path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:49.664 DEBUG:boto:Method: GET 2022-09-13T07:28:49.664 DEBUG:boto:Path: /yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:49.665 DEBUG:boto:Data: 2022-09-13T07:28:49.665 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.666 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:49.666 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.667 DEBUG:boto:Params: {} 2022-09-13T07:28:49.667 DEBUG:boto:Token: None 2022-09-13T07:28:49.668 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/?versions 2022-09-13T07:28:49.668 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:m5X0cLxIV79HG1jFPv8nrd+H09A= 2022-09-13T07:28:49.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:m5X0cLxIV79HG1jFPv8nrd+H09A=', 'Content-Length': '0'} 2022-09-13T07:28:49.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0c344c2a4a4bdef-0063203131-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.672 DEBUG:boto:b'yxvvuh-11999false&nulltrue2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:49.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:49.674 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:49.675 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:49.675 DEBUG:boto:path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.676 DEBUG:boto:auth_path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.676 DEBUG:boto:Method: GET 2022-09-13T07:28:49.677 DEBUG:boto:Path: /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.677 DEBUG:boto:Data: 2022-09-13T07:28:49.678 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.678 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:49.679 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.679 DEBUG:boto:Params: {} 2022-09-13T07:28:49.680 DEBUG:boto:Token: None 2022-09-13T07:28:49.680 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.681 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kaRLRL8EpfbrlNabUAy7b52DYAM= 2022-09-13T07:28:49.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kaRLRL8EpfbrlNabUAy7b52DYAM=', 'Content-Length': '0'} 2022-09-13T07:28:49.684 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011f9fe789f7b8c6b-0063203131-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.730 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:49.731 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:49.732 DEBUG:boto:path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.732 DEBUG:boto:auth_path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.733 DEBUG:boto:Method: GET 2022-09-13T07:28:49.734 DEBUG:boto:Path: /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.734 DEBUG:boto:Data: 2022-09-13T07:28:49.735 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.735 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:49.736 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.737 DEBUG:boto:Params: {} 2022-09-13T07:28:49.738 DEBUG:boto:Token: None 2022-09-13T07:28:49.738 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.739 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kaRLRL8EpfbrlNabUAy7b52DYAM= 2022-09-13T07:28:49.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kaRLRL8EpfbrlNabUAy7b52DYAM=', 'Content-Length': '0'} 2022-09-13T07:28:49.742 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029a2951bbb4cda73-0063203131-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.744 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:49.744 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:49.745 DEBUG:boto:path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.746 DEBUG:boto:auth_path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.747 DEBUG:boto:Method: HEAD 2022-09-13T07:28:49.747 DEBUG:boto:Path: /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.748 DEBUG:boto:Data: 2022-09-13T07:28:49.749 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.750 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:49.750 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.751 DEBUG:boto:Params: {} 2022-09-13T07:28:49.752 DEBUG:boto:Token: None 2022-09-13T07:28:49.752 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.753 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TBzSzZ3TuLdS0ZstXxBJZVY45pk= 2022-09-13T07:28:49.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TBzSzZ3TuLdS0ZstXxBJZVY45pk=', 'Content-Length': '0'} 2022-09-13T07:28:49.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091925fcc7b57b779-0063203131-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.757 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:49.758 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:49.758 DEBUG:boto:path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.759 DEBUG:boto:auth_path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.759 DEBUG:boto:Method: HEAD 2022-09-13T07:28:49.760 DEBUG:boto:Path: /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.760 DEBUG:boto:Data: 2022-09-13T07:28:49.761 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.761 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:49.762 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.763 DEBUG:boto:Params: {} 2022-09-13T07:28:49.763 DEBUG:boto:Token: None 2022-09-13T07:28:49.764 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.764 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TBzSzZ3TuLdS0ZstXxBJZVY45pk= 2022-09-13T07:28:49.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TBzSzZ3TuLdS0ZstXxBJZVY45pk=', 'Content-Length': '0'} 2022-09-13T07:28:49.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000627952440c2c40e8-0063203131-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:49.768 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:49.769 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:49.770 DEBUG:boto:path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.771 DEBUG:boto:auth_path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.771 DEBUG:boto:Method: GET 2022-09-13T07:28:49.772 DEBUG:boto:Path: /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.772 DEBUG:boto:Data: 2022-09-13T07:28:49.773 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.774 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:49.774 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.775 DEBUG:boto:Params: {} 2022-09-13T07:28:49.776 DEBUG:boto:Token: None 2022-09-13T07:28:49.776 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.777 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kaRLRL8EpfbrlNabUAy7b52DYAM= 2022-09-13T07:28:49.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kaRLRL8EpfbrlNabUAy7b52DYAM=', 'Content-Length': '0'} 2022-09-13T07:28:49.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae712344d1c34111-0063203131-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.781 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:49.781 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:49.782 DEBUG:boto:path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.782 DEBUG:boto:auth_path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.783 DEBUG:boto:Method: GET 2022-09-13T07:28:49.783 DEBUG:boto:Path: /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.784 DEBUG:boto:Data: 2022-09-13T07:28:49.784 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.785 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:49.785 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.786 DEBUG:boto:Params: {} 2022-09-13T07:28:49.787 DEBUG:boto:Token: None 2022-09-13T07:28:49.787 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:49.788 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kaRLRL8EpfbrlNabUAy7b52DYAM= 2022-09-13T07:28:49.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kaRLRL8EpfbrlNabUAy7b52DYAM=', 'Content-Length': '0'} 2022-09-13T07:28:49.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092c74d71ea401b0d-0063203131-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.795 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:49.795 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:49.796 DEBUG:boto:path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:49.796 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:49.797 DEBUG:boto:Method: GET 2022-09-13T07:28:49.797 DEBUG:boto:Path: /yxvvuh-11/?prefix=%26 2022-09-13T07:28:49.798 DEBUG:boto:Data: 2022-09-13T07:28:49.800 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.800 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:49.801 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.801 DEBUG:boto:Params: {} 2022-09-13T07:28:49.802 DEBUG:boto:Token: None 2022-09-13T07:28:49.803 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/ 2022-09-13T07:28:49.803 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gYmFG0CAxA88+digsR+FobHJlKw= 2022-09-13T07:28:49.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gYmFG0CAxA88+digsR+FobHJlKw=', 'Content-Length': '0'} 2022-09-13T07:28:49.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a97d67dbaa6812fd-0063203131-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.807 DEBUG:boto:b'yxvvuh-11&1000false&2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:49.808 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:49.809 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:49.809 DEBUG:boto:path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:49.810 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:49.810 DEBUG:boto:Method: GET 2022-09-13T07:28:49.811 DEBUG:boto:Path: /yxvvuh-11/?prefix=%26 2022-09-13T07:28:49.811 DEBUG:boto:Data: 2022-09-13T07:28:49.812 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.812 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:49.813 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.814 DEBUG:boto:Params: {} 2022-09-13T07:28:49.814 DEBUG:boto:Token: None 2022-09-13T07:28:49.815 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/ 2022-09-13T07:28:49.817 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gYmFG0CAxA88+digsR+FobHJlKw= 2022-09-13T07:28:49.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gYmFG0CAxA88+digsR+FobHJlKw=', 'Content-Length': '0'} 2022-09-13T07:28:49.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000859902136a4ce8b7-0063203131-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.821 DEBUG:boto:b'yxvvuh-11&1000false&2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:49.823 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:49.823 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:49.824 DEBUG:boto:Method: HEAD 2022-09-13T07:28:49.825 DEBUG:boto:Path: /yxvvuh-11/%26 2022-09-13T07:28:49.826 DEBUG:boto:Data: 2022-09-13T07:28:49.826 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.827 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:49.828 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.829 DEBUG:boto:Params: {} 2022-09-13T07:28:49.830 DEBUG:boto:Token: None 2022-09-13T07:28:49.832 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/%26 2022-09-13T07:28:49.833 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZEnvDxdc97gBECgRYL9dk3UpQgA= 2022-09-13T07:28:49.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZEnvDxdc97gBECgRYL9dk3UpQgA=', 'Content-Length': '0'} 2022-09-13T07:28:49.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bb1741f2fb9c8ca-0063203131-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.837 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:49.838 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:49.838 DEBUG:boto:Method: HEAD 2022-09-13T07:28:49.839 DEBUG:boto:Path: /yxvvuh-11/%26 2022-09-13T07:28:49.839 DEBUG:boto:Data: 2022-09-13T07:28:49.840 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.840 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:49.841 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.842 DEBUG:boto:Params: {} 2022-09-13T07:28:49.842 DEBUG:boto:Token: None 2022-09-13T07:28:49.843 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/%26 2022-09-13T07:28:49.843 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZEnvDxdc97gBECgRYL9dk3UpQgA= 2022-09-13T07:28:49.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZEnvDxdc97gBECgRYL9dk3UpQgA=', 'Content-Length': '0'} 2022-09-13T07:28:49.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8c5e6084dedf4bd-0063203131-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:49.848 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:49.848 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:49.849 DEBUG:boto:Method: GET 2022-09-13T07:28:49.849 DEBUG:boto:Path: /yxvvuh-11/%26 2022-09-13T07:28:49.850 DEBUG:boto:Data: 2022-09-13T07:28:49.851 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.851 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:49.852 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.852 DEBUG:boto:Params: {} 2022-09-13T07:28:49.853 DEBUG:boto:Token: None 2022-09-13T07:28:49.853 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/%26 2022-09-13T07:28:49.854 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cwUMkicAoRlefpeEjPmU32wuOAY= 2022-09-13T07:28:49.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cwUMkicAoRlefpeEjPmU32wuOAY=', 'Content-Length': '0'} 2022-09-13T07:28:49.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004865bf047e29d5a8-0063203131-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.858 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:49.858 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:49.859 DEBUG:boto:Method: GET 2022-09-13T07:28:49.859 DEBUG:boto:Path: /yxvvuh-11/%26 2022-09-13T07:28:49.860 DEBUG:boto:Data: 2022-09-13T07:28:49.860 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.861 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:49.861 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.862 DEBUG:boto:Params: {} 2022-09-13T07:28:49.863 DEBUG:boto:Token: None 2022-09-13T07:28:49.863 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/%26 2022-09-13T07:28:49.864 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cwUMkicAoRlefpeEjPmU32wuOAY= 2022-09-13T07:28:49.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cwUMkicAoRlefpeEjPmU32wuOAY=', 'Content-Length': '0'} 2022-09-13T07:28:49.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061e4b8b3d6a2eb0a-0063203131-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.914 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:49.915 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:49.915 DEBUG:boto:path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:49.916 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:49.917 DEBUG:boto:Method: GET 2022-09-13T07:28:49.917 DEBUG:boto:Path: /yxvvuh-11/?prefix=%26 2022-09-13T07:28:49.918 DEBUG:boto:Data: 2022-09-13T07:28:49.919 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.919 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:49.920 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.921 DEBUG:boto:Params: {} 2022-09-13T07:28:49.922 DEBUG:boto:Token: None 2022-09-13T07:28:49.922 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/ 2022-09-13T07:28:49.923 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gYmFG0CAxA88+digsR+FobHJlKw= 2022-09-13T07:28:49.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gYmFG0CAxA88+digsR+FobHJlKw=', 'Content-Length': '0'} 2022-09-13T07:28:49.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df81977eebff3d75-0063203131-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.927 DEBUG:boto:b'yxvvuh-11&1000false&2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:49.928 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:49.929 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:49.930 DEBUG:boto:path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:49.930 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:49.931 DEBUG:boto:Method: GET 2022-09-13T07:28:49.931 DEBUG:boto:Path: /yxvvuh-11/?prefix=%26 2022-09-13T07:28:49.932 DEBUG:boto:Data: 2022-09-13T07:28:49.932 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.933 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:49.934 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.934 DEBUG:boto:Params: {} 2022-09-13T07:28:49.935 DEBUG:boto:Token: None 2022-09-13T07:28:49.935 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/ 2022-09-13T07:28:49.936 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gYmFG0CAxA88+digsR+FobHJlKw= 2022-09-13T07:28:49.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gYmFG0CAxA88+digsR+FobHJlKw=', 'Content-Length': '0'} 2022-09-13T07:28:49.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9fe599c4b647d3e-0063203131-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.940 DEBUG:boto:b'yxvvuh-11&1000false&2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:49.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:49.942 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:49.942 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:49.943 DEBUG:boto:path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:49.943 DEBUG:boto:auth_path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:49.944 DEBUG:boto:Method: GET 2022-09-13T07:28:49.944 DEBUG:boto:Path: /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:49.945 DEBUG:boto:Data: 2022-09-13T07:28:49.945 DEBUG:boto:Headers: {} 2022-09-13T07:28:49.946 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:49.947 DEBUG:boto:Port: 8000 2022-09-13T07:28:49.947 DEBUG:boto:Params: {} 2022-09-13T07:28:49.948 DEBUG:boto:Token: None 2022-09-13T07:28:49.948 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:49 GMT /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:49.949 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hd6NQGB270d78fXc1UMiaSK6/wM= 2022-09-13T07:28:49.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hd6NQGB270d78fXc1UMiaSK6/wM=', 'Content-Length': '0'} 2022-09-13T07:28:49.952 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b606189c3c5dfaa3-0063203131-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:49.998 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:49.998 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:49.999 DEBUG:boto:path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:49.999 DEBUG:boto:auth_path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.000 DEBUG:boto:Method: GET 2022-09-13T07:28:50.000 DEBUG:boto:Path: /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.001 DEBUG:boto:Data: 2022-09-13T07:28:50.002 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.002 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.003 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.003 DEBUG:boto:Params: {} 2022-09-13T07:28:50.004 DEBUG:boto:Token: None 2022-09-13T07:28:50.004 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.005 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HJinZ+9kA2g6oOQq++cJKkHp8N8= 2022-09-13T07:28:50.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HJinZ+9kA2g6oOQq++cJKkHp8N8=', 'Content-Length': '0'} 2022-09-13T07:28:50.008 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083ff6ea5dd76ff46-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.009 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:50.009 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:50.011 DEBUG:boto:path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.012 DEBUG:boto:auth_path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.013 DEBUG:boto:Method: HEAD 2022-09-13T07:28:50.014 DEBUG:boto:Path: /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.015 DEBUG:boto:Data: 2022-09-13T07:28:50.016 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.017 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.018 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.019 DEBUG:boto:Params: {} 2022-09-13T07:28:50.020 DEBUG:boto:Token: None 2022-09-13T07:28:50.021 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.022 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j2g8vpljFJ31Yo+Zgszc+lCal+c= 2022-09-13T07:28:50.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j2g8vpljFJ31Yo+Zgszc+lCal+c=', 'Content-Length': '0'} 2022-09-13T07:28:50.025 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf10379387239df4-0063203132-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.027 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:50.028 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:50.028 DEBUG:boto:path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.029 DEBUG:boto:auth_path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.029 DEBUG:boto:Method: HEAD 2022-09-13T07:28:50.030 DEBUG:boto:Path: /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.030 DEBUG:boto:Data: 2022-09-13T07:28:50.031 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.031 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.032 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.032 DEBUG:boto:Params: {} 2022-09-13T07:28:50.033 DEBUG:boto:Token: None 2022-09-13T07:28:50.033 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.034 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j2g8vpljFJ31Yo+Zgszc+lCal+c= 2022-09-13T07:28:50.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j2g8vpljFJ31Yo+Zgszc+lCal+c=', 'Content-Length': '0'} 2022-09-13T07:28:50.036 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c5f2b4066a4e0e5-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:50.039 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:50.040 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:50.041 DEBUG:boto:path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.042 DEBUG:boto:auth_path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.043 DEBUG:boto:Method: GET 2022-09-13T07:28:50.043 DEBUG:boto:Path: /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.044 DEBUG:boto:Data: 2022-09-13T07:28:50.046 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.047 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.048 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.048 DEBUG:boto:Params: {} 2022-09-13T07:28:50.049 DEBUG:boto:Token: None 2022-09-13T07:28:50.055 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.056 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HJinZ+9kA2g6oOQq++cJKkHp8N8= 2022-09-13T07:28:50.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HJinZ+9kA2g6oOQq++cJKkHp8N8=', 'Content-Length': '0'} 2022-09-13T07:28:50.060 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000095dcae033ed9001-0063203132-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.061 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:50.062 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:50.063 DEBUG:boto:path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.064 DEBUG:boto:auth_path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.065 DEBUG:boto:Method: GET 2022-09-13T07:28:50.066 DEBUG:boto:Path: /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.067 DEBUG:boto:Data: 2022-09-13T07:28:50.068 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.069 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.069 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.070 DEBUG:boto:Params: {} 2022-09-13T07:28:50.070 DEBUG:boto:Token: None 2022-09-13T07:28:50.071 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:50.072 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HJinZ+9kA2g6oOQq++cJKkHp8N8= 2022-09-13T07:28:50.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HJinZ+9kA2g6oOQq++cJKkHp8N8=', 'Content-Length': '0'} 2022-09-13T07:28:50.074 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8a31c3216972b27-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.075 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.076 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.076 DEBUG:boto:path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:50.077 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:50.078 DEBUG:boto:Method: GET 2022-09-13T07:28:50.078 DEBUG:boto:Path: /yxvvuh-11/?prefix=%3A 2022-09-13T07:28:50.079 DEBUG:boto:Data: 2022-09-13T07:28:50.079 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.080 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.080 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.081 DEBUG:boto:Params: {} 2022-09-13T07:28:50.081 DEBUG:boto:Token: None 2022-09-13T07:28:50.082 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/ 2022-09-13T07:28:50.083 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4= 2022-09-13T07:28:50.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4=', 'Content-Length': '0'} 2022-09-13T07:28:50.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6ce150a2c58eb2d-0063203132-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.087 DEBUG:boto:b'yxvvuh-11:1000false:2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.088 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.088 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.089 DEBUG:boto:path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:50.089 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:50.090 DEBUG:boto:Method: GET 2022-09-13T07:28:50.090 DEBUG:boto:Path: /yxvvuh-11/?prefix=%3A 2022-09-13T07:28:50.091 DEBUG:boto:Data: 2022-09-13T07:28:50.091 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.092 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.092 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.093 DEBUG:boto:Params: {} 2022-09-13T07:28:50.093 DEBUG:boto:Token: None 2022-09-13T07:28:50.094 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/ 2022-09-13T07:28:50.095 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4= 2022-09-13T07:28:50.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4=', 'Content-Length': '0'} 2022-09-13T07:28:50.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001acfe83c23f07424-0063203132-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.101 DEBUG:boto:b'yxvvuh-11:1000false:2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.102 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:50.103 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:50.104 DEBUG:boto:Method: HEAD 2022-09-13T07:28:50.104 DEBUG:boto:Path: /yxvvuh-11/%3A 2022-09-13T07:28:50.105 DEBUG:boto:Data: 2022-09-13T07:28:50.105 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.106 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.106 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.107 DEBUG:boto:Params: {} 2022-09-13T07:28:50.107 DEBUG:boto:Token: None 2022-09-13T07:28:50.108 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/%3A 2022-09-13T07:28:50.109 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gvX6VU+9vumHGnRXsnEgai0oluE= 2022-09-13T07:28:50.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gvX6VU+9vumHGnRXsnEgai0oluE=', 'Content-Length': '0'} 2022-09-13T07:28:50.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000060ed063d4eb6f1f-0063203132-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.112 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:50.113 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:50.113 DEBUG:boto:Method: HEAD 2022-09-13T07:28:50.114 DEBUG:boto:Path: /yxvvuh-11/%3A 2022-09-13T07:28:50.114 DEBUG:boto:Data: 2022-09-13T07:28:50.115 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.117 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.118 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.119 DEBUG:boto:Params: {} 2022-09-13T07:28:50.120 DEBUG:boto:Token: None 2022-09-13T07:28:50.121 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/%3A 2022-09-13T07:28:50.121 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gvX6VU+9vumHGnRXsnEgai0oluE= 2022-09-13T07:28:50.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gvX6VU+9vumHGnRXsnEgai0oluE=', 'Content-Length': '0'} 2022-09-13T07:28:50.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e96b43a55ca5906-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:50.126 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:50.126 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:50.127 DEBUG:boto:Method: GET 2022-09-13T07:28:50.128 DEBUG:boto:Path: /yxvvuh-11/%3A 2022-09-13T07:28:50.128 DEBUG:boto:Data: 2022-09-13T07:28:50.129 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.130 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.130 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.131 DEBUG:boto:Params: {} 2022-09-13T07:28:50.132 DEBUG:boto:Token: None 2022-09-13T07:28:50.132 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/%3A 2022-09-13T07:28:50.133 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zrALWEf4+h4A6C7cua0mCzpii2M= 2022-09-13T07:28:50.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zrALWEf4+h4A6C7cua0mCzpii2M=', 'Content-Length': '0'} 2022-09-13T07:28:50.138 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000616aa9ddbdc31eb-0063203132-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.139 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:50.140 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:50.141 DEBUG:boto:Method: GET 2022-09-13T07:28:50.142 DEBUG:boto:Path: /yxvvuh-11/%3A 2022-09-13T07:28:50.143 DEBUG:boto:Data: 2022-09-13T07:28:50.144 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.144 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.145 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.146 DEBUG:boto:Params: {} 2022-09-13T07:28:50.147 DEBUG:boto:Token: None 2022-09-13T07:28:50.148 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/%3A 2022-09-13T07:28:50.148 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zrALWEf4+h4A6C7cua0mCzpii2M= 2022-09-13T07:28:50.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zrALWEf4+h4A6C7cua0mCzpii2M=', 'Content-Length': '0'} 2022-09-13T07:28:50.151 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae9a35b3e2116251-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.194 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.195 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.196 DEBUG:boto:path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:50.196 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:50.197 DEBUG:boto:Method: GET 2022-09-13T07:28:50.197 DEBUG:boto:Path: /yxvvuh-11/?prefix=%3A 2022-09-13T07:28:50.198 DEBUG:boto:Data: 2022-09-13T07:28:50.199 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.199 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.200 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.200 DEBUG:boto:Params: {} 2022-09-13T07:28:50.201 DEBUG:boto:Token: None 2022-09-13T07:28:50.201 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/ 2022-09-13T07:28:50.202 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4= 2022-09-13T07:28:50.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4=', 'Content-Length': '0'} 2022-09-13T07:28:50.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e0a6e090552ca10-0063203132-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.206 DEBUG:boto:b'yxvvuh-11:1000false:2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.207 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.208 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.208 DEBUG:boto:path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:50.209 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:50.209 DEBUG:boto:Method: GET 2022-09-13T07:28:50.210 DEBUG:boto:Path: /yxvvuh-11/?prefix=%3A 2022-09-13T07:28:50.210 DEBUG:boto:Data: 2022-09-13T07:28:50.211 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.212 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.212 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.213 DEBUG:boto:Params: {} 2022-09-13T07:28:50.213 DEBUG:boto:Token: None 2022-09-13T07:28:50.214 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/ 2022-09-13T07:28:50.215 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4= 2022-09-13T07:28:50.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4=', 'Content-Length': '0'} 2022-09-13T07:28:50.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002798c97e21466455-0063203132-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.220 DEBUG:boto:b'yxvvuh-11:1000false:2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:50.223 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.224 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.225 DEBUG:boto:path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.226 DEBUG:boto:auth_path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.226 DEBUG:boto:Method: GET 2022-09-13T07:28:50.227 DEBUG:boto:Path: /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.228 DEBUG:boto:Data: 2022-09-13T07:28:50.229 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.230 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.231 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.232 DEBUG:boto:Params: {} 2022-09-13T07:28:50.233 DEBUG:boto:Token: None 2022-09-13T07:28:50.234 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.235 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HneNUSNTOeg8XjE8M8zYE4Zrd5g= 2022-09-13T07:28:50.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HneNUSNTOeg8XjE8M8zYE4Zrd5g=', 'Content-Length': '0'} 2022-09-13T07:28:50.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9a3f3f5ef3e224e-0063203132-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.240 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.240 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.241 DEBUG:boto:path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.241 DEBUG:boto:auth_path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.242 DEBUG:boto:Method: GET 2022-09-13T07:28:50.243 DEBUG:boto:Path: /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.243 DEBUG:boto:Data: 2022-09-13T07:28:50.244 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.245 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.246 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.247 DEBUG:boto:Params: {} 2022-09-13T07:28:50.248 DEBUG:boto:Token: None 2022-09-13T07:28:50.249 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.250 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HneNUSNTOeg8XjE8M8zYE4Zrd5g= 2022-09-13T07:28:50.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HneNUSNTOeg8XjE8M8zYE4Zrd5g=', 'Content-Length': '0'} 2022-09-13T07:28:50.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004340264ec42689c4-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.255 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.256 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.257 DEBUG:boto:path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.258 DEBUG:boto:auth_path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.259 DEBUG:boto:Method: HEAD 2022-09-13T07:28:50.259 DEBUG:boto:Path: /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.260 DEBUG:boto:Data: 2022-09-13T07:28:50.260 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.261 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.261 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.262 DEBUG:boto:Params: {} 2022-09-13T07:28:50.263 DEBUG:boto:Token: None 2022-09-13T07:28:50.264 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.265 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LVZ47RSz4SfxyWXrTzQQhQSx7og= 2022-09-13T07:28:50.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LVZ47RSz4SfxyWXrTzQQhQSx7og=', 'Content-Length': '0'} 2022-09-13T07:28:50.268 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008849db28b0816a8b-0063203132-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.269 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.269 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.270 DEBUG:boto:path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.270 DEBUG:boto:auth_path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.271 DEBUG:boto:Method: HEAD 2022-09-13T07:28:50.272 DEBUG:boto:Path: /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.272 DEBUG:boto:Data: 2022-09-13T07:28:50.273 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.273 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.274 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.274 DEBUG:boto:Params: {} 2022-09-13T07:28:50.275 DEBUG:boto:Token: None 2022-09-13T07:28:50.276 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.276 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LVZ47RSz4SfxyWXrTzQQhQSx7og= 2022-09-13T07:28:50.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LVZ47RSz4SfxyWXrTzQQhQSx7og=', 'Content-Length': '0'} 2022-09-13T07:28:50.279 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008db46f6c4b2f17c-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:50.280 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.281 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.282 DEBUG:boto:path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.282 DEBUG:boto:auth_path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.283 DEBUG:boto:Method: GET 2022-09-13T07:28:50.283 DEBUG:boto:Path: /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.284 DEBUG:boto:Data: 2022-09-13T07:28:50.285 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.285 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.286 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.286 DEBUG:boto:Params: {} 2022-09-13T07:28:50.287 DEBUG:boto:Token: None 2022-09-13T07:28:50.288 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.288 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HneNUSNTOeg8XjE8M8zYE4Zrd5g= 2022-09-13T07:28:50.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HneNUSNTOeg8XjE8M8zYE4Zrd5g=', 'Content-Length': '0'} 2022-09-13T07:28:50.292 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be608c8b6fd6d34a-0063203132-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.334 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.335 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.335 DEBUG:boto:path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.336 DEBUG:boto:auth_path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.336 DEBUG:boto:Method: GET 2022-09-13T07:28:50.337 DEBUG:boto:Path: /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.338 DEBUG:boto:Data: 2022-09-13T07:28:50.338 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.339 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.339 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.340 DEBUG:boto:Params: {} 2022-09-13T07:28:50.340 DEBUG:boto:Token: None 2022-09-13T07:28:50.341 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:50.342 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HneNUSNTOeg8XjE8M8zYE4Zrd5g= 2022-09-13T07:28:50.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HneNUSNTOeg8XjE8M8zYE4Zrd5g=', 'Content-Length': '0'} 2022-09-13T07:28:50.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088f8b13dda980c50-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.346 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.346 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.347 DEBUG:boto:path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:50.347 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:50.348 DEBUG:boto:Method: GET 2022-09-13T07:28:50.349 DEBUG:boto:Path: /yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:50.349 DEBUG:boto:Data: 2022-09-13T07:28:50.350 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.351 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.351 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.352 DEBUG:boto:Params: {} 2022-09-13T07:28:50.352 DEBUG:boto:Token: None 2022-09-13T07:28:50.353 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/ 2022-09-13T07:28:50.354 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4= 2022-09-13T07:28:50.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4=', 'Content-Length': '0'} 2022-09-13T07:28:50.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001abcea1fe66a6a53-0063203132-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.359 DEBUG:boto:b'yxvvuh-11_myobj1000false_myobj2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.360 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.361 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.362 DEBUG:boto:path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:50.363 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:50.364 DEBUG:boto:Method: GET 2022-09-13T07:28:50.365 DEBUG:boto:Path: /yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:50.365 DEBUG:boto:Data: 2022-09-13T07:28:50.366 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.366 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.367 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.367 DEBUG:boto:Params: {} 2022-09-13T07:28:50.368 DEBUG:boto:Token: None 2022-09-13T07:28:50.369 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/ 2022-09-13T07:28:50.369 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4= 2022-09-13T07:28:50.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4=', 'Content-Length': '0'} 2022-09-13T07:28:50.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f57ebb49d07ccf57-0063203132-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.373 DEBUG:boto:b'yxvvuh-11_myobj1000false_myobj2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.374 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.375 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.375 DEBUG:boto:Method: HEAD 2022-09-13T07:28:50.376 DEBUG:boto:Path: /yxvvuh-11/_myobj 2022-09-13T07:28:50.376 DEBUG:boto:Data: 2022-09-13T07:28:50.377 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.377 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.378 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.379 DEBUG:boto:Params: {} 2022-09-13T07:28:50.379 DEBUG:boto:Token: None 2022-09-13T07:28:50.380 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/_myobj 2022-09-13T07:28:50.380 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hnHvXFF7EGzm71TWG4Fjo20MePE= 2022-09-13T07:28:50.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hnHvXFF7EGzm71TWG4Fjo20MePE=', 'Content-Length': '0'} 2022-09-13T07:28:50.384 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c582a45af1e16e63-0063203132-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.385 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.386 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.386 DEBUG:boto:Method: HEAD 2022-09-13T07:28:50.387 DEBUG:boto:Path: /yxvvuh-11/_myobj 2022-09-13T07:28:50.387 DEBUG:boto:Data: 2022-09-13T07:28:50.388 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.388 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.389 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.389 DEBUG:boto:Params: {} 2022-09-13T07:28:50.390 DEBUG:boto:Token: None 2022-09-13T07:28:50.391 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/_myobj 2022-09-13T07:28:50.391 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hnHvXFF7EGzm71TWG4Fjo20MePE= 2022-09-13T07:28:50.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hnHvXFF7EGzm71TWG4Fjo20MePE=', 'Content-Length': '0'} 2022-09-13T07:28:50.395 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bc2c207ef97f580-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:50.396 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.397 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.397 DEBUG:boto:Method: GET 2022-09-13T07:28:50.398 DEBUG:boto:Path: /yxvvuh-11/_myobj 2022-09-13T07:28:50.398 DEBUG:boto:Data: 2022-09-13T07:28:50.399 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.400 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.400 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.401 DEBUG:boto:Params: {} 2022-09-13T07:28:50.401 DEBUG:boto:Token: None 2022-09-13T07:28:50.402 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/_myobj 2022-09-13T07:28:50.403 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Wnbwu3x/wiD54aFmo40R7Rc1vOI= 2022-09-13T07:28:50.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Wnbwu3x/wiD54aFmo40R7Rc1vOI=', 'Content-Length': '0'} 2022-09-13T07:28:50.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fcb317fab133e73-0063203132-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.407 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.407 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:50.408 DEBUG:boto:Method: GET 2022-09-13T07:28:50.408 DEBUG:boto:Path: /yxvvuh-11/_myobj 2022-09-13T07:28:50.409 DEBUG:boto:Data: 2022-09-13T07:28:50.409 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.410 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.410 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.411 DEBUG:boto:Params: {} 2022-09-13T07:28:50.411 DEBUG:boto:Token: None 2022-09-13T07:28:50.412 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/_myobj 2022-09-13T07:28:50.413 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Wnbwu3x/wiD54aFmo40R7Rc1vOI= 2022-09-13T07:28:50.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Wnbwu3x/wiD54aFmo40R7Rc1vOI=', 'Content-Length': '0'} 2022-09-13T07:28:50.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f02383501679ad94-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.416 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.417 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.417 DEBUG:boto:path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:50.418 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:50.418 DEBUG:boto:Method: GET 2022-09-13T07:28:50.419 DEBUG:boto:Path: /yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:50.419 DEBUG:boto:Data: 2022-09-13T07:28:50.420 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.420 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.421 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.422 DEBUG:boto:Params: {} 2022-09-13T07:28:50.422 DEBUG:boto:Token: None 2022-09-13T07:28:50.423 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/ 2022-09-13T07:28:50.423 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4= 2022-09-13T07:28:50.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4=', 'Content-Length': '0'} 2022-09-13T07:28:50.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7937455852b314c-0063203132-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.427 DEBUG:boto:b'yxvvuh-11_myobj1000false_myobj2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.428 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.429 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.429 DEBUG:boto:path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:50.430 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:50.431 DEBUG:boto:Method: GET 2022-09-13T07:28:50.431 DEBUG:boto:Path: /yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:50.432 DEBUG:boto:Data: 2022-09-13T07:28:50.432 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.433 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.433 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.434 DEBUG:boto:Params: {} 2022-09-13T07:28:50.435 DEBUG:boto:Token: None 2022-09-13T07:28:50.435 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/ 2022-09-13T07:28:50.436 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4= 2022-09-13T07:28:50.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4=', 'Content-Length': '0'} 2022-09-13T07:28:50.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9710a4972f36126-0063203132-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.440 DEBUG:boto:b'yxvvuh-11_myobj1000false_myobj2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:50.442 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:50.443 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:50.443 DEBUG:boto:path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.444 DEBUG:boto:auth_path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.445 DEBUG:boto:Method: GET 2022-09-13T07:28:50.446 DEBUG:boto:Path: /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.447 DEBUG:boto:Data: 2022-09-13T07:28:50.448 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.449 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.450 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.451 DEBUG:boto:Params: {} 2022-09-13T07:28:50.452 DEBUG:boto:Token: None 2022-09-13T07:28:50.453 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.454 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zpc5+uyRV+M/UCHl14eTtCCypLw= 2022-09-13T07:28:50.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zpc5+uyRV+M/UCHl14eTtCCypLw=', 'Content-Length': '0'} 2022-09-13T07:28:50.457 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082b8b049cb9ae35f-0063203132-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.457 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:50.458 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:50.459 DEBUG:boto:path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.459 DEBUG:boto:auth_path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.460 DEBUG:boto:Method: GET 2022-09-13T07:28:50.460 DEBUG:boto:Path: /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.461 DEBUG:boto:Data: 2022-09-13T07:28:50.461 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.462 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.462 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.463 DEBUG:boto:Params: {} 2022-09-13T07:28:50.464 DEBUG:boto:Token: None 2022-09-13T07:28:50.464 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.465 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zpc5+uyRV+M/UCHl14eTtCCypLw= 2022-09-13T07:28:50.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zpc5+uyRV+M/UCHl14eTtCCypLw=', 'Content-Length': '0'} 2022-09-13T07:28:50.468 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab9cc681d4e8a4cc-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.469 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:50.470 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:50.471 DEBUG:boto:path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.472 DEBUG:boto:auth_path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.473 DEBUG:boto:Method: HEAD 2022-09-13T07:28:50.473 DEBUG:boto:Path: /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.474 DEBUG:boto:Data: 2022-09-13T07:28:50.475 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.476 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.476 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.477 DEBUG:boto:Params: {} 2022-09-13T07:28:50.478 DEBUG:boto:Token: None 2022-09-13T07:28:50.479 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.480 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZLqxjJgLXuEntdQEbiGXSpifw94= 2022-09-13T07:28:50.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZLqxjJgLXuEntdQEbiGXSpifw94=', 'Content-Length': '0'} 2022-09-13T07:28:50.483 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000759856fb728d9810-0063203132-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.485 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:50.485 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:50.486 DEBUG:boto:path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.487 DEBUG:boto:auth_path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.488 DEBUG:boto:Method: HEAD 2022-09-13T07:28:50.489 DEBUG:boto:Path: /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.490 DEBUG:boto:Data: 2022-09-13T07:28:50.490 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.491 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.492 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.493 DEBUG:boto:Params: {} 2022-09-13T07:28:50.494 DEBUG:boto:Token: None 2022-09-13T07:28:50.494 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.495 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZLqxjJgLXuEntdQEbiGXSpifw94= 2022-09-13T07:28:50.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZLqxjJgLXuEntdQEbiGXSpifw94=', 'Content-Length': '0'} 2022-09-13T07:28:50.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060d137c88c076502-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:50.501 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:50.502 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:50.503 DEBUG:boto:path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.503 DEBUG:boto:auth_path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.504 DEBUG:boto:Method: GET 2022-09-13T07:28:50.505 DEBUG:boto:Path: /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.506 DEBUG:boto:Data: 2022-09-13T07:28:50.507 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.508 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.509 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.510 DEBUG:boto:Params: {} 2022-09-13T07:28:50.511 DEBUG:boto:Token: None 2022-09-13T07:28:50.512 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.513 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zpc5+uyRV+M/UCHl14eTtCCypLw= 2022-09-13T07:28:50.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zpc5+uyRV+M/UCHl14eTtCCypLw=', 'Content-Length': '0'} 2022-09-13T07:28:50.516 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f06405beabf7921-0063203132-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.517 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:50.517 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:50.518 DEBUG:boto:path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.519 DEBUG:boto:auth_path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.519 DEBUG:boto:Method: GET 2022-09-13T07:28:50.520 DEBUG:boto:Path: /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.520 DEBUG:boto:Data: 2022-09-13T07:28:50.521 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.521 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.522 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.522 DEBUG:boto:Params: {} 2022-09-13T07:28:50.523 DEBUG:boto:Token: None 2022-09-13T07:28:50.524 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:50.524 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zpc5+uyRV+M/UCHl14eTtCCypLw= 2022-09-13T07:28:50.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zpc5+uyRV+M/UCHl14eTtCCypLw=', 'Content-Length': '0'} 2022-09-13T07:28:50.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4116f2eb8f29a9a-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.570 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.571 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.572 DEBUG:boto:path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:50.573 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:50.574 DEBUG:boto:Method: GET 2022-09-13T07:28:50.575 DEBUG:boto:Path: /yxvvuh-11/?prefix=myobj 2022-09-13T07:28:50.576 DEBUG:boto:Data: 2022-09-13T07:28:50.576 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.577 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.578 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.578 DEBUG:boto:Params: {} 2022-09-13T07:28:50.579 DEBUG:boto:Token: None 2022-09-13T07:28:50.579 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/ 2022-09-13T07:28:50.580 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4= 2022-09-13T07:28:50.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4=', 'Content-Length': '0'} 2022-09-13T07:28:50.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3fb4b297e1db27b-0063203132-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.585 DEBUG:boto:b'yxvvuh-11myobj1000falsemyobj2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.586 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.587 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.587 DEBUG:boto:path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:50.588 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:50.589 DEBUG:boto:Method: GET 2022-09-13T07:28:50.590 DEBUG:boto:Path: /yxvvuh-11/?prefix=myobj 2022-09-13T07:28:50.591 DEBUG:boto:Data: 2022-09-13T07:28:50.592 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.593 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.594 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.595 DEBUG:boto:Params: {} 2022-09-13T07:28:50.596 DEBUG:boto:Token: None 2022-09-13T07:28:50.597 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/ 2022-09-13T07:28:50.597 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4= 2022-09-13T07:28:50.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4=', 'Content-Length': '0'} 2022-09-13T07:28:50.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e68af3d9bef687fb-0063203132-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.601 DEBUG:boto:b'yxvvuh-11myobj1000falsemyobj2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.602 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:50.603 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:50.603 DEBUG:boto:Method: HEAD 2022-09-13T07:28:50.604 DEBUG:boto:Path: /yxvvuh-11/myobj 2022-09-13T07:28:50.604 DEBUG:boto:Data: 2022-09-13T07:28:50.605 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.605 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.605 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.606 DEBUG:boto:Params: {} 2022-09-13T07:28:50.607 DEBUG:boto:Token: None 2022-09-13T07:28:50.607 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/myobj 2022-09-13T07:28:50.608 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8H/ZGk2rTM+BLj0IGMw6W4CiGV4= 2022-09-13T07:28:50.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8H/ZGk2rTM+BLj0IGMw6W4CiGV4=', 'Content-Length': '0'} 2022-09-13T07:28:50.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000316e13941dd71ac7-0063203132-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.611 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:50.612 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:50.612 DEBUG:boto:Method: HEAD 2022-09-13T07:28:50.613 DEBUG:boto:Path: /yxvvuh-11/myobj 2022-09-13T07:28:50.613 DEBUG:boto:Data: 2022-09-13T07:28:50.614 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.614 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.615 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.616 DEBUG:boto:Params: {} 2022-09-13T07:28:50.616 DEBUG:boto:Token: None 2022-09-13T07:28:50.617 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/myobj 2022-09-13T07:28:50.617 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8H/ZGk2rTM+BLj0IGMw6W4CiGV4= 2022-09-13T07:28:50.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8H/ZGk2rTM+BLj0IGMw6W4CiGV4=', 'Content-Length': '0'} 2022-09-13T07:28:50.620 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a0b1fd718838bb1-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:50.622 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:50.622 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:50.623 DEBUG:boto:Method: GET 2022-09-13T07:28:50.623 DEBUG:boto:Path: /yxvvuh-11/myobj 2022-09-13T07:28:50.624 DEBUG:boto:Data: 2022-09-13T07:28:50.624 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.625 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.626 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.626 DEBUG:boto:Params: {} 2022-09-13T07:28:50.627 DEBUG:boto:Token: None 2022-09-13T07:28:50.627 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/myobj 2022-09-13T07:28:50.628 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ilDAFQfMbzZuCW2kJzD871MWD+g= 2022-09-13T07:28:50.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ilDAFQfMbzZuCW2kJzD871MWD+g=', 'Content-Length': '0'} 2022-09-13T07:28:50.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c76c49ee0f48e95-0063203132-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.674 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:50.675 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:50.675 DEBUG:boto:Method: GET 2022-09-13T07:28:50.676 DEBUG:boto:Path: /yxvvuh-11/myobj 2022-09-13T07:28:50.677 DEBUG:boto:Data: 2022-09-13T07:28:50.677 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.678 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.678 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.679 DEBUG:boto:Params: {} 2022-09-13T07:28:50.680 DEBUG:boto:Token: None 2022-09-13T07:28:50.680 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/myobj 2022-09-13T07:28:50.681 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ilDAFQfMbzZuCW2kJzD871MWD+g= 2022-09-13T07:28:50.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ilDAFQfMbzZuCW2kJzD871MWD+g=', 'Content-Length': '0'} 2022-09-13T07:28:50.684 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000070cae2786c8d6cc-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.685 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.685 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.686 DEBUG:boto:path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:50.686 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:50.687 DEBUG:boto:Method: GET 2022-09-13T07:28:50.688 DEBUG:boto:Path: /yxvvuh-11/?prefix=myobj 2022-09-13T07:28:50.688 DEBUG:boto:Data: 2022-09-13T07:28:50.689 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.690 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:50.690 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.691 DEBUG:boto:Params: {} 2022-09-13T07:28:50.691 DEBUG:boto:Token: None 2022-09-13T07:28:50.692 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/ 2022-09-13T07:28:50.693 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4= 2022-09-13T07:28:50.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4=', 'Content-Length': '0'} 2022-09-13T07:28:50.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c138d28dee389c5-0063203132-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.698 DEBUG:boto:b'yxvvuh-11myobj1000falsemyobj2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.699 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.700 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.701 DEBUG:boto:path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:50.702 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:50.703 DEBUG:boto:Method: GET 2022-09-13T07:28:50.704 DEBUG:boto:Path: /yxvvuh-11/?prefix=myobj 2022-09-13T07:28:50.705 DEBUG:boto:Data: 2022-09-13T07:28:50.706 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.707 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.707 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.708 DEBUG:boto:Params: {} 2022-09-13T07:28:50.709 DEBUG:boto:Token: None 2022-09-13T07:28:50.710 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/ 2022-09-13T07:28:50.711 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4= 2022-09-13T07:28:50.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:czZEMXdQtGKdlvJOFFz2PhGmsC4=', 'Content-Length': '0'} 2022-09-13T07:28:50.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035be81fa05953ab0-0063203132-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.716 DEBUG:boto:b'yxvvuh-11myobj1000falsemyobj2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.718 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-11 zones={test-zone1, test-zone2} 2022-09-13T07:28:50.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:28:50.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:28:50.720 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:28:50.788 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:50.788 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:50.837 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:50.833+0000 7ff82c9a3600 1 bucket sync caught up with source: 2022-09-13T07:28:50.838 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.18.5, , 00000000001.7.5, , , , 00000000001.5.5, 00000000001.16.5, , ] 2022-09-13T07:28:50.838 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000001.18.5, , 00000000001.7.5, , , , 00000000001.5.5, 00000000001.16.5, , ] 2022-09-13T07:28:50.839 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:50.833+0000 7ff82c9a3600 0 bucket checkpoint complete 2022-09-13T07:28:50.854 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-11 zones={test-zone3, test-zone2} 2022-09-13T07:28:50.855 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.856 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.857 DEBUG:boto:Method: HEAD 2022-09-13T07:28:50.858 DEBUG:boto:Path: /yxvvuh-11/ 2022-09-13T07:28:50.859 DEBUG:boto:Data: 2022-09-13T07:28:50.860 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.861 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:50.862 DEBUG:boto:Port: 8001 2022-09-13T07:28:50.863 DEBUG:boto:Params: {} 2022-09-13T07:28:50.864 DEBUG:boto:Token: None 2022-09-13T07:28:50.865 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/ 2022-09-13T07:28:50.865 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:90dXonx9OMoBWwYt6sXEjfeEFgs= 2022-09-13T07:28:50.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:90dXonx9OMoBWwYt6sXEjfeEFgs=', 'Content-Length': '0'} 2022-09-13T07:28:50.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a6f8ebcba4b5cfe-0063203132-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.869 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.870 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.870 DEBUG:boto:Method: HEAD 2022-09-13T07:28:50.871 DEBUG:boto:Path: /yxvvuh-11/ 2022-09-13T07:28:50.871 DEBUG:boto:Data: 2022-09-13T07:28:50.872 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.872 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.873 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.873 DEBUG:boto:Params: {} 2022-09-13T07:28:50.874 DEBUG:boto:Token: None 2022-09-13T07:28:50.875 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/ 2022-09-13T07:28:50.876 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:90dXonx9OMoBWwYt6sXEjfeEFgs= 2022-09-13T07:28:50.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:90dXonx9OMoBWwYt6sXEjfeEFgs=', 'Content-Length': '0'} 2022-09-13T07:28:50.879 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009225529a5811519c-0063203132-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.880 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:28:50.881 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.881 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.882 DEBUG:boto:path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:50.883 DEBUG:boto:auth_path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:50.883 DEBUG:boto:Method: GET 2022-09-13T07:28:50.884 DEBUG:boto:Path: /yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:50.884 DEBUG:boto:Data: 2022-09-13T07:28:50.885 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.886 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:50.886 DEBUG:boto:Port: 8001 2022-09-13T07:28:50.887 DEBUG:boto:Params: {} 2022-09-13T07:28:50.888 DEBUG:boto:Token: None 2022-09-13T07:28:50.888 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/?versions 2022-09-13T07:28:50.889 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KvFYUcrJmKTTH91wal8aPmClyIg= 2022-09-13T07:28:50.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KvFYUcrJmKTTH91wal8aPmClyIg=', 'Content-Length': '0'} 2022-09-13T07:28:50.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005243927b5b7ee52d-0063203132-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.893 DEBUG:boto:b'yxvvuh-11999false&nulltrue2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.894 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-13T07:28:50.894 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-13T07:28:50.895 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-13T07:28:50.895 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:28:50.896 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:28:50.896 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.897 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.897 DEBUG:boto:path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:50.898 DEBUG:boto:auth_path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:50.899 DEBUG:boto:Method: GET 2022-09-13T07:28:50.899 DEBUG:boto:Path: /yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:50.900 DEBUG:boto:Data: 2022-09-13T07:28:50.901 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.902 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.903 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.904 DEBUG:boto:Params: {} 2022-09-13T07:28:50.905 DEBUG:boto:Token: None 2022-09-13T07:28:50.906 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/?versions 2022-09-13T07:28:50.907 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KvFYUcrJmKTTH91wal8aPmClyIg= 2022-09-13T07:28:50.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KvFYUcrJmKTTH91wal8aPmClyIg=', 'Content-Length': '0'} 2022-09-13T07:28:50.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eedcaffe79dde236-0063203132-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.913 DEBUG:boto:b'yxvvuh-11999false&nulltrue2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.914 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-13T07:28:50.915 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-13T07:28:50.916 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-13T07:28:50.916 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:28:50.917 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.917 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.918 DEBUG:boto:path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:50.918 DEBUG:boto:auth_path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:50.919 DEBUG:boto:Method: GET 2022-09-13T07:28:50.919 DEBUG:boto:Path: /yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:50.920 DEBUG:boto:Data: 2022-09-13T07:28:50.920 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.921 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:50.921 DEBUG:boto:Port: 8001 2022-09-13T07:28:50.922 DEBUG:boto:Params: {} 2022-09-13T07:28:50.922 DEBUG:boto:Token: None 2022-09-13T07:28:50.923 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/?versions 2022-09-13T07:28:50.923 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KvFYUcrJmKTTH91wal8aPmClyIg= 2022-09-13T07:28:50.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KvFYUcrJmKTTH91wal8aPmClyIg=', 'Content-Length': '0'} 2022-09-13T07:28:50.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5876f5d07263365-0063203132-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.927 DEBUG:boto:b'yxvvuh-11999false&nulltrue2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.928 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:50.929 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:50.930 DEBUG:boto:path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:50.930 DEBUG:boto:auth_path=/yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:50.931 DEBUG:boto:Method: GET 2022-09-13T07:28:50.931 DEBUG:boto:Path: /yxvvuh-11/?versions&max-keys=999 2022-09-13T07:28:50.932 DEBUG:boto:Data: 2022-09-13T07:28:50.933 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.933 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.934 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.934 DEBUG:boto:Params: {} 2022-09-13T07:28:50.935 DEBUG:boto:Token: None 2022-09-13T07:28:50.936 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/?versions 2022-09-13T07:28:50.936 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KvFYUcrJmKTTH91wal8aPmClyIg= 2022-09-13T07:28:50.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KvFYUcrJmKTTH91wal8aPmClyIg=', 'Content-Length': '0'} 2022-09-13T07:28:50.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007070477df50bdfff-0063203132-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.940 DEBUG:boto:b'yxvvuh-11999false&nulltrue2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:50.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:50.942 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:50.942 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:50.943 DEBUG:boto:path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:50.944 DEBUG:boto:auth_path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:50.944 DEBUG:boto:Method: GET 2022-09-13T07:28:50.945 DEBUG:boto:Path: /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:50.945 DEBUG:boto:Data: 2022-09-13T07:28:50.946 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.946 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:50.947 DEBUG:boto:Port: 8001 2022-09-13T07:28:50.947 DEBUG:boto:Params: {} 2022-09-13T07:28:50.948 DEBUG:boto:Token: None 2022-09-13T07:28:50.949 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:50.949 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:miha0I/898EHCeJCPz82jq7hmC0= 2022-09-13T07:28:50.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:miha0I/898EHCeJCPz82jq7hmC0=', 'Content-Length': '0'} 2022-09-13T07:28:50.952 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e958b823d9f30c7-0063203132-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:50.953 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:50.954 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:50.954 DEBUG:boto:path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:50.955 DEBUG:boto:auth_path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:50.955 DEBUG:boto:Method: GET 2022-09-13T07:28:50.956 DEBUG:boto:Path: /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:50.957 DEBUG:boto:Data: 2022-09-13T07:28:50.957 DEBUG:boto:Headers: {} 2022-09-13T07:28:50.958 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:50.958 DEBUG:boto:Port: 8000 2022-09-13T07:28:50.959 DEBUG:boto:Params: {} 2022-09-13T07:28:50.960 DEBUG:boto:Token: None 2022-09-13T07:28:50.960 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:50 GMT /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:50.961 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:miha0I/898EHCeJCPz82jq7hmC0= 2022-09-13T07:28:50.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:miha0I/898EHCeJCPz82jq7hmC0=', 'Content-Length': '0'} 2022-09-13T07:28:50.964 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004291403991db47ff-0063203132-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.006 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:51.008 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:51.009 DEBUG:boto:path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.010 DEBUG:boto:auth_path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.010 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.011 DEBUG:boto:Path: /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.012 DEBUG:boto:Data: 2022-09-13T07:28:51.013 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.014 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.014 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.015 DEBUG:boto:Params: {} 2022-09-13T07:28:51.015 DEBUG:boto:Token: None 2022-09-13T07:28:51.016 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.017 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tk5fKsga5gG6sWlR5TPY2w7b6y0= 2022-09-13T07:28:51.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tk5fKsga5gG6sWlR5TPY2w7b6y0=', 'Content-Length': '0'} 2022-09-13T07:28:51.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000064fe66335fd89bd-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.022 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:51.022 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:51.023 DEBUG:boto:path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.024 DEBUG:boto:auth_path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.025 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.026 DEBUG:boto:Path: /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.026 DEBUG:boto:Data: 2022-09-13T07:28:51.027 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.028 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.028 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.029 DEBUG:boto:Params: {} 2022-09-13T07:28:51.030 DEBUG:boto:Token: None 2022-09-13T07:28:51.031 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.032 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tk5fKsga5gG6sWlR5TPY2w7b6y0= 2022-09-13T07:28:51.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tk5fKsga5gG6sWlR5TPY2w7b6y0=', 'Content-Length': '0'} 2022-09-13T07:28:51.036 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000955f83908df15e07-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:51.039 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:51.040 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:51.040 DEBUG:boto:path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.041 DEBUG:boto:auth_path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.042 DEBUG:boto:Method: GET 2022-09-13T07:28:51.042 DEBUG:boto:Path: /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.043 DEBUG:boto:Data: 2022-09-13T07:28:51.043 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.044 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.044 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.045 DEBUG:boto:Params: {} 2022-09-13T07:28:51.046 DEBUG:boto:Token: None 2022-09-13T07:28:51.047 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.048 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KieHzT9gD7Bh6Tk+GTQy2b/Vjnk= 2022-09-13T07:28:51.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KieHzT9gD7Bh6Tk+GTQy2b/Vjnk=', 'Content-Length': '0'} 2022-09-13T07:28:51.050 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002bb9febad11e0334-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.052 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:51.052 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:51.053 DEBUG:boto:path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.053 DEBUG:boto:auth_path=/yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.054 DEBUG:boto:Method: GET 2022-09-13T07:28:51.055 DEBUG:boto:Path: /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.055 DEBUG:boto:Data: 2022-09-13T07:28:51.056 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.056 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.057 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.058 DEBUG:boto:Params: {} 2022-09-13T07:28:51.058 DEBUG:boto:Token: None 2022-09-13T07:28:51.059 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%26?versionId=null 2022-09-13T07:28:51.060 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KieHzT9gD7Bh6Tk+GTQy2b/Vjnk= 2022-09-13T07:28:51.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KieHzT9gD7Bh6Tk+GTQy2b/Vjnk=', 'Content-Length': '0'} 2022-09-13T07:28:51.062 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cffb86d3d4cbe24-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.063 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:51.064 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:51.064 DEBUG:boto:path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:51.065 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:51.065 DEBUG:boto:Method: GET 2022-09-13T07:28:51.066 DEBUG:boto:Path: /yxvvuh-11/?prefix=%26 2022-09-13T07:28:51.067 DEBUG:boto:Data: 2022-09-13T07:28:51.068 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.069 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.069 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.070 DEBUG:boto:Params: {} 2022-09-13T07:28:51.071 DEBUG:boto:Token: None 2022-09-13T07:28:51.072 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/ 2022-09-13T07:28:51.073 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI= 2022-09-13T07:28:51.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI=', 'Content-Length': '0'} 2022-09-13T07:28:51.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3f8265f21610df6-0063203133-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.118 DEBUG:boto:b'yxvvuh-11&1000false&2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:51.119 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:51.120 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:51.120 DEBUG:boto:path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:51.121 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:51.121 DEBUG:boto:Method: GET 2022-09-13T07:28:51.122 DEBUG:boto:Path: /yxvvuh-11/?prefix=%26 2022-09-13T07:28:51.122 DEBUG:boto:Data: 2022-09-13T07:28:51.123 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.123 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.124 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.124 DEBUG:boto:Params: {} 2022-09-13T07:28:51.125 DEBUG:boto:Token: None 2022-09-13T07:28:51.125 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/ 2022-09-13T07:28:51.126 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI= 2022-09-13T07:28:51.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI=', 'Content-Length': '0'} 2022-09-13T07:28:51.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000261f0eece99e41e8-0063203133-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.130 DEBUG:boto:b'yxvvuh-11&1000false&2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:51.131 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:51.132 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:51.132 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.133 DEBUG:boto:Path: /yxvvuh-11/%26 2022-09-13T07:28:51.133 DEBUG:boto:Data: 2022-09-13T07:28:51.134 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.134 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.135 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.135 DEBUG:boto:Params: {} 2022-09-13T07:28:51.137 DEBUG:boto:Token: None 2022-09-13T07:28:51.138 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%26 2022-09-13T07:28:51.138 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6q2fDdLEvcW+EGoCwtzvctz19Vs= 2022-09-13T07:28:51.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6q2fDdLEvcW+EGoCwtzvctz19Vs=', 'Content-Length': '0'} 2022-09-13T07:28:51.141 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000febe9411a8dfdaa4-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.143 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:51.144 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:51.144 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.145 DEBUG:boto:Path: /yxvvuh-11/%26 2022-09-13T07:28:51.146 DEBUG:boto:Data: 2022-09-13T07:28:51.147 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.148 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.148 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.149 DEBUG:boto:Params: {} 2022-09-13T07:28:51.150 DEBUG:boto:Token: None 2022-09-13T07:28:51.151 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%26 2022-09-13T07:28:51.152 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6q2fDdLEvcW+EGoCwtzvctz19Vs= 2022-09-13T07:28:51.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6q2fDdLEvcW+EGoCwtzvctz19Vs=', 'Content-Length': '0'} 2022-09-13T07:28:51.155 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000633752d27d61565c-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:51.157 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:51.159 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:51.160 DEBUG:boto:Method: GET 2022-09-13T07:28:51.161 DEBUG:boto:Path: /yxvvuh-11/%26 2022-09-13T07:28:51.161 DEBUG:boto:Data: 2022-09-13T07:28:51.162 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.162 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.163 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.164 DEBUG:boto:Params: {} 2022-09-13T07:28:51.164 DEBUG:boto:Token: None 2022-09-13T07:28:51.165 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%26 2022-09-13T07:28:51.165 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h8X/ucoI5FnLjoBeBKpBIIaSkY4= 2022-09-13T07:28:51.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h8X/ucoI5FnLjoBeBKpBIIaSkY4=', 'Content-Length': '0'} 2022-09-13T07:28:51.168 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6cbf67bb6197c89-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.169 DEBUG:boto:path=/yxvvuh-11/%26 2022-09-13T07:28:51.169 DEBUG:boto:auth_path=/yxvvuh-11/%26 2022-09-13T07:28:51.170 DEBUG:boto:Method: GET 2022-09-13T07:28:51.171 DEBUG:boto:Path: /yxvvuh-11/%26 2022-09-13T07:28:51.171 DEBUG:boto:Data: 2022-09-13T07:28:51.172 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.172 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.173 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.173 DEBUG:boto:Params: {} 2022-09-13T07:28:51.174 DEBUG:boto:Token: None 2022-09-13T07:28:51.175 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%26 2022-09-13T07:28:51.175 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h8X/ucoI5FnLjoBeBKpBIIaSkY4= 2022-09-13T07:28:51.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h8X/ucoI5FnLjoBeBKpBIIaSkY4=', 'Content-Length': '0'} 2022-09-13T07:28:51.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2ae18a5bcf2d674-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.179 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:51.179 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:51.180 DEBUG:boto:path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:51.181 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:51.181 DEBUG:boto:Method: GET 2022-09-13T07:28:51.182 DEBUG:boto:Path: /yxvvuh-11/?prefix=%26 2022-09-13T07:28:51.182 DEBUG:boto:Data: 2022-09-13T07:28:51.183 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.183 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.184 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.184 DEBUG:boto:Params: {} 2022-09-13T07:28:51.185 DEBUG:boto:Token: None 2022-09-13T07:28:51.186 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/ 2022-09-13T07:28:51.186 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI= 2022-09-13T07:28:51.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI=', 'Content-Length': '0'} 2022-09-13T07:28:51.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000368575c70dd99bbe-0063203133-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.190 DEBUG:boto:b'yxvvuh-11&1000false&2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:51.191 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:51.192 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:51.194 DEBUG:boto:path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:51.194 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%26 2022-09-13T07:28:51.195 DEBUG:boto:Method: GET 2022-09-13T07:28:51.195 DEBUG:boto:Path: /yxvvuh-11/?prefix=%26 2022-09-13T07:28:51.196 DEBUG:boto:Data: 2022-09-13T07:28:51.197 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.197 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.198 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.198 DEBUG:boto:Params: {} 2022-09-13T07:28:51.199 DEBUG:boto:Token: None 2022-09-13T07:28:51.200 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/ 2022-09-13T07:28:51.200 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI= 2022-09-13T07:28:51.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI=', 'Content-Length': '0'} 2022-09-13T07:28:51.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b31d6afc5442e767-0063203133-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.205 DEBUG:boto:b'yxvvuh-11&1000false&2022-09-13T07:28:26.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:51.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:51.207 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:51.208 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:51.209 DEBUG:boto:path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.209 DEBUG:boto:auth_path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.210 DEBUG:boto:Method: GET 2022-09-13T07:28:51.211 DEBUG:boto:Path: /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.212 DEBUG:boto:Data: 2022-09-13T07:28:51.213 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.214 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.215 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.216 DEBUG:boto:Params: {} 2022-09-13T07:28:51.217 DEBUG:boto:Token: None 2022-09-13T07:28:51.218 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.219 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZIvsz5UhEQrDQYoAjJmiqAVdZRU= 2022-09-13T07:28:51.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZIvsz5UhEQrDQYoAjJmiqAVdZRU=', 'Content-Length': '0'} 2022-09-13T07:28:51.223 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4abd9ef8a3543d4-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.224 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:51.225 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:51.226 DEBUG:boto:path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.226 DEBUG:boto:auth_path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.227 DEBUG:boto:Method: GET 2022-09-13T07:28:51.228 DEBUG:boto:Path: /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.228 DEBUG:boto:Data: 2022-09-13T07:28:51.229 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.230 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.230 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.231 DEBUG:boto:Params: {} 2022-09-13T07:28:51.232 DEBUG:boto:Token: None 2022-09-13T07:28:51.232 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.233 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZIvsz5UhEQrDQYoAjJmiqAVdZRU= 2022-09-13T07:28:51.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZIvsz5UhEQrDQYoAjJmiqAVdZRU=', 'Content-Length': '0'} 2022-09-13T07:28:51.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f637fe5dd38091b-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.278 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:51.279 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:51.279 DEBUG:boto:path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.280 DEBUG:boto:auth_path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.280 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.281 DEBUG:boto:Path: /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.282 DEBUG:boto:Data: 2022-09-13T07:28:51.283 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.284 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.285 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.286 DEBUG:boto:Params: {} 2022-09-13T07:28:51.287 DEBUG:boto:Token: None 2022-09-13T07:28:51.288 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.289 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j0O2V4MkYeCTVASexyyUNOJQKCA= 2022-09-13T07:28:51.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j0O2V4MkYeCTVASexyyUNOJQKCA=', 'Content-Length': '0'} 2022-09-13T07:28:51.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ed1cc5a0438ea57-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.294 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:51.295 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:51.296 DEBUG:boto:path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.297 DEBUG:boto:auth_path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.298 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.299 DEBUG:boto:Path: /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.300 DEBUG:boto:Data: 2022-09-13T07:28:51.301 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.302 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.303 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.304 DEBUG:boto:Params: {} 2022-09-13T07:28:51.305 DEBUG:boto:Token: None 2022-09-13T07:28:51.306 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.307 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j0O2V4MkYeCTVASexyyUNOJQKCA= 2022-09-13T07:28:51.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j0O2V4MkYeCTVASexyyUNOJQKCA=', 'Content-Length': '0'} 2022-09-13T07:28:51.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008530249fa2aa88c3-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:51.312 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:51.313 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:51.313 DEBUG:boto:path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.314 DEBUG:boto:auth_path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.314 DEBUG:boto:Method: GET 2022-09-13T07:28:51.315 DEBUG:boto:Path: /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.315 DEBUG:boto:Data: 2022-09-13T07:28:51.316 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.316 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.317 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.317 DEBUG:boto:Params: {} 2022-09-13T07:28:51.318 DEBUG:boto:Token: None 2022-09-13T07:28:51.319 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.319 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZIvsz5UhEQrDQYoAjJmiqAVdZRU= 2022-09-13T07:28:51.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZIvsz5UhEQrDQYoAjJmiqAVdZRU=', 'Content-Length': '0'} 2022-09-13T07:28:51.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eff521b4cd1f2597-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.331 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:51.332 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:51.333 DEBUG:boto:path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.334 DEBUG:boto:auth_path=/yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.335 DEBUG:boto:Method: GET 2022-09-13T07:28:51.336 DEBUG:boto:Path: /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.337 DEBUG:boto:Data: 2022-09-13T07:28:51.338 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.339 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.340 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.341 DEBUG:boto:Params: {} 2022-09-13T07:28:51.342 DEBUG:boto:Token: None 2022-09-13T07:28:51.343 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%3A?versionId=null 2022-09-13T07:28:51.344 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZIvsz5UhEQrDQYoAjJmiqAVdZRU= 2022-09-13T07:28:51.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZIvsz5UhEQrDQYoAjJmiqAVdZRU=', 'Content-Length': '0'} 2022-09-13T07:28:51.348 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008791bbe7c34a1e04-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.349 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:51.350 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:51.351 DEBUG:boto:path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:51.352 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:51.353 DEBUG:boto:Method: GET 2022-09-13T07:28:51.354 DEBUG:boto:Path: /yxvvuh-11/?prefix=%3A 2022-09-13T07:28:51.355 DEBUG:boto:Data: 2022-09-13T07:28:51.356 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.356 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.357 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.358 DEBUG:boto:Params: {} 2022-09-13T07:28:51.360 DEBUG:boto:Token: None 2022-09-13T07:28:51.361 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/ 2022-09-13T07:28:51.362 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI= 2022-09-13T07:28:51.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI=', 'Content-Length': '0'} 2022-09-13T07:28:51.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea2a3d6ccda80930-0063203133-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.367 DEBUG:boto:b'yxvvuh-11:1000false:2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:51.369 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:51.370 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:51.371 DEBUG:boto:path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:51.372 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:51.372 DEBUG:boto:Method: GET 2022-09-13T07:28:51.373 DEBUG:boto:Path: /yxvvuh-11/?prefix=%3A 2022-09-13T07:28:51.374 DEBUG:boto:Data: 2022-09-13T07:28:51.374 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.375 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.375 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.376 DEBUG:boto:Params: {} 2022-09-13T07:28:51.377 DEBUG:boto:Token: None 2022-09-13T07:28:51.377 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/ 2022-09-13T07:28:51.378 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI= 2022-09-13T07:28:51.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI=', 'Content-Length': '0'} 2022-09-13T07:28:51.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088333a0f1586cc95-0063203133-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.382 DEBUG:boto:b'yxvvuh-11:1000false:2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:51.383 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:51.384 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:51.384 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.385 DEBUG:boto:Path: /yxvvuh-11/%3A 2022-09-13T07:28:51.385 DEBUG:boto:Data: 2022-09-13T07:28:51.386 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.387 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.387 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.388 DEBUG:boto:Params: {} 2022-09-13T07:28:51.389 DEBUG:boto:Token: None 2022-09-13T07:28:51.390 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%3A 2022-09-13T07:28:51.391 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Ag9ztF1tBlQwQ/rvyooSMnYlVQY= 2022-09-13T07:28:51.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Ag9ztF1tBlQwQ/rvyooSMnYlVQY=', 'Content-Length': '0'} 2022-09-13T07:28:51.395 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf02f83cbbe5c9c0-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.396 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:51.397 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:51.398 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.399 DEBUG:boto:Path: /yxvvuh-11/%3A 2022-09-13T07:28:51.399 DEBUG:boto:Data: 2022-09-13T07:28:51.400 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.400 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.401 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.402 DEBUG:boto:Params: {} 2022-09-13T07:28:51.402 DEBUG:boto:Token: None 2022-09-13T07:28:51.403 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%3A 2022-09-13T07:28:51.403 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Ag9ztF1tBlQwQ/rvyooSMnYlVQY= 2022-09-13T07:28:51.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Ag9ztF1tBlQwQ/rvyooSMnYlVQY=', 'Content-Length': '0'} 2022-09-13T07:28:51.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8068db0ac69a42e-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:51.408 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:51.409 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:51.409 DEBUG:boto:Method: GET 2022-09-13T07:28:51.410 DEBUG:boto:Path: /yxvvuh-11/%3A 2022-09-13T07:28:51.410 DEBUG:boto:Data: 2022-09-13T07:28:51.411 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.412 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.412 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.413 DEBUG:boto:Params: {} 2022-09-13T07:28:51.414 DEBUG:boto:Token: None 2022-09-13T07:28:51.414 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%3A 2022-09-13T07:28:51.415 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9dazxpaz/Mp3jIh53dss3TGW+PQ= 2022-09-13T07:28:51.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9dazxpaz/Mp3jIh53dss3TGW+PQ=', 'Content-Length': '0'} 2022-09-13T07:28:51.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006438399f3dbeb400-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.458 DEBUG:boto:path=/yxvvuh-11/%3A 2022-09-13T07:28:51.459 DEBUG:boto:auth_path=/yxvvuh-11/%3A 2022-09-13T07:28:51.460 DEBUG:boto:Method: GET 2022-09-13T07:28:51.460 DEBUG:boto:Path: /yxvvuh-11/%3A 2022-09-13T07:28:51.461 DEBUG:boto:Data: 2022-09-13T07:28:51.462 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.462 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.463 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.464 DEBUG:boto:Params: {} 2022-09-13T07:28:51.464 DEBUG:boto:Token: None 2022-09-13T07:28:51.465 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/%3A 2022-09-13T07:28:51.466 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9dazxpaz/Mp3jIh53dss3TGW+PQ= 2022-09-13T07:28:51.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9dazxpaz/Mp3jIh53dss3TGW+PQ=', 'Content-Length': '0'} 2022-09-13T07:28:51.468 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023f2cd22232859ff-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.469 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:51.471 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:51.472 DEBUG:boto:path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:51.473 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:51.474 DEBUG:boto:Method: GET 2022-09-13T07:28:51.475 DEBUG:boto:Path: /yxvvuh-11/?prefix=%3A 2022-09-13T07:28:51.476 DEBUG:boto:Data: 2022-09-13T07:28:51.477 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.478 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.479 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.480 DEBUG:boto:Params: {} 2022-09-13T07:28:51.481 DEBUG:boto:Token: None 2022-09-13T07:28:51.482 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/ 2022-09-13T07:28:51.484 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI= 2022-09-13T07:28:51.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI=', 'Content-Length': '0'} 2022-09-13T07:28:51.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a65bcabb10c407-0063203133-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.489 DEBUG:boto:b'yxvvuh-11:1000false:2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:51.491 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:51.492 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:51.493 DEBUG:boto:path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:51.494 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=%3A 2022-09-13T07:28:51.495 DEBUG:boto:Method: GET 2022-09-13T07:28:51.496 DEBUG:boto:Path: /yxvvuh-11/?prefix=%3A 2022-09-13T07:28:51.497 DEBUG:boto:Data: 2022-09-13T07:28:51.498 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.499 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.500 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.501 DEBUG:boto:Params: {} 2022-09-13T07:28:51.502 DEBUG:boto:Token: None 2022-09-13T07:28:51.504 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/ 2022-09-13T07:28:51.505 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI= 2022-09-13T07:28:51.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI=', 'Content-Length': '0'} 2022-09-13T07:28:51.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000590ffaafebe7e8d3-0063203133-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.511 DEBUG:boto:b'yxvvuh-11:1000false:2022-09-13T07:28:26.378Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:51.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:51.515 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.516 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.517 DEBUG:boto:path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.518 DEBUG:boto:auth_path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.519 DEBUG:boto:Method: GET 2022-09-13T07:28:51.520 DEBUG:boto:Path: /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.521 DEBUG:boto:Data: 2022-09-13T07:28:51.522 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.523 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.524 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.525 DEBUG:boto:Params: {} 2022-09-13T07:28:51.527 DEBUG:boto:Token: None 2022-09-13T07:28:51.528 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.529 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW+gKF2RyfNQwFd9pDl3H7z+Y4= 2022-09-13T07:28:51.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW+gKF2RyfNQwFd9pDl3H7z+Y4=', 'Content-Length': '0'} 2022-09-13T07:28:51.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069f703f4086eb577-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.535 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.536 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.537 DEBUG:boto:path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.538 DEBUG:boto:auth_path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.539 DEBUG:boto:Method: GET 2022-09-13T07:28:51.540 DEBUG:boto:Path: /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.541 DEBUG:boto:Data: 2022-09-13T07:28:51.541 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.542 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.543 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.543 DEBUG:boto:Params: {} 2022-09-13T07:28:51.544 DEBUG:boto:Token: None 2022-09-13T07:28:51.545 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.546 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW+gKF2RyfNQwFd9pDl3H7z+Y4= 2022-09-13T07:28:51.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW+gKF2RyfNQwFd9pDl3H7z+Y4=', 'Content-Length': '0'} 2022-09-13T07:28:51.548 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000994fe08030fd1ca-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.590 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.591 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.591 DEBUG:boto:path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.592 DEBUG:boto:auth_path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.593 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.593 DEBUG:boto:Path: /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.594 DEBUG:boto:Data: 2022-09-13T07:28:51.594 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.595 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.595 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.596 DEBUG:boto:Params: {} 2022-09-13T07:28:51.596 DEBUG:boto:Token: None 2022-09-13T07:28:51.597 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.598 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:21/BWBZzVO2eaU6h7oetUJYN1uc= 2022-09-13T07:28:51.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:21/BWBZzVO2eaU6h7oetUJYN1uc=', 'Content-Length': '0'} 2022-09-13T07:28:51.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f37d826f9c2ceb8-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.603 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.603 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.604 DEBUG:boto:path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.606 DEBUG:boto:auth_path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.607 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.608 DEBUG:boto:Path: /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.608 DEBUG:boto:Data: 2022-09-13T07:28:51.610 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.611 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.612 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.613 DEBUG:boto:Params: {} 2022-09-13T07:28:51.614 DEBUG:boto:Token: None 2022-09-13T07:28:51.615 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.616 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:21/BWBZzVO2eaU6h7oetUJYN1uc= 2022-09-13T07:28:51.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:21/BWBZzVO2eaU6h7oetUJYN1uc=', 'Content-Length': '0'} 2022-09-13T07:28:51.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1998ca3a95db93e-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:51.623 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.624 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.625 DEBUG:boto:path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.627 DEBUG:boto:auth_path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.628 DEBUG:boto:Method: GET 2022-09-13T07:28:51.629 DEBUG:boto:Path: /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.630 DEBUG:boto:Data: 2022-09-13T07:28:51.631 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.631 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.632 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.633 DEBUG:boto:Params: {} 2022-09-13T07:28:51.634 DEBUG:boto:Token: None 2022-09-13T07:28:51.635 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.635 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW+gKF2RyfNQwFd9pDl3H7z+Y4= 2022-09-13T07:28:51.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW+gKF2RyfNQwFd9pDl3H7z+Y4=', 'Content-Length': '0'} 2022-09-13T07:28:51.638 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043e9aa193397427b-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.639 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.640 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.640 DEBUG:boto:path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.641 DEBUG:boto:auth_path=/yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.641 DEBUG:boto:Method: GET 2022-09-13T07:28:51.642 DEBUG:boto:Path: /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.642 DEBUG:boto:Data: 2022-09-13T07:28:51.643 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.644 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.645 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.646 DEBUG:boto:Params: {} 2022-09-13T07:28:51.647 DEBUG:boto:Token: None 2022-09-13T07:28:51.648 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/_myobj?versionId=null 2022-09-13T07:28:51.649 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW+gKF2RyfNQwFd9pDl3H7z+Y4= 2022-09-13T07:28:51.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW+gKF2RyfNQwFd9pDl3H7z+Y4=', 'Content-Length': '0'} 2022-09-13T07:28:51.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fae8e53ab4a3478f-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.655 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:51.656 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:51.657 DEBUG:boto:path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:51.657 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:51.658 DEBUG:boto:Method: GET 2022-09-13T07:28:51.659 DEBUG:boto:Path: /yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:51.659 DEBUG:boto:Data: 2022-09-13T07:28:51.660 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.660 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.661 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.662 DEBUG:boto:Params: {} 2022-09-13T07:28:51.663 DEBUG:boto:Token: None 2022-09-13T07:28:51.663 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/ 2022-09-13T07:28:51.664 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI= 2022-09-13T07:28:51.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI=', 'Content-Length': '0'} 2022-09-13T07:28:51.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebafe47322c2a621-0063203133-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.668 DEBUG:boto:b'yxvvuh-11_myobj1000false_myobj2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:51.669 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:51.670 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:51.671 DEBUG:boto:path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:51.672 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:51.673 DEBUG:boto:Method: GET 2022-09-13T07:28:51.674 DEBUG:boto:Path: /yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:51.674 DEBUG:boto:Data: 2022-09-13T07:28:51.675 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.676 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.677 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.678 DEBUG:boto:Params: {} 2022-09-13T07:28:51.679 DEBUG:boto:Token: None 2022-09-13T07:28:51.680 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/ 2022-09-13T07:28:51.681 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI= 2022-09-13T07:28:51.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI=', 'Content-Length': '0'} 2022-09-13T07:28:51.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f07348d07f99868b-0063203133-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.686 DEBUG:boto:b'yxvvuh-11_myobj1000false_myobj2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:51.687 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.688 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.688 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.689 DEBUG:boto:Path: /yxvvuh-11/_myobj 2022-09-13T07:28:51.689 DEBUG:boto:Data: 2022-09-13T07:28:51.689 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.690 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.691 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.691 DEBUG:boto:Params: {} 2022-09-13T07:28:51.692 DEBUG:boto:Token: None 2022-09-13T07:28:51.692 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/_myobj 2022-09-13T07:28:51.693 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KhZNN5m5jSKeldaxhJmby3qktJ8= 2022-09-13T07:28:51.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KhZNN5m5jSKeldaxhJmby3qktJ8=', 'Content-Length': '0'} 2022-09-13T07:28:51.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002ce04263bace982-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.697 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.698 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.698 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.699 DEBUG:boto:Path: /yxvvuh-11/_myobj 2022-09-13T07:28:51.699 DEBUG:boto:Data: 2022-09-13T07:28:51.700 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.700 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.701 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.701 DEBUG:boto:Params: {} 2022-09-13T07:28:51.702 DEBUG:boto:Token: None 2022-09-13T07:28:51.703 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/_myobj 2022-09-13T07:28:51.704 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KhZNN5m5jSKeldaxhJmby3qktJ8= 2022-09-13T07:28:51.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KhZNN5m5jSKeldaxhJmby3qktJ8=', 'Content-Length': '0'} 2022-09-13T07:28:51.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c165b756f44e51b-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:51.710 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.710 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.711 DEBUG:boto:Method: GET 2022-09-13T07:28:51.712 DEBUG:boto:Path: /yxvvuh-11/_myobj 2022-09-13T07:28:51.713 DEBUG:boto:Data: 2022-09-13T07:28:51.713 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.714 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.715 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.716 DEBUG:boto:Params: {} 2022-09-13T07:28:51.716 DEBUG:boto:Token: None 2022-09-13T07:28:51.717 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/_myobj 2022-09-13T07:28:51.717 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:sx9XjpsJZn7Jy15lHPYWmxWYLtg= 2022-09-13T07:28:51.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:sx9XjpsJZn7Jy15lHPYWmxWYLtg=', 'Content-Length': '0'} 2022-09-13T07:28:51.720 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000352c9a43b1d61d7f-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.766 DEBUG:boto:path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.767 DEBUG:boto:auth_path=/yxvvuh-11/_myobj 2022-09-13T07:28:51.768 DEBUG:boto:Method: GET 2022-09-13T07:28:51.769 DEBUG:boto:Path: /yxvvuh-11/_myobj 2022-09-13T07:28:51.770 DEBUG:boto:Data: 2022-09-13T07:28:51.771 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.772 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.773 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.774 DEBUG:boto:Params: {} 2022-09-13T07:28:51.775 DEBUG:boto:Token: None 2022-09-13T07:28:51.776 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/_myobj 2022-09-13T07:28:51.777 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:sx9XjpsJZn7Jy15lHPYWmxWYLtg= 2022-09-13T07:28:51.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:sx9XjpsJZn7Jy15lHPYWmxWYLtg=', 'Content-Length': '0'} 2022-09-13T07:28:51.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac477408dd0ae381-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.782 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:51.783 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:51.783 DEBUG:boto:path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:51.784 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:51.785 DEBUG:boto:Method: GET 2022-09-13T07:28:51.785 DEBUG:boto:Path: /yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:51.786 DEBUG:boto:Data: 2022-09-13T07:28:51.786 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.787 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.787 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.788 DEBUG:boto:Params: {} 2022-09-13T07:28:51.789 DEBUG:boto:Token: None 2022-09-13T07:28:51.790 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/ 2022-09-13T07:28:51.791 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI= 2022-09-13T07:28:51.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI=', 'Content-Length': '0'} 2022-09-13T07:28:51.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8f2390be93972f5-0063203133-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.795 DEBUG:boto:b'yxvvuh-11_myobj1000false_myobj2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:51.796 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:51.797 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:51.798 DEBUG:boto:path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:51.799 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:51.800 DEBUG:boto:Method: GET 2022-09-13T07:28:51.800 DEBUG:boto:Path: /yxvvuh-11/?prefix=_myobj 2022-09-13T07:28:51.801 DEBUG:boto:Data: 2022-09-13T07:28:51.802 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.803 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.804 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.805 DEBUG:boto:Params: {} 2022-09-13T07:28:51.805 DEBUG:boto:Token: None 2022-09-13T07:28:51.806 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/ 2022-09-13T07:28:51.807 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI= 2022-09-13T07:28:51.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI=', 'Content-Length': '0'} 2022-09-13T07:28:51.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6ebe7df37db7ffd-0063203133-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.812 DEBUG:boto:b'yxvvuh-11_myobj1000false_myobj2022-09-13T07:28:26.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:51.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:51.815 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:51.816 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:51.816 DEBUG:boto:path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.817 DEBUG:boto:auth_path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.818 DEBUG:boto:Method: GET 2022-09-13T07:28:51.819 DEBUG:boto:Path: /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.820 DEBUG:boto:Data: 2022-09-13T07:28:51.821 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.822 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.823 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.824 DEBUG:boto:Params: {} 2022-09-13T07:28:51.825 DEBUG:boto:Token: None 2022-09-13T07:28:51.826 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.827 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c/j2KNmHSYESxiM9ssaB32LVkSU= 2022-09-13T07:28:51.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c/j2KNmHSYESxiM9ssaB32LVkSU=', 'Content-Length': '0'} 2022-09-13T07:28:51.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cde651fa608fa83-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.832 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:51.833 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:51.834 DEBUG:boto:path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.834 DEBUG:boto:auth_path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.836 DEBUG:boto:Method: GET 2022-09-13T07:28:51.837 DEBUG:boto:Path: /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.838 DEBUG:boto:Data: 2022-09-13T07:28:51.839 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.840 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.841 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.842 DEBUG:boto:Params: {} 2022-09-13T07:28:51.843 DEBUG:boto:Token: None 2022-09-13T07:28:51.844 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.845 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c/j2KNmHSYESxiM9ssaB32LVkSU= 2022-09-13T07:28:51.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c/j2KNmHSYESxiM9ssaB32LVkSU=', 'Content-Length': '0'} 2022-09-13T07:28:51.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c42167acc9b89707-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.850 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:51.851 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:51.852 DEBUG:boto:path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.852 DEBUG:boto:auth_path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.853 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.854 DEBUG:boto:Path: /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.855 DEBUG:boto:Data: 2022-09-13T07:28:51.855 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.856 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.856 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.857 DEBUG:boto:Params: {} 2022-09-13T07:28:51.857 DEBUG:boto:Token: None 2022-09-13T07:28:51.858 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.858 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bEafF2fn9JrsrTjs8gZv6dQ/B8s= 2022-09-13T07:28:51.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bEafF2fn9JrsrTjs8gZv6dQ/B8s=', 'Content-Length': '0'} 2022-09-13T07:28:51.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075914ac5d544b012-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.862 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:51.862 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:51.863 DEBUG:boto:path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.863 DEBUG:boto:auth_path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.864 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.864 DEBUG:boto:Path: /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.865 DEBUG:boto:Data: 2022-09-13T07:28:51.865 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.866 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.866 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.867 DEBUG:boto:Params: {} 2022-09-13T07:28:51.867 DEBUG:boto:Token: None 2022-09-13T07:28:51.868 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.868 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bEafF2fn9JrsrTjs8gZv6dQ/B8s= 2022-09-13T07:28:51.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bEafF2fn9JrsrTjs8gZv6dQ/B8s=', 'Content-Length': '0'} 2022-09-13T07:28:51.871 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e7f7cfdadd08cc4-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:51.872 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:51.873 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:51.873 DEBUG:boto:path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.874 DEBUG:boto:auth_path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.874 DEBUG:boto:Method: GET 2022-09-13T07:28:51.875 DEBUG:boto:Path: /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.875 DEBUG:boto:Data: 2022-09-13T07:28:51.876 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.876 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.877 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.878 DEBUG:boto:Params: {} 2022-09-13T07:28:51.878 DEBUG:boto:Token: None 2022-09-13T07:28:51.879 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.879 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c/j2KNmHSYESxiM9ssaB32LVkSU= 2022-09-13T07:28:51.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c/j2KNmHSYESxiM9ssaB32LVkSU=', 'Content-Length': '0'} 2022-09-13T07:28:51.882 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000e7fb8878c58fe3-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.926 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:51.927 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:51.927 DEBUG:boto:path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.928 DEBUG:boto:auth_path=/yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.928 DEBUG:boto:Method: GET 2022-09-13T07:28:51.929 DEBUG:boto:Path: /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.930 DEBUG:boto:Data: 2022-09-13T07:28:51.930 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.931 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.931 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.932 DEBUG:boto:Params: {} 2022-09-13T07:28:51.932 DEBUG:boto:Token: None 2022-09-13T07:28:51.933 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/myobj?versionId=null 2022-09-13T07:28:51.933 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c/j2KNmHSYESxiM9ssaB32LVkSU= 2022-09-13T07:28:51.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c/j2KNmHSYESxiM9ssaB32LVkSU=', 'Content-Length': '0'} 2022-09-13T07:28:51.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b56e1563c6c7223-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.937 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:51.938 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:51.938 DEBUG:boto:path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:51.939 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:51.939 DEBUG:boto:Method: GET 2022-09-13T07:28:51.940 DEBUG:boto:Path: /yxvvuh-11/?prefix=myobj 2022-09-13T07:28:51.940 DEBUG:boto:Data: 2022-09-13T07:28:51.941 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.941 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.942 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.943 DEBUG:boto:Params: {} 2022-09-13T07:28:51.943 DEBUG:boto:Token: None 2022-09-13T07:28:51.944 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/ 2022-09-13T07:28:51.944 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI= 2022-09-13T07:28:51.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI=', 'Content-Length': '0'} 2022-09-13T07:28:51.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fb8645f06ac517b-0063203133-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.948 DEBUG:boto:b'yxvvuh-11myobj1000falsemyobj2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:51.949 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:51.950 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:51.950 DEBUG:boto:path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:51.951 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:51.951 DEBUG:boto:Method: GET 2022-09-13T07:28:51.952 DEBUG:boto:Path: /yxvvuh-11/?prefix=myobj 2022-09-13T07:28:51.952 DEBUG:boto:Data: 2022-09-13T07:28:51.953 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.953 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.954 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.954 DEBUG:boto:Params: {} 2022-09-13T07:28:51.955 DEBUG:boto:Token: None 2022-09-13T07:28:51.956 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/ 2022-09-13T07:28:51.956 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI= 2022-09-13T07:28:51.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QJilTEDS4vjpEYPpFzDYYUNXzqI=', 'Content-Length': '0'} 2022-09-13T07:28:51.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057a270a2f499589a-0063203133-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.960 DEBUG:boto:b'yxvvuh-11myobj1000falsemyobj2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:51.961 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:51.962 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:51.963 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.963 DEBUG:boto:Path: /yxvvuh-11/myobj 2022-09-13T07:28:51.964 DEBUG:boto:Data: 2022-09-13T07:28:51.964 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.965 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.965 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.965 DEBUG:boto:Params: {} 2022-09-13T07:28:51.966 DEBUG:boto:Token: None 2022-09-13T07:28:51.967 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/myobj 2022-09-13T07:28:51.967 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wm4zz6AFk6cOu4oo+POqXe2bwmE= 2022-09-13T07:28:51.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wm4zz6AFk6cOu4oo+POqXe2bwmE=', 'Content-Length': '0'} 2022-09-13T07:28:51.970 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a63eba08275fd15-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.971 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:51.972 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:51.972 DEBUG:boto:Method: HEAD 2022-09-13T07:28:51.973 DEBUG:boto:Path: /yxvvuh-11/myobj 2022-09-13T07:28:51.974 DEBUG:boto:Data: 2022-09-13T07:28:51.974 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.975 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.976 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.976 DEBUG:boto:Params: {} 2022-09-13T07:28:51.977 DEBUG:boto:Token: None 2022-09-13T07:28:51.978 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/myobj 2022-09-13T07:28:51.978 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wm4zz6AFk6cOu4oo+POqXe2bwmE= 2022-09-13T07:28:51.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wm4zz6AFk6cOu4oo+POqXe2bwmE=', 'Content-Length': '0'} 2022-09-13T07:28:51.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5dd21803a143ef9-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:51.982 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:51.983 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:51.983 DEBUG:boto:Method: GET 2022-09-13T07:28:51.984 DEBUG:boto:Path: /yxvvuh-11/myobj 2022-09-13T07:28:51.985 DEBUG:boto:Data: 2022-09-13T07:28:51.985 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.986 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:51.986 DEBUG:boto:Port: 8001 2022-09-13T07:28:51.987 DEBUG:boto:Params: {} 2022-09-13T07:28:51.987 DEBUG:boto:Token: None 2022-09-13T07:28:51.988 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/myobj 2022-09-13T07:28:51.989 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S0QvHjPu7kVrcKwzqpGnnHvwO3A= 2022-09-13T07:28:51.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S0QvHjPu7kVrcKwzqpGnnHvwO3A=', 'Content-Length': '0'} 2022-09-13T07:28:51.991 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000878fd1e947ff3ef1-0063203133-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:51.992 DEBUG:boto:path=/yxvvuh-11/myobj 2022-09-13T07:28:51.993 DEBUG:boto:auth_path=/yxvvuh-11/myobj 2022-09-13T07:28:51.993 DEBUG:boto:Method: GET 2022-09-13T07:28:51.994 DEBUG:boto:Path: /yxvvuh-11/myobj 2022-09-13T07:28:51.994 DEBUG:boto:Data: 2022-09-13T07:28:51.995 DEBUG:boto:Headers: {} 2022-09-13T07:28:51.995 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:51.996 DEBUG:boto:Port: 8000 2022-09-13T07:28:51.997 DEBUG:boto:Params: {} 2022-09-13T07:28:51.998 DEBUG:boto:Token: None 2022-09-13T07:28:51.999 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:51 GMT /yxvvuh-11/myobj 2022-09-13T07:28:52.000 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S0QvHjPu7kVrcKwzqpGnnHvwO3A= 2022-09-13T07:28:52.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S0QvHjPu7kVrcKwzqpGnnHvwO3A=', 'Content-Length': '0'} 2022-09-13T07:28:52.003 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f8ecbef11a39849-0063203133-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.046 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:52.048 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:52.048 DEBUG:boto:path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:52.050 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:52.051 DEBUG:boto:Method: GET 2022-09-13T07:28:52.051 DEBUG:boto:Path: /yxvvuh-11/?prefix=myobj 2022-09-13T07:28:52.052 DEBUG:boto:Data: 2022-09-13T07:28:52.053 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.054 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.054 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.055 DEBUG:boto:Params: {} 2022-09-13T07:28:52.056 DEBUG:boto:Token: None 2022-09-13T07:28:52.057 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-11/ 2022-09-13T07:28:52.058 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LcnIia/VcTo1ca/WXKYi12Xl/sQ= 2022-09-13T07:28:52.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LcnIia/VcTo1ca/WXKYi12Xl/sQ=', 'Content-Length': '0'} 2022-09-13T07:28:52.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073604a081fb93ce6-0063203134-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.064 DEBUG:boto:b'yxvvuh-11myobj1000falsemyobj2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.065 DEBUG:boto:path=/yxvvuh-11/ 2022-09-13T07:28:52.066 DEBUG:boto:auth_path=/yxvvuh-11/ 2022-09-13T07:28:52.067 DEBUG:boto:path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:52.068 DEBUG:boto:auth_path=/yxvvuh-11/?prefix=myobj 2022-09-13T07:28:52.068 DEBUG:boto:Method: GET 2022-09-13T07:28:52.069 DEBUG:boto:Path: /yxvvuh-11/?prefix=myobj 2022-09-13T07:28:52.069 DEBUG:boto:Data: 2022-09-13T07:28:52.070 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.070 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:52.071 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.071 DEBUG:boto:Params: {} 2022-09-13T07:28:52.072 DEBUG:boto:Token: None 2022-09-13T07:28:52.073 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-11/ 2022-09-13T07:28:52.073 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LcnIia/VcTo1ca/WXKYi12Xl/sQ= 2022-09-13T07:28:52.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LcnIia/VcTo1ca/WXKYi12Xl/sQ=', 'Content-Length': '0'} 2022-09-13T07:28:52.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000359d7267110e8110-0063203134-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.078 DEBUG:boto:b'yxvvuh-11myobj1000falsemyobj2022-09-13T07:28:26.292Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.079 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-11 zones={test-zone3, test-zone2} 2022-09-13T07:28:52.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:28:52.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:28:52.082 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:28:52.153 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:52.154 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:52.231 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:52.228+0000 7fbeb569d600 1 bucket sync caught up with source: 2022-09-13T07:28:52.232 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.27.5, , 00000000001.7.5, , , , 00000000001.8.5, 00000000001.29.5, , ] 2022-09-13T07:28:52.233 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000001.27.5, , 00000000001.7.5, , , , 00000000001.8.5, 00000000001.29.5, , ] 2022-09-13T07:28:52.233 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:28:52.228+0000 7fbeb569d600 0 bucket checkpoint complete 2022-09-13T07:28:52.261 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-12 zones={test-zone1, test-zone3} 2022-09-13T07:28:52.262 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.262 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.263 DEBUG:boto:Method: HEAD 2022-09-13T07:28:52.263 DEBUG:boto:Path: /yxvvuh-12/ 2022-09-13T07:28:52.264 DEBUG:boto:Data: 2022-09-13T07:28:52.264 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.265 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.265 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.266 DEBUG:boto:Params: {} 2022-09-13T07:28:52.266 DEBUG:boto:Token: None 2022-09-13T07:28:52.267 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/ 2022-09-13T07:28:52.267 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wZwHL5e6gJnCwagtDoFC5LHoFHQ= 2022-09-13T07:28:52.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wZwHL5e6gJnCwagtDoFC5LHoFHQ=', 'Content-Length': '0'} 2022-09-13T07:28:52.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028d95cf50966eb04-0063203134-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.271 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.272 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.272 DEBUG:boto:Method: HEAD 2022-09-13T07:28:52.273 DEBUG:boto:Path: /yxvvuh-12/ 2022-09-13T07:28:52.273 DEBUG:boto:Data: 2022-09-13T07:28:52.274 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.274 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.275 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.275 DEBUG:boto:Params: {} 2022-09-13T07:28:52.276 DEBUG:boto:Token: None 2022-09-13T07:28:52.277 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/ 2022-09-13T07:28:52.278 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wZwHL5e6gJnCwagtDoFC5LHoFHQ= 2022-09-13T07:28:52.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wZwHL5e6gJnCwagtDoFC5LHoFHQ=', 'Content-Length': '0'} 2022-09-13T07:28:52.281 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049c532a83bedeed3-0063203134-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.281 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:28:52.282 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.282 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.283 DEBUG:boto:path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:52.283 DEBUG:boto:auth_path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:52.284 DEBUG:boto:Method: GET 2022-09-13T07:28:52.285 DEBUG:boto:Path: /yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:52.285 DEBUG:boto:Data: 2022-09-13T07:28:52.286 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.286 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.287 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.287 DEBUG:boto:Params: {} 2022-09-13T07:28:52.288 DEBUG:boto:Token: None 2022-09-13T07:28:52.289 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/?versions 2022-09-13T07:28:52.290 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TmUs2oPg6nGOdDiUFj1Ob+7A200= 2022-09-13T07:28:52.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TmUs2oPg6nGOdDiUFj1Ob+7A200=', 'Content-Length': '0'} 2022-09-13T07:28:52.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c16870e259f79e33-0063203134-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.296 DEBUG:boto:b'yxvvuh-12999false&nulltrue2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.297 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-13T07:28:52.298 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-13T07:28:52.301 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-13T07:28:52.302 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:28:52.303 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:28:52.304 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.304 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.305 DEBUG:boto:path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:52.306 DEBUG:boto:auth_path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:52.307 DEBUG:boto:Method: GET 2022-09-13T07:28:52.308 DEBUG:boto:Path: /yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:52.309 DEBUG:boto:Data: 2022-09-13T07:28:52.310 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.311 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.312 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.313 DEBUG:boto:Params: {} 2022-09-13T07:28:52.314 DEBUG:boto:Token: None 2022-09-13T07:28:52.315 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/?versions 2022-09-13T07:28:52.316 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TmUs2oPg6nGOdDiUFj1Ob+7A200= 2022-09-13T07:28:52.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TmUs2oPg6nGOdDiUFj1Ob+7A200=', 'Content-Length': '0'} 2022-09-13T07:28:52.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0801b7d131cb1f5-0063203134-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.322 DEBUG:boto:b'yxvvuh-12999false&nulltrue2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.324 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-13T07:28:52.324 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-13T07:28:52.325 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-13T07:28:52.326 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:28:52.327 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.328 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.329 DEBUG:boto:path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:52.329 DEBUG:boto:auth_path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:52.330 DEBUG:boto:Method: GET 2022-09-13T07:28:52.331 DEBUG:boto:Path: /yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:52.332 DEBUG:boto:Data: 2022-09-13T07:28:52.333 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.333 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.334 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.334 DEBUG:boto:Params: {} 2022-09-13T07:28:52.335 DEBUG:boto:Token: None 2022-09-13T07:28:52.335 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/?versions 2022-09-13T07:28:52.336 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TmUs2oPg6nGOdDiUFj1Ob+7A200= 2022-09-13T07:28:52.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TmUs2oPg6nGOdDiUFj1Ob+7A200=', 'Content-Length': '0'} 2022-09-13T07:28:52.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f378c4652d7f7bb-0063203134-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.340 DEBUG:boto:b'yxvvuh-12999false&nulltrue2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.342 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.342 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.343 DEBUG:boto:path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:52.343 DEBUG:boto:auth_path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:52.344 DEBUG:boto:Method: GET 2022-09-13T07:28:52.344 DEBUG:boto:Path: /yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:52.345 DEBUG:boto:Data: 2022-09-13T07:28:52.346 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.346 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.347 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.347 DEBUG:boto:Params: {} 2022-09-13T07:28:52.348 DEBUG:boto:Token: None 2022-09-13T07:28:52.349 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/?versions 2022-09-13T07:28:52.350 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TmUs2oPg6nGOdDiUFj1Ob+7A200= 2022-09-13T07:28:52.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TmUs2oPg6nGOdDiUFj1Ob+7A200=', 'Content-Length': '0'} 2022-09-13T07:28:52.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009700ec0b9f108441-0063203134-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.355 DEBUG:boto:b'yxvvuh-12999false&nulltrue2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:52.357 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:52.358 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:52.359 DEBUG:boto:path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.360 DEBUG:boto:auth_path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.361 DEBUG:boto:Method: GET 2022-09-13T07:28:52.362 DEBUG:boto:Path: /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.363 DEBUG:boto:Data: 2022-09-13T07:28:52.364 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.365 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.366 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.367 DEBUG:boto:Params: {} 2022-09-13T07:28:52.368 DEBUG:boto:Token: None 2022-09-13T07:28:52.368 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.369 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z0B0NVDzUGD1jLfGNlAvhNwgVHI= 2022-09-13T07:28:52.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z0B0NVDzUGD1jLfGNlAvhNwgVHI=', 'Content-Length': '0'} 2022-09-13T07:28:52.373 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001be02d1b45e27ab1-0063203134-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.414 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:52.415 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:52.416 DEBUG:boto:path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.416 DEBUG:boto:auth_path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.417 DEBUG:boto:Method: GET 2022-09-13T07:28:52.417 DEBUG:boto:Path: /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.418 DEBUG:boto:Data: 2022-09-13T07:28:52.419 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.420 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.420 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.421 DEBUG:boto:Params: {} 2022-09-13T07:28:52.421 DEBUG:boto:Token: None 2022-09-13T07:28:52.422 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.423 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z0B0NVDzUGD1jLfGNlAvhNwgVHI= 2022-09-13T07:28:52.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z0B0NVDzUGD1jLfGNlAvhNwgVHI=', 'Content-Length': '0'} 2022-09-13T07:28:52.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004448e26f08f18db9-0063203134-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.427 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:52.427 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:52.428 DEBUG:boto:path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.428 DEBUG:boto:auth_path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.429 DEBUG:boto:Method: HEAD 2022-09-13T07:28:52.429 DEBUG:boto:Path: /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.430 DEBUG:boto:Data: 2022-09-13T07:28:52.431 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.431 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.432 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.432 DEBUG:boto:Params: {} 2022-09-13T07:28:52.433 DEBUG:boto:Token: None 2022-09-13T07:28:52.433 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.434 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gprXfclqDPdI8nrniguojxLdGc0= 2022-09-13T07:28:52.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gprXfclqDPdI8nrniguojxLdGc0=', 'Content-Length': '0'} 2022-09-13T07:28:52.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000711e941e58a42897-0063203134-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.437 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:52.438 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:52.438 DEBUG:boto:path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.439 DEBUG:boto:auth_path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.439 DEBUG:boto:Method: HEAD 2022-09-13T07:28:52.440 DEBUG:boto:Path: /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.440 DEBUG:boto:Data: 2022-09-13T07:28:52.441 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.441 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.442 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.442 DEBUG:boto:Params: {} 2022-09-13T07:28:52.443 DEBUG:boto:Token: None 2022-09-13T07:28:52.444 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.444 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gprXfclqDPdI8nrniguojxLdGc0= 2022-09-13T07:28:52.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gprXfclqDPdI8nrniguojxLdGc0=', 'Content-Length': '0'} 2022-09-13T07:28:52.447 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcc5075da3bcae1d-0063203134-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:52.448 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:52.449 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:52.449 DEBUG:boto:path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.450 DEBUG:boto:auth_path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.450 DEBUG:boto:Method: GET 2022-09-13T07:28:52.451 DEBUG:boto:Path: /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.451 DEBUG:boto:Data: 2022-09-13T07:28:52.452 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.452 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.453 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.453 DEBUG:boto:Params: {} 2022-09-13T07:28:52.454 DEBUG:boto:Token: None 2022-09-13T07:28:52.455 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.456 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z0B0NVDzUGD1jLfGNlAvhNwgVHI= 2022-09-13T07:28:52.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z0B0NVDzUGD1jLfGNlAvhNwgVHI=', 'Content-Length': '0'} 2022-09-13T07:28:52.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a943e3aaab29b61-0063203134-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.459 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:52.459 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:52.460 DEBUG:boto:path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.460 DEBUG:boto:auth_path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.461 DEBUG:boto:Method: GET 2022-09-13T07:28:52.461 DEBUG:boto:Path: /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.462 DEBUG:boto:Data: 2022-09-13T07:28:52.462 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.463 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.464 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.464 DEBUG:boto:Params: {} 2022-09-13T07:28:52.465 DEBUG:boto:Token: None 2022-09-13T07:28:52.465 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:52.466 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z0B0NVDzUGD1jLfGNlAvhNwgVHI= 2022-09-13T07:28:52.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z0B0NVDzUGD1jLfGNlAvhNwgVHI=', 'Content-Length': '0'} 2022-09-13T07:28:52.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a4b95e3a78d0a78-0063203134-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.510 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.511 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.511 DEBUG:boto:path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:52.512 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:52.513 DEBUG:boto:Method: GET 2022-09-13T07:28:52.513 DEBUG:boto:Path: /yxvvuh-12/?prefix=%26 2022-09-13T07:28:52.514 DEBUG:boto:Data: 2022-09-13T07:28:52.514 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.515 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.515 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.516 DEBUG:boto:Params: {} 2022-09-13T07:28:52.517 DEBUG:boto:Token: None 2022-09-13T07:28:52.517 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/ 2022-09-13T07:28:52.518 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY= 2022-09-13T07:28:52.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY=', 'Content-Length': '0'} 2022-09-13T07:28:52.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018389a5af75989cc-0063203134-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.521 DEBUG:boto:b'yxvvuh-12&1000false&2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.522 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.523 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.523 DEBUG:boto:path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:52.524 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:52.524 DEBUG:boto:Method: GET 2022-09-13T07:28:52.525 DEBUG:boto:Path: /yxvvuh-12/?prefix=%26 2022-09-13T07:28:52.525 DEBUG:boto:Data: 2022-09-13T07:28:52.526 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.526 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.527 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.527 DEBUG:boto:Params: {} 2022-09-13T07:28:52.528 DEBUG:boto:Token: None 2022-09-13T07:28:52.529 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/ 2022-09-13T07:28:52.529 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY= 2022-09-13T07:28:52.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY=', 'Content-Length': '0'} 2022-09-13T07:28:52.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc02bc317f1b2cf8-0063203134-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.533 DEBUG:boto:b'yxvvuh-12&1000false&2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.534 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:52.535 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:52.535 DEBUG:boto:Method: HEAD 2022-09-13T07:28:52.536 DEBUG:boto:Path: /yxvvuh-12/%26 2022-09-13T07:28:52.537 DEBUG:boto:Data: 2022-09-13T07:28:52.538 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.538 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.539 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.539 DEBUG:boto:Params: {} 2022-09-13T07:28:52.540 DEBUG:boto:Token: None 2022-09-13T07:28:52.540 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%26 2022-09-13T07:28:52.541 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZQQxzQuTh/IMqx4Bx45BOpTX5A= 2022-09-13T07:28:52.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZQQxzQuTh/IMqx4Bx45BOpTX5A=', 'Content-Length': '0'} 2022-09-13T07:28:52.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3b9c531439ebe79-0063203134-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.544 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:52.545 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:52.545 DEBUG:boto:Method: HEAD 2022-09-13T07:28:52.546 DEBUG:boto:Path: /yxvvuh-12/%26 2022-09-13T07:28:52.546 DEBUG:boto:Data: 2022-09-13T07:28:52.547 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.547 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.548 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.548 DEBUG:boto:Params: {} 2022-09-13T07:28:52.549 DEBUG:boto:Token: None 2022-09-13T07:28:52.549 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%26 2022-09-13T07:28:52.550 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZQQxzQuTh/IMqx4Bx45BOpTX5A= 2022-09-13T07:28:52.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZQQxzQuTh/IMqx4Bx45BOpTX5A=', 'Content-Length': '0'} 2022-09-13T07:28:52.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae03483cfaecd850-0063203134-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:52.554 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:52.554 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:52.555 DEBUG:boto:Method: GET 2022-09-13T07:28:52.555 DEBUG:boto:Path: /yxvvuh-12/%26 2022-09-13T07:28:52.556 DEBUG:boto:Data: 2022-09-13T07:28:52.556 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.557 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.557 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.558 DEBUG:boto:Params: {} 2022-09-13T07:28:52.559 DEBUG:boto:Token: None 2022-09-13T07:28:52.559 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%26 2022-09-13T07:28:52.560 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bu2KG+w9E2NbtmCiUtDkPkeWJeo= 2022-09-13T07:28:52.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bu2KG+w9E2NbtmCiUtDkPkeWJeo=', 'Content-Length': '0'} 2022-09-13T07:28:52.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000165a1ac311bdddf1-0063203134-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.563 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:52.564 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:52.564 DEBUG:boto:Method: GET 2022-09-13T07:28:52.565 DEBUG:boto:Path: /yxvvuh-12/%26 2022-09-13T07:28:52.565 DEBUG:boto:Data: 2022-09-13T07:28:52.566 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.566 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.567 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.567 DEBUG:boto:Params: {} 2022-09-13T07:28:52.568 DEBUG:boto:Token: None 2022-09-13T07:28:52.569 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%26 2022-09-13T07:28:52.569 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bu2KG+w9E2NbtmCiUtDkPkeWJeo= 2022-09-13T07:28:52.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bu2KG+w9E2NbtmCiUtDkPkeWJeo=', 'Content-Length': '0'} 2022-09-13T07:28:52.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028824aa461dc6de6-0063203134-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.575 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.576 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.576 DEBUG:boto:path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:52.577 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:52.577 DEBUG:boto:Method: GET 2022-09-13T07:28:52.578 DEBUG:boto:Path: /yxvvuh-12/?prefix=%26 2022-09-13T07:28:52.578 DEBUG:boto:Data: 2022-09-13T07:28:52.579 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.580 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.580 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.581 DEBUG:boto:Params: {} 2022-09-13T07:28:52.581 DEBUG:boto:Token: None 2022-09-13T07:28:52.582 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/ 2022-09-13T07:28:52.583 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY= 2022-09-13T07:28:52.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY=', 'Content-Length': '0'} 2022-09-13T07:28:52.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fff993dfb4d43773-0063203134-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.586 DEBUG:boto:b'yxvvuh-12&1000false&2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.588 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.588 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.589 DEBUG:boto:path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:52.590 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:52.590 DEBUG:boto:Method: GET 2022-09-13T07:28:52.591 DEBUG:boto:Path: /yxvvuh-12/?prefix=%26 2022-09-13T07:28:52.591 DEBUG:boto:Data: 2022-09-13T07:28:52.592 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.592 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.593 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.593 DEBUG:boto:Params: {} 2022-09-13T07:28:52.594 DEBUG:boto:Token: None 2022-09-13T07:28:52.594 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/ 2022-09-13T07:28:52.595 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY= 2022-09-13T07:28:52.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY=', 'Content-Length': '0'} 2022-09-13T07:28:52.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aadd9080c522e1a-0063203134-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.599 DEBUG:boto:b'yxvvuh-12&1000false&2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:52.600 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:52.601 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:52.601 DEBUG:boto:path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.602 DEBUG:boto:auth_path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.602 DEBUG:boto:Method: GET 2022-09-13T07:28:52.603 DEBUG:boto:Path: /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.603 DEBUG:boto:Data: 2022-09-13T07:28:52.604 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.605 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.605 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.607 DEBUG:boto:Params: {} 2022-09-13T07:28:52.608 DEBUG:boto:Token: None 2022-09-13T07:28:52.610 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.611 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:C4iuge4X3Ta+qkLHqChHugI19vQ= 2022-09-13T07:28:52.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:C4iuge4X3Ta+qkLHqChHugI19vQ=', 'Content-Length': '0'} 2022-09-13T07:28:52.614 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001da3912844900f42-0063203134-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.616 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:52.616 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:52.617 DEBUG:boto:path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.618 DEBUG:boto:auth_path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.622 DEBUG:boto:Method: GET 2022-09-13T07:28:52.623 DEBUG:boto:Path: /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.623 DEBUG:boto:Data: 2022-09-13T07:28:52.624 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.625 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.625 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.626 DEBUG:boto:Params: {} 2022-09-13T07:28:52.627 DEBUG:boto:Token: None 2022-09-13T07:28:52.627 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.628 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:C4iuge4X3Ta+qkLHqChHugI19vQ= 2022-09-13T07:28:52.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:C4iuge4X3Ta+qkLHqChHugI19vQ=', 'Content-Length': '0'} 2022-09-13T07:28:52.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080bbe0225a027f6a-0063203134-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.683 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:52.684 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:52.684 DEBUG:boto:path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.685 DEBUG:boto:auth_path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.685 DEBUG:boto:Method: HEAD 2022-09-13T07:28:52.686 DEBUG:boto:Path: /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.687 DEBUG:boto:Data: 2022-09-13T07:28:52.687 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.688 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.688 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.689 DEBUG:boto:Params: {} 2022-09-13T07:28:52.689 DEBUG:boto:Token: None 2022-09-13T07:28:52.690 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.690 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hWmyioJqBqWDDWjfd5NLbAK8oSY= 2022-09-13T07:28:52.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hWmyioJqBqWDDWjfd5NLbAK8oSY=', 'Content-Length': '0'} 2022-09-13T07:28:52.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022c07d240bae3f59-0063203134-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.694 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:52.694 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:52.695 DEBUG:boto:path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.695 DEBUG:boto:auth_path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.696 DEBUG:boto:Method: HEAD 2022-09-13T07:28:52.696 DEBUG:boto:Path: /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.697 DEBUG:boto:Data: 2022-09-13T07:28:52.697 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.698 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.699 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.699 DEBUG:boto:Params: {} 2022-09-13T07:28:52.700 DEBUG:boto:Token: None 2022-09-13T07:28:52.700 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.701 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hWmyioJqBqWDDWjfd5NLbAK8oSY= 2022-09-13T07:28:52.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hWmyioJqBqWDDWjfd5NLbAK8oSY=', 'Content-Length': '0'} 2022-09-13T07:28:52.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3e71ef2c5a9178f-0063203134-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:52.706 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:52.707 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:52.708 DEBUG:boto:path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.709 DEBUG:boto:auth_path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.710 DEBUG:boto:Method: GET 2022-09-13T07:28:52.711 DEBUG:boto:Path: /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.712 DEBUG:boto:Data: 2022-09-13T07:28:52.713 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.714 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.715 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.717 DEBUG:boto:Params: {} 2022-09-13T07:28:52.718 DEBUG:boto:Token: None 2022-09-13T07:28:52.719 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.720 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:C4iuge4X3Ta+qkLHqChHugI19vQ= 2022-09-13T07:28:52.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:C4iuge4X3Ta+qkLHqChHugI19vQ=', 'Content-Length': '0'} 2022-09-13T07:28:52.724 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000009d9cefd4de2e02-0063203134-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.726 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:52.727 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:52.728 DEBUG:boto:path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.729 DEBUG:boto:auth_path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.730 DEBUG:boto:Method: GET 2022-09-13T07:28:52.731 DEBUG:boto:Path: /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.732 DEBUG:boto:Data: 2022-09-13T07:28:52.733 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.737 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.738 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.739 DEBUG:boto:Params: {} 2022-09-13T07:28:52.740 DEBUG:boto:Token: None 2022-09-13T07:28:52.741 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:52.741 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:C4iuge4X3Ta+qkLHqChHugI19vQ= 2022-09-13T07:28:52.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:C4iuge4X3Ta+qkLHqChHugI19vQ=', 'Content-Length': '0'} 2022-09-13T07:28:52.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed92108ceab4b594-0063203134-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.745 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.746 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.746 DEBUG:boto:path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:52.747 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:52.747 DEBUG:boto:Method: GET 2022-09-13T07:28:52.748 DEBUG:boto:Path: /yxvvuh-12/?prefix=%3A 2022-09-13T07:28:52.748 DEBUG:boto:Data: 2022-09-13T07:28:52.749 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.749 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.750 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.751 DEBUG:boto:Params: {} 2022-09-13T07:28:52.751 DEBUG:boto:Token: None 2022-09-13T07:28:52.752 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/ 2022-09-13T07:28:52.753 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY= 2022-09-13T07:28:52.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY=', 'Content-Length': '0'} 2022-09-13T07:28:52.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e604d933788fe176-0063203134-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.757 DEBUG:boto:b'yxvvuh-12:1000false:2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.758 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.758 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.759 DEBUG:boto:path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:52.759 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:52.760 DEBUG:boto:Method: GET 2022-09-13T07:28:52.760 DEBUG:boto:Path: /yxvvuh-12/?prefix=%3A 2022-09-13T07:28:52.761 DEBUG:boto:Data: 2022-09-13T07:28:52.762 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.762 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.763 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.763 DEBUG:boto:Params: {} 2022-09-13T07:28:52.764 DEBUG:boto:Token: None 2022-09-13T07:28:52.765 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/ 2022-09-13T07:28:52.765 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY= 2022-09-13T07:28:52.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY=', 'Content-Length': '0'} 2022-09-13T07:28:52.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002299ab5f231dcd3f-0063203134-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.769 DEBUG:boto:b'yxvvuh-12:1000false:2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.770 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:52.771 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:52.771 DEBUG:boto:Method: HEAD 2022-09-13T07:28:52.772 DEBUG:boto:Path: /yxvvuh-12/%3A 2022-09-13T07:28:52.772 DEBUG:boto:Data: 2022-09-13T07:28:52.773 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.773 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.774 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.775 DEBUG:boto:Params: {} 2022-09-13T07:28:52.775 DEBUG:boto:Token: None 2022-09-13T07:28:52.776 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%3A 2022-09-13T07:28:52.776 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BxBLtdXj9ZR8GQim0MePF6MPWWc= 2022-09-13T07:28:52.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BxBLtdXj9ZR8GQim0MePF6MPWWc=', 'Content-Length': '0'} 2022-09-13T07:28:52.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001940a580e16135d7-0063203134-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.780 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:52.780 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:52.781 DEBUG:boto:Method: HEAD 2022-09-13T07:28:52.781 DEBUG:boto:Path: /yxvvuh-12/%3A 2022-09-13T07:28:52.782 DEBUG:boto:Data: 2022-09-13T07:28:52.783 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.784 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.785 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.786 DEBUG:boto:Params: {} 2022-09-13T07:28:52.787 DEBUG:boto:Token: None 2022-09-13T07:28:52.788 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%3A 2022-09-13T07:28:52.789 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BxBLtdXj9ZR8GQim0MePF6MPWWc= 2022-09-13T07:28:52.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BxBLtdXj9ZR8GQim0MePF6MPWWc=', 'Content-Length': '0'} 2022-09-13T07:28:52.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000314242c3257091-0063203134-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:52.795 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:52.796 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:52.797 DEBUG:boto:Method: GET 2022-09-13T07:28:52.798 DEBUG:boto:Path: /yxvvuh-12/%3A 2022-09-13T07:28:52.799 DEBUG:boto:Data: 2022-09-13T07:28:52.799 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.800 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.801 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.804 DEBUG:boto:Params: {} 2022-09-13T07:28:52.805 DEBUG:boto:Token: None 2022-09-13T07:28:52.806 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%3A 2022-09-13T07:28:52.807 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:buFRXnyKg6QXuHquI1beBo3NSf8= 2022-09-13T07:28:52.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:buFRXnyKg6QXuHquI1beBo3NSf8=', 'Content-Length': '0'} 2022-09-13T07:28:52.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e63f83dc8ed9b18c-0063203134-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.812 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:52.814 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:52.814 DEBUG:boto:Method: GET 2022-09-13T07:28:52.815 DEBUG:boto:Path: /yxvvuh-12/%3A 2022-09-13T07:28:52.816 DEBUG:boto:Data: 2022-09-13T07:28:52.817 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.818 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.819 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.820 DEBUG:boto:Params: {} 2022-09-13T07:28:52.821 DEBUG:boto:Token: None 2022-09-13T07:28:52.822 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/%3A 2022-09-13T07:28:52.823 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:buFRXnyKg6QXuHquI1beBo3NSf8= 2022-09-13T07:28:52.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:buFRXnyKg6QXuHquI1beBo3NSf8=', 'Content-Length': '0'} 2022-09-13T07:28:52.827 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd1064a6aa41ae75-0063203134-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.828 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.829 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.830 DEBUG:boto:path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:52.831 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:52.832 DEBUG:boto:Method: GET 2022-09-13T07:28:52.833 DEBUG:boto:Path: /yxvvuh-12/?prefix=%3A 2022-09-13T07:28:52.834 DEBUG:boto:Data: 2022-09-13T07:28:52.835 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.836 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.837 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.839 DEBUG:boto:Params: {} 2022-09-13T07:28:52.840 DEBUG:boto:Token: None 2022-09-13T07:28:52.841 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/ 2022-09-13T07:28:52.842 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY= 2022-09-13T07:28:52.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY=', 'Content-Length': '0'} 2022-09-13T07:28:52.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000300d9d4f5b6fdc11-0063203134-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.891 DEBUG:boto:b'yxvvuh-12:1000false:2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.895 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.895 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.896 DEBUG:boto:path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:52.897 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:52.897 DEBUG:boto:Method: GET 2022-09-13T07:28:52.898 DEBUG:boto:Path: /yxvvuh-12/?prefix=%3A 2022-09-13T07:28:52.898 DEBUG:boto:Data: 2022-09-13T07:28:52.899 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.899 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.900 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.900 DEBUG:boto:Params: {} 2022-09-13T07:28:52.901 DEBUG:boto:Token: None 2022-09-13T07:28:52.901 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/ 2022-09-13T07:28:52.902 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY= 2022-09-13T07:28:52.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY=', 'Content-Length': '0'} 2022-09-13T07:28:52.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000258acddce178010a-0063203134-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.906 DEBUG:boto:b'yxvvuh-12:1000false:2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:52.907 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:52.908 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:52.908 DEBUG:boto:path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.909 DEBUG:boto:auth_path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.909 DEBUG:boto:Method: GET 2022-09-13T07:28:52.910 DEBUG:boto:Path: /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.910 DEBUG:boto:Data: 2022-09-13T07:28:52.911 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.911 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.912 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.912 DEBUG:boto:Params: {} 2022-09-13T07:28:52.913 DEBUG:boto:Token: None 2022-09-13T07:28:52.914 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.914 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7tVKYZdofO5ObsufmcXDQDvoEuc= 2022-09-13T07:28:52.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7tVKYZdofO5ObsufmcXDQDvoEuc=', 'Content-Length': '0'} 2022-09-13T07:28:52.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c75453c3c566504-0063203134-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.919 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:52.920 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:52.921 DEBUG:boto:path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.922 DEBUG:boto:auth_path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.923 DEBUG:boto:Method: GET 2022-09-13T07:28:52.924 DEBUG:boto:Path: /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.924 DEBUG:boto:Data: 2022-09-13T07:28:52.925 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.925 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.926 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.926 DEBUG:boto:Params: {} 2022-09-13T07:28:52.927 DEBUG:boto:Token: None 2022-09-13T07:28:52.928 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.928 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7tVKYZdofO5ObsufmcXDQDvoEuc= 2022-09-13T07:28:52.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7tVKYZdofO5ObsufmcXDQDvoEuc=', 'Content-Length': '0'} 2022-09-13T07:28:52.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007582ef0ef82f881d-0063203134-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.932 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:52.932 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:52.933 DEBUG:boto:path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.933 DEBUG:boto:auth_path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.934 DEBUG:boto:Method: HEAD 2022-09-13T07:28:52.934 DEBUG:boto:Path: /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.935 DEBUG:boto:Data: 2022-09-13T07:28:52.935 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.936 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.936 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.937 DEBUG:boto:Params: {} 2022-09-13T07:28:52.937 DEBUG:boto:Token: None 2022-09-13T07:28:52.938 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.938 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:f/WU4CrzCZwH9+wihGq1CcWMC/Q= 2022-09-13T07:28:52.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:f/WU4CrzCZwH9+wihGq1CcWMC/Q=', 'Content-Length': '0'} 2022-09-13T07:28:52.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000132ae4a37d661712-0063203134-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.941 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:52.942 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:52.943 DEBUG:boto:path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.943 DEBUG:boto:auth_path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.943 DEBUG:boto:Method: HEAD 2022-09-13T07:28:52.944 DEBUG:boto:Path: /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.944 DEBUG:boto:Data: 2022-09-13T07:28:52.945 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.945 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.946 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.946 DEBUG:boto:Params: {} 2022-09-13T07:28:52.947 DEBUG:boto:Token: None 2022-09-13T07:28:52.948 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.948 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:f/WU4CrzCZwH9+wihGq1CcWMC/Q= 2022-09-13T07:28:52.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:f/WU4CrzCZwH9+wihGq1CcWMC/Q=', 'Content-Length': '0'} 2022-09-13T07:28:52.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006792b7ee8489e8ec-0063203134-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:52.952 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:52.952 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:52.953 DEBUG:boto:path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.953 DEBUG:boto:auth_path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.954 DEBUG:boto:Method: GET 2022-09-13T07:28:52.955 DEBUG:boto:Path: /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.955 DEBUG:boto:Data: 2022-09-13T07:28:52.956 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.956 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.957 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.957 DEBUG:boto:Params: {} 2022-09-13T07:28:52.958 DEBUG:boto:Token: None 2022-09-13T07:28:52.958 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.959 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7tVKYZdofO5ObsufmcXDQDvoEuc= 2022-09-13T07:28:52.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7tVKYZdofO5ObsufmcXDQDvoEuc=', 'Content-Length': '0'} 2022-09-13T07:28:52.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdeb7af565fffb76-0063203134-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.962 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:52.963 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:52.963 DEBUG:boto:path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.964 DEBUG:boto:auth_path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.964 DEBUG:boto:Method: GET 2022-09-13T07:28:52.965 DEBUG:boto:Path: /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.965 DEBUG:boto:Data: 2022-09-13T07:28:52.966 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.966 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.967 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.968 DEBUG:boto:Params: {} 2022-09-13T07:28:52.968 DEBUG:boto:Token: None 2022-09-13T07:28:52.969 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:52.969 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7tVKYZdofO5ObsufmcXDQDvoEuc= 2022-09-13T07:28:52.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7tVKYZdofO5ObsufmcXDQDvoEuc=', 'Content-Length': '0'} 2022-09-13T07:28:52.972 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064757c7428a704d1-0063203134-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.973 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.974 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.974 DEBUG:boto:path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:52.975 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:52.975 DEBUG:boto:Method: GET 2022-09-13T07:28:52.976 DEBUG:boto:Path: /yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:52.976 DEBUG:boto:Data: 2022-09-13T07:28:52.977 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.977 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:52.978 DEBUG:boto:Port: 8000 2022-09-13T07:28:52.978 DEBUG:boto:Params: {} 2022-09-13T07:28:52.979 DEBUG:boto:Token: None 2022-09-13T07:28:52.979 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/ 2022-09-13T07:28:52.980 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY= 2022-09-13T07:28:52.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY=', 'Content-Length': '0'} 2022-09-13T07:28:52.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002712a469d589008-0063203134-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.984 DEBUG:boto:b'yxvvuh-12_myobj1000false_myobj2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.984 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:52.985 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:52.986 DEBUG:boto:path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:52.986 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:52.987 DEBUG:boto:Method: GET 2022-09-13T07:28:52.987 DEBUG:boto:Path: /yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:52.988 DEBUG:boto:Data: 2022-09-13T07:28:52.988 DEBUG:boto:Headers: {} 2022-09-13T07:28:52.989 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:52.989 DEBUG:boto:Port: 8001 2022-09-13T07:28:52.990 DEBUG:boto:Params: {} 2022-09-13T07:28:52.991 DEBUG:boto:Token: None 2022-09-13T07:28:52.991 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:52 GMT /yxvvuh-12/ 2022-09-13T07:28:52.992 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY= 2022-09-13T07:28:52.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vPzrGfrM9F812/n4wrLV07SeImY=', 'Content-Length': '0'} 2022-09-13T07:28:52.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6337e55e48afce1-0063203134-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:52.996 DEBUG:boto:b'yxvvuh-12_myobj1000false_myobj2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:52.996 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:52.997 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:52.997 DEBUG:boto:Method: HEAD 2022-09-13T07:28:52.998 DEBUG:boto:Path: /yxvvuh-12/_myobj 2022-09-13T07:28:52.999 DEBUG:boto:Data: 2022-09-13T07:28:52.999 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.000 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:53.000 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.001 DEBUG:boto:Params: {} 2022-09-13T07:28:53.001 DEBUG:boto:Token: None 2022-09-13T07:28:53.002 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/_myobj 2022-09-13T07:28:53.002 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6d/ul8m6tFM/FCBGQopTVRMODvU= 2022-09-13T07:28:53.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6d/ul8m6tFM/FCBGQopTVRMODvU=', 'Content-Length': '0'} 2022-09-13T07:28:53.005 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0533215eab132f2-0063203135-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.005 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:53.006 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:53.006 DEBUG:boto:Method: HEAD 2022-09-13T07:28:53.007 DEBUG:boto:Path: /yxvvuh-12/_myobj 2022-09-13T07:28:53.007 DEBUG:boto:Data: 2022-09-13T07:28:53.008 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.008 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.009 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.009 DEBUG:boto:Params: {} 2022-09-13T07:28:53.010 DEBUG:boto:Token: None 2022-09-13T07:28:53.011 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/_myobj 2022-09-13T07:28:53.011 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6d/ul8m6tFM/FCBGQopTVRMODvU= 2022-09-13T07:28:53.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6d/ul8m6tFM/FCBGQopTVRMODvU=', 'Content-Length': '0'} 2022-09-13T07:28:53.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000942c0003d8bc70af-0063203135-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:53.015 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:53.015 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:53.016 DEBUG:boto:Method: GET 2022-09-13T07:28:53.017 DEBUG:boto:Path: /yxvvuh-12/_myobj 2022-09-13T07:28:53.017 DEBUG:boto:Data: 2022-09-13T07:28:53.018 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.019 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:53.020 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.021 DEBUG:boto:Params: {} 2022-09-13T07:28:53.022 DEBUG:boto:Token: None 2022-09-13T07:28:53.023 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/_myobj 2022-09-13T07:28:53.024 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vj5XwTQWooTJJas90TxTZIicG3o= 2022-09-13T07:28:53.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vj5XwTQWooTJJas90TxTZIicG3o=', 'Content-Length': '0'} 2022-09-13T07:28:53.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8f21fec9c56f087-0063203135-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.030 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:53.031 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:53.032 DEBUG:boto:Method: GET 2022-09-13T07:28:53.033 DEBUG:boto:Path: /yxvvuh-12/_myobj 2022-09-13T07:28:53.034 DEBUG:boto:Data: 2022-09-13T07:28:53.035 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.036 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.037 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.037 DEBUG:boto:Params: {} 2022-09-13T07:28:53.038 DEBUG:boto:Token: None 2022-09-13T07:28:53.039 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/_myobj 2022-09-13T07:28:53.039 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vj5XwTQWooTJJas90TxTZIicG3o= 2022-09-13T07:28:53.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vj5XwTQWooTJJas90TxTZIicG3o=', 'Content-Length': '0'} 2022-09-13T07:28:53.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000013d7b5f51e78961-0063203135-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.086 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.087 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.088 DEBUG:boto:path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:53.088 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:53.089 DEBUG:boto:Method: GET 2022-09-13T07:28:53.089 DEBUG:boto:Path: /yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:53.090 DEBUG:boto:Data: 2022-09-13T07:28:53.090 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.091 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:53.091 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.092 DEBUG:boto:Params: {} 2022-09-13T07:28:53.092 DEBUG:boto:Token: None 2022-09-13T07:28:53.093 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/ 2022-09-13T07:28:53.094 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0= 2022-09-13T07:28:53.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0=', 'Content-Length': '0'} 2022-09-13T07:28:53.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5d5bf88d0ceee3f-0063203135-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.098 DEBUG:boto:b'yxvvuh-12_myobj1000false_myobj2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.098 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.099 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.099 DEBUG:boto:path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:53.100 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:53.101 DEBUG:boto:Method: GET 2022-09-13T07:28:53.101 DEBUG:boto:Path: /yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:53.102 DEBUG:boto:Data: 2022-09-13T07:28:53.103 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.103 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.104 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.104 DEBUG:boto:Params: {} 2022-09-13T07:28:53.105 DEBUG:boto:Token: None 2022-09-13T07:28:53.105 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/ 2022-09-13T07:28:53.106 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0= 2022-09-13T07:28:53.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0=', 'Content-Length': '0'} 2022-09-13T07:28:53.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064679cd90b3810f5-0063203135-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.111 DEBUG:boto:b'yxvvuh-12_myobj1000false_myobj2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:53.113 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:53.114 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:53.114 DEBUG:boto:path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.115 DEBUG:boto:auth_path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.115 DEBUG:boto:Method: GET 2022-09-13T07:28:53.116 DEBUG:boto:Path: /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.117 DEBUG:boto:Data: 2022-09-13T07:28:53.118 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.118 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:53.119 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.121 DEBUG:boto:Params: {} 2022-09-13T07:28:53.122 DEBUG:boto:Token: None 2022-09-13T07:28:53.122 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.123 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MiXdaUr3BnQ5PkpSRhxPuTZNqzg= 2022-09-13T07:28:53.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MiXdaUr3BnQ5PkpSRhxPuTZNqzg=', 'Content-Length': '0'} 2022-09-13T07:28:53.125 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c69f0794cafc98b-0063203135-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.136 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:53.137 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:53.138 DEBUG:boto:path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.138 DEBUG:boto:auth_path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.139 DEBUG:boto:Method: GET 2022-09-13T07:28:53.140 DEBUG:boto:Path: /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.140 DEBUG:boto:Data: 2022-09-13T07:28:53.141 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.141 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.142 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.143 DEBUG:boto:Params: {} 2022-09-13T07:28:53.143 DEBUG:boto:Token: None 2022-09-13T07:28:53.144 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.144 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MiXdaUr3BnQ5PkpSRhxPuTZNqzg= 2022-09-13T07:28:53.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MiXdaUr3BnQ5PkpSRhxPuTZNqzg=', 'Content-Length': '0'} 2022-09-13T07:28:53.147 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004acab68b5a037a23-0063203135-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.148 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:53.149 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:53.149 DEBUG:boto:path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.150 DEBUG:boto:auth_path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.150 DEBUG:boto:Method: HEAD 2022-09-13T07:28:53.151 DEBUG:boto:Path: /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.151 DEBUG:boto:Data: 2022-09-13T07:28:53.152 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.152 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:53.153 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.153 DEBUG:boto:Params: {} 2022-09-13T07:28:53.154 DEBUG:boto:Token: None 2022-09-13T07:28:53.154 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.155 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pn3urdtw4TKjGmAh80l8lzaLw2E= 2022-09-13T07:28:53.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pn3urdtw4TKjGmAh80l8lzaLw2E=', 'Content-Length': '0'} 2022-09-13T07:28:53.158 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000437de7bceffbbdcf-0063203135-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.158 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:53.159 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:53.159 DEBUG:boto:path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.160 DEBUG:boto:auth_path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.160 DEBUG:boto:Method: HEAD 2022-09-13T07:28:53.161 DEBUG:boto:Path: /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.161 DEBUG:boto:Data: 2022-09-13T07:28:53.162 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.163 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.163 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.164 DEBUG:boto:Params: {} 2022-09-13T07:28:53.166 DEBUG:boto:Token: None 2022-09-13T07:28:53.167 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.168 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pn3urdtw4TKjGmAh80l8lzaLw2E= 2022-09-13T07:28:53.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pn3urdtw4TKjGmAh80l8lzaLw2E=', 'Content-Length': '0'} 2022-09-13T07:28:53.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002eebfa32d13f1f2b-0063203135-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:53.172 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:53.173 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:53.173 DEBUG:boto:path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.174 DEBUG:boto:auth_path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.174 DEBUG:boto:Method: GET 2022-09-13T07:28:53.175 DEBUG:boto:Path: /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.176 DEBUG:boto:Data: 2022-09-13T07:28:53.176 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.177 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:53.177 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.178 DEBUG:boto:Params: {} 2022-09-13T07:28:53.178 DEBUG:boto:Token: None 2022-09-13T07:28:53.179 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.179 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MiXdaUr3BnQ5PkpSRhxPuTZNqzg= 2022-09-13T07:28:53.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MiXdaUr3BnQ5PkpSRhxPuTZNqzg=', 'Content-Length': '0'} 2022-09-13T07:28:53.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e834b38d1720ce13-0063203135-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.230 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:53.231 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:53.232 DEBUG:boto:path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.233 DEBUG:boto:auth_path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.234 DEBUG:boto:Method: GET 2022-09-13T07:28:53.235 DEBUG:boto:Path: /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.236 DEBUG:boto:Data: 2022-09-13T07:28:53.237 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.238 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.239 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.240 DEBUG:boto:Params: {} 2022-09-13T07:28:53.241 DEBUG:boto:Token: None 2022-09-13T07:28:53.242 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:53.243 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MiXdaUr3BnQ5PkpSRhxPuTZNqzg= 2022-09-13T07:28:53.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MiXdaUr3BnQ5PkpSRhxPuTZNqzg=', 'Content-Length': '0'} 2022-09-13T07:28:53.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071628179aebc562a-0063203135-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.249 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.250 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.251 DEBUG:boto:path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:53.252 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:53.253 DEBUG:boto:Method: GET 2022-09-13T07:28:53.254 DEBUG:boto:Path: /yxvvuh-12/?prefix=myobj 2022-09-13T07:28:53.255 DEBUG:boto:Data: 2022-09-13T07:28:53.256 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.257 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:53.258 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.259 DEBUG:boto:Params: {} 2022-09-13T07:28:53.260 DEBUG:boto:Token: None 2022-09-13T07:28:53.261 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/ 2022-09-13T07:28:53.262 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0= 2022-09-13T07:28:53.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0=', 'Content-Length': '0'} 2022-09-13T07:28:53.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000466b15b82712be03-0063203135-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.267 DEBUG:boto:b'yxvvuh-12myobj1000falsemyobj2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.269 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.270 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.271 DEBUG:boto:path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:53.272 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:53.273 DEBUG:boto:Method: GET 2022-09-13T07:28:53.274 DEBUG:boto:Path: /yxvvuh-12/?prefix=myobj 2022-09-13T07:28:53.275 DEBUG:boto:Data: 2022-09-13T07:28:53.276 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.277 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.278 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.279 DEBUG:boto:Params: {} 2022-09-13T07:28:53.280 DEBUG:boto:Token: None 2022-09-13T07:28:53.281 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/ 2022-09-13T07:28:53.282 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0= 2022-09-13T07:28:53.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0=', 'Content-Length': '0'} 2022-09-13T07:28:53.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000787228b73bd76f55-0063203135-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.287 DEBUG:boto:b'yxvvuh-12myobj1000falsemyobj2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.288 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:53.288 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:53.289 DEBUG:boto:Method: HEAD 2022-09-13T07:28:53.289 DEBUG:boto:Path: /yxvvuh-12/myobj 2022-09-13T07:28:53.290 DEBUG:boto:Data: 2022-09-13T07:28:53.290 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.291 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:53.291 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.292 DEBUG:boto:Params: {} 2022-09-13T07:28:53.293 DEBUG:boto:Token: None 2022-09-13T07:28:53.293 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/myobj 2022-09-13T07:28:53.294 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5eY0AKvBHY8vaFTwU0Vwl7SplV0= 2022-09-13T07:28:53.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5eY0AKvBHY8vaFTwU0Vwl7SplV0=', 'Content-Length': '0'} 2022-09-13T07:28:53.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000512d4cf9a57f955-0063203135-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.297 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:53.298 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:53.298 DEBUG:boto:Method: HEAD 2022-09-13T07:28:53.299 DEBUG:boto:Path: /yxvvuh-12/myobj 2022-09-13T07:28:53.300 DEBUG:boto:Data: 2022-09-13T07:28:53.301 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.301 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.302 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.303 DEBUG:boto:Params: {} 2022-09-13T07:28:53.304 DEBUG:boto:Token: None 2022-09-13T07:28:53.305 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/myobj 2022-09-13T07:28:53.306 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5eY0AKvBHY8vaFTwU0Vwl7SplV0= 2022-09-13T07:28:53.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5eY0AKvBHY8vaFTwU0Vwl7SplV0=', 'Content-Length': '0'} 2022-09-13T07:28:53.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017f0a4878fc13fdf-0063203135-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:53.312 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:53.313 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:53.314 DEBUG:boto:Method: GET 2022-09-13T07:28:53.315 DEBUG:boto:Path: /yxvvuh-12/myobj 2022-09-13T07:28:53.315 DEBUG:boto:Data: 2022-09-13T07:28:53.316 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.317 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:53.318 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.318 DEBUG:boto:Params: {} 2022-09-13T07:28:53.319 DEBUG:boto:Token: None 2022-09-13T07:28:53.320 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/myobj 2022-09-13T07:28:53.320 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:526SUTSc1a2BecrK8X4IyU3fEd8= 2022-09-13T07:28:53.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:526SUTSc1a2BecrK8X4IyU3fEd8=', 'Content-Length': '0'} 2022-09-13T07:28:53.323 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a61d271aef994da-0063203135-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.324 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:53.324 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:53.325 DEBUG:boto:Method: GET 2022-09-13T07:28:53.326 DEBUG:boto:Path: /yxvvuh-12/myobj 2022-09-13T07:28:53.326 DEBUG:boto:Data: 2022-09-13T07:28:53.327 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.327 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.328 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.328 DEBUG:boto:Params: {} 2022-09-13T07:28:53.329 DEBUG:boto:Token: None 2022-09-13T07:28:53.330 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/myobj 2022-09-13T07:28:53.330 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:526SUTSc1a2BecrK8X4IyU3fEd8= 2022-09-13T07:28:53.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:526SUTSc1a2BecrK8X4IyU3fEd8=', 'Content-Length': '0'} 2022-09-13T07:28:53.333 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a89b619a3fdebea6-0063203135-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.374 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.375 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.375 DEBUG:boto:path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:53.376 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:53.376 DEBUG:boto:Method: GET 2022-09-13T07:28:53.377 DEBUG:boto:Path: /yxvvuh-12/?prefix=myobj 2022-09-13T07:28:53.378 DEBUG:boto:Data: 2022-09-13T07:28:53.380 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.380 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:53.381 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.381 DEBUG:boto:Params: {} 2022-09-13T07:28:53.382 DEBUG:boto:Token: None 2022-09-13T07:28:53.382 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/ 2022-09-13T07:28:53.383 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0= 2022-09-13T07:28:53.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0=', 'Content-Length': '0'} 2022-09-13T07:28:53.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e7797534103429a-0063203135-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.388 DEBUG:boto:b'yxvvuh-12myobj1000falsemyobj2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.389 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.390 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.391 DEBUG:boto:path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:53.391 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:53.392 DEBUG:boto:Method: GET 2022-09-13T07:28:53.392 DEBUG:boto:Path: /yxvvuh-12/?prefix=myobj 2022-09-13T07:28:53.393 DEBUG:boto:Data: 2022-09-13T07:28:53.393 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.394 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.394 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.395 DEBUG:boto:Params: {} 2022-09-13T07:28:53.395 DEBUG:boto:Token: None 2022-09-13T07:28:53.396 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/ 2022-09-13T07:28:53.397 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0= 2022-09-13T07:28:53.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0=', 'Content-Length': '0'} 2022-09-13T07:28:53.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fdf8f16fcda80c4-0063203135-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.407 DEBUG:boto:b'yxvvuh-12myobj1000falsemyobj2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.408 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-12 zones={test-zone1, test-zone3} 2022-09-13T07:28:53.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:28:53.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:28:53.410 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:28:53.475 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:53.476 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:53.514 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:53.510+0000 7f27e5af8600 1 bucket sync caught up with source: 2022-09-13T07:28:53.516 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.27.5, , 00000000001.7.5, , , , 00000000001.8.5, 00000000001.29.5, , ] 2022-09-13T07:28:53.517 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000001.27.5, , 00000000001.7.5, , , , 00000000001.8.5, 00000000001.29.5, , ] 2022-09-13T07:28:53.517 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:53.510+0000 7f27e5af8600 0 bucket checkpoint complete 2022-09-13T07:28:53.527 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-12 zones={test-zone2, test-zone3} 2022-09-13T07:28:53.528 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.529 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.529 DEBUG:boto:Method: HEAD 2022-09-13T07:28:53.530 DEBUG:boto:Path: /yxvvuh-12/ 2022-09-13T07:28:53.530 DEBUG:boto:Data: 2022-09-13T07:28:53.531 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.531 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:53.532 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.532 DEBUG:boto:Params: {} 2022-09-13T07:28:53.533 DEBUG:boto:Token: None 2022-09-13T07:28:53.533 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/ 2022-09-13T07:28:53.534 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:axXEMUjhoGTCFu1sZAtsYEbsDoY= 2022-09-13T07:28:53.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:axXEMUjhoGTCFu1sZAtsYEbsDoY=', 'Content-Length': '0'} 2022-09-13T07:28:53.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7e795c16a96bc62-0063203135-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.538 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.539 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.539 DEBUG:boto:Method: HEAD 2022-09-13T07:28:53.540 DEBUG:boto:Path: /yxvvuh-12/ 2022-09-13T07:28:53.540 DEBUG:boto:Data: 2022-09-13T07:28:53.541 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.541 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.542 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.543 DEBUG:boto:Params: {} 2022-09-13T07:28:53.543 DEBUG:boto:Token: None 2022-09-13T07:28:53.544 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/ 2022-09-13T07:28:53.544 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:axXEMUjhoGTCFu1sZAtsYEbsDoY= 2022-09-13T07:28:53.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:axXEMUjhoGTCFu1sZAtsYEbsDoY=', 'Content-Length': '0'} 2022-09-13T07:28:53.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b701ec6de28d6ab4-0063203135-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.551 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:28:53.552 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.553 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.553 DEBUG:boto:path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:53.554 DEBUG:boto:auth_path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:53.554 DEBUG:boto:Method: GET 2022-09-13T07:28:53.555 DEBUG:boto:Path: /yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:53.555 DEBUG:boto:Data: 2022-09-13T07:28:53.556 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.556 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:53.557 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.557 DEBUG:boto:Params: {} 2022-09-13T07:28:53.558 DEBUG:boto:Token: None 2022-09-13T07:28:53.559 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/?versions 2022-09-13T07:28:53.560 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yJGh4SWSBG/SPSaDvN9SH6zFVpc= 2022-09-13T07:28:53.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yJGh4SWSBG/SPSaDvN9SH6zFVpc=', 'Content-Length': '0'} 2022-09-13T07:28:53.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e180e99726cf8417-0063203135-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.563 DEBUG:boto:b'yxvvuh-12999false&nulltrue2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.564 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-13T07:28:53.565 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-13T07:28:53.565 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-13T07:28:53.566 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:28:53.566 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:28:53.567 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.567 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.568 DEBUG:boto:path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:53.568 DEBUG:boto:auth_path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:53.569 DEBUG:boto:Method: GET 2022-09-13T07:28:53.569 DEBUG:boto:Path: /yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:53.570 DEBUG:boto:Data: 2022-09-13T07:28:53.570 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.571 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.571 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.572 DEBUG:boto:Params: {} 2022-09-13T07:28:53.573 DEBUG:boto:Token: None 2022-09-13T07:28:53.573 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/?versions 2022-09-13T07:28:53.574 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yJGh4SWSBG/SPSaDvN9SH6zFVpc= 2022-09-13T07:28:53.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yJGh4SWSBG/SPSaDvN9SH6zFVpc=', 'Content-Length': '0'} 2022-09-13T07:28:53.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000698730e3af1a40bf-0063203135-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.578 DEBUG:boto:b'yxvvuh-12999false&nulltrue2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.579 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-13T07:28:53.579 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-13T07:28:53.580 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-13T07:28:53.580 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:28:53.581 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.581 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.582 DEBUG:boto:path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:53.582 DEBUG:boto:auth_path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:53.583 DEBUG:boto:Method: GET 2022-09-13T07:28:53.583 DEBUG:boto:Path: /yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:53.584 DEBUG:boto:Data: 2022-09-13T07:28:53.584 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.585 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:53.585 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.586 DEBUG:boto:Params: {} 2022-09-13T07:28:53.587 DEBUG:boto:Token: None 2022-09-13T07:28:53.587 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/?versions 2022-09-13T07:28:53.588 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yJGh4SWSBG/SPSaDvN9SH6zFVpc= 2022-09-13T07:28:53.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yJGh4SWSBG/SPSaDvN9SH6zFVpc=', 'Content-Length': '0'} 2022-09-13T07:28:53.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d825c5e5cf3c0cb-0063203135-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.592 DEBUG:boto:b'yxvvuh-12999false&nulltrue2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.593 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.594 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.594 DEBUG:boto:path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:53.595 DEBUG:boto:auth_path=/yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:53.596 DEBUG:boto:Method: GET 2022-09-13T07:28:53.597 DEBUG:boto:Path: /yxvvuh-12/?versions&max-keys=999 2022-09-13T07:28:53.598 DEBUG:boto:Data: 2022-09-13T07:28:53.599 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.600 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.600 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.601 DEBUG:boto:Params: {} 2022-09-13T07:28:53.602 DEBUG:boto:Token: None 2022-09-13T07:28:53.603 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/?versions 2022-09-13T07:28:53.604 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yJGh4SWSBG/SPSaDvN9SH6zFVpc= 2022-09-13T07:28:53.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yJGh4SWSBG/SPSaDvN9SH6zFVpc=', 'Content-Length': '0'} 2022-09-13T07:28:53.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000443c6993237e0a14-0063203135-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.609 DEBUG:boto:b'yxvvuh-12999false&nulltrue2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:53.612 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:53.612 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:53.613 DEBUG:boto:path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.613 DEBUG:boto:auth_path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.614 DEBUG:boto:Method: GET 2022-09-13T07:28:53.615 DEBUG:boto:Path: /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.615 DEBUG:boto:Data: 2022-09-13T07:28:53.616 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.616 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:53.617 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.617 DEBUG:boto:Params: {} 2022-09-13T07:28:53.618 DEBUG:boto:Token: None 2022-09-13T07:28:53.619 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.619 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5oxpXek2Bun8cfkvJyIM4xfphxg= 2022-09-13T07:28:53.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5oxpXek2Bun8cfkvJyIM4xfphxg=', 'Content-Length': '0'} 2022-09-13T07:28:53.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c4fea4493fd72d3-0063203135-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.666 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:53.667 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:53.668 DEBUG:boto:path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.668 DEBUG:boto:auth_path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.669 DEBUG:boto:Method: GET 2022-09-13T07:28:53.669 DEBUG:boto:Path: /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.670 DEBUG:boto:Data: 2022-09-13T07:28:53.671 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.671 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.672 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.672 DEBUG:boto:Params: {} 2022-09-13T07:28:53.673 DEBUG:boto:Token: None 2022-09-13T07:28:53.673 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.674 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5oxpXek2Bun8cfkvJyIM4xfphxg= 2022-09-13T07:28:53.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5oxpXek2Bun8cfkvJyIM4xfphxg=', 'Content-Length': '0'} 2022-09-13T07:28:53.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000265d137eb7e208b6-0063203135-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.678 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:53.679 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:53.679 DEBUG:boto:path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.680 DEBUG:boto:auth_path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.680 DEBUG:boto:Method: HEAD 2022-09-13T07:28:53.681 DEBUG:boto:Path: /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.681 DEBUG:boto:Data: 2022-09-13T07:28:53.682 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.682 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:53.683 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.683 DEBUG:boto:Params: {} 2022-09-13T07:28:53.684 DEBUG:boto:Token: None 2022-09-13T07:28:53.684 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.685 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0V7Z1eot3JIc9K95xUUZIHwkBUw= 2022-09-13T07:28:53.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0V7Z1eot3JIc9K95xUUZIHwkBUw=', 'Content-Length': '0'} 2022-09-13T07:28:53.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e3bcd8a37796553-0063203135-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.689 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:53.690 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:53.691 DEBUG:boto:path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.692 DEBUG:boto:auth_path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.692 DEBUG:boto:Method: HEAD 2022-09-13T07:28:53.693 DEBUG:boto:Path: /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.694 DEBUG:boto:Data: 2022-09-13T07:28:53.694 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.694 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.695 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.695 DEBUG:boto:Params: {} 2022-09-13T07:28:53.696 DEBUG:boto:Token: None 2022-09-13T07:28:53.696 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.697 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0V7Z1eot3JIc9K95xUUZIHwkBUw= 2022-09-13T07:28:53.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0V7Z1eot3JIc9K95xUUZIHwkBUw=', 'Content-Length': '0'} 2022-09-13T07:28:53.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f113c06a4b2fe7b4-0063203135-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:53.701 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:53.701 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:53.702 DEBUG:boto:path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.702 DEBUG:boto:auth_path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.703 DEBUG:boto:Method: GET 2022-09-13T07:28:53.703 DEBUG:boto:Path: /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.704 DEBUG:boto:Data: 2022-09-13T07:28:53.704 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.705 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:53.705 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.706 DEBUG:boto:Params: {} 2022-09-13T07:28:53.706 DEBUG:boto:Token: None 2022-09-13T07:28:53.707 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.707 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5oxpXek2Bun8cfkvJyIM4xfphxg= 2022-09-13T07:28:53.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5oxpXek2Bun8cfkvJyIM4xfphxg=', 'Content-Length': '0'} 2022-09-13T07:28:53.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d363895b0a38a29-0063203135-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.711 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:53.712 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:53.713 DEBUG:boto:path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.713 DEBUG:boto:auth_path=/yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.714 DEBUG:boto:Method: GET 2022-09-13T07:28:53.715 DEBUG:boto:Path: /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.716 DEBUG:boto:Data: 2022-09-13T07:28:53.717 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.718 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.719 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.720 DEBUG:boto:Params: {} 2022-09-13T07:28:53.721 DEBUG:boto:Token: None 2022-09-13T07:28:53.722 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%26?versionId=null 2022-09-13T07:28:53.722 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5oxpXek2Bun8cfkvJyIM4xfphxg= 2022-09-13T07:28:53.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5oxpXek2Bun8cfkvJyIM4xfphxg=', 'Content-Length': '0'} 2022-09-13T07:28:53.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054ebc775333a262b-0063203135-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.727 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.728 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.729 DEBUG:boto:path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:53.730 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:53.731 DEBUG:boto:Method: GET 2022-09-13T07:28:53.732 DEBUG:boto:Path: /yxvvuh-12/?prefix=%26 2022-09-13T07:28:53.733 DEBUG:boto:Data: 2022-09-13T07:28:53.734 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.735 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:53.736 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.737 DEBUG:boto:Params: {} 2022-09-13T07:28:53.738 DEBUG:boto:Token: None 2022-09-13T07:28:53.739 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/ 2022-09-13T07:28:53.740 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0= 2022-09-13T07:28:53.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0=', 'Content-Length': '0'} 2022-09-13T07:28:53.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b025300d72598e45-0063203135-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.745 DEBUG:boto:b'yxvvuh-12&1000false&2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.746 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.747 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.747 DEBUG:boto:path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:53.748 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:53.748 DEBUG:boto:Method: GET 2022-09-13T07:28:53.749 DEBUG:boto:Path: /yxvvuh-12/?prefix=%26 2022-09-13T07:28:53.749 DEBUG:boto:Data: 2022-09-13T07:28:53.750 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.751 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.751 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.752 DEBUG:boto:Params: {} 2022-09-13T07:28:53.752 DEBUG:boto:Token: None 2022-09-13T07:28:53.753 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/ 2022-09-13T07:28:53.753 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0= 2022-09-13T07:28:53.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0=', 'Content-Length': '0'} 2022-09-13T07:28:53.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000499fb70d02d7e2df-0063203135-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.757 DEBUG:boto:b'yxvvuh-12&1000false&2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.758 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:53.758 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:53.759 DEBUG:boto:Method: HEAD 2022-09-13T07:28:53.760 DEBUG:boto:Path: /yxvvuh-12/%26 2022-09-13T07:28:53.760 DEBUG:boto:Data: 2022-09-13T07:28:53.760 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.761 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:53.761 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.762 DEBUG:boto:Params: {} 2022-09-13T07:28:53.762 DEBUG:boto:Token: None 2022-09-13T07:28:53.763 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%26 2022-09-13T07:28:53.764 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qUhMdPgAkGG/R+z0syZYEBWbXlE= 2022-09-13T07:28:53.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qUhMdPgAkGG/R+z0syZYEBWbXlE=', 'Content-Length': '0'} 2022-09-13T07:28:53.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007284e4981b6599a5-0063203135-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.767 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:53.767 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:53.768 DEBUG:boto:Method: HEAD 2022-09-13T07:28:53.768 DEBUG:boto:Path: /yxvvuh-12/%26 2022-09-13T07:28:53.769 DEBUG:boto:Data: 2022-09-13T07:28:53.769 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.770 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.771 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.771 DEBUG:boto:Params: {} 2022-09-13T07:28:53.772 DEBUG:boto:Token: None 2022-09-13T07:28:53.772 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%26 2022-09-13T07:28:53.773 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qUhMdPgAkGG/R+z0syZYEBWbXlE= 2022-09-13T07:28:53.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qUhMdPgAkGG/R+z0syZYEBWbXlE=', 'Content-Length': '0'} 2022-09-13T07:28:53.776 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000536046e0de1afc28-0063203135-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-13T07:28:53.777 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:53.778 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:53.778 DEBUG:boto:Method: GET 2022-09-13T07:28:53.779 DEBUG:boto:Path: /yxvvuh-12/%26 2022-09-13T07:28:53.779 DEBUG:boto:Data: 2022-09-13T07:28:53.780 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.780 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:53.781 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.782 DEBUG:boto:Params: {} 2022-09-13T07:28:53.783 DEBUG:boto:Token: None 2022-09-13T07:28:53.783 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%26 2022-09-13T07:28:53.784 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4tEUGPrb7dgwxhcp7c06mRBnTFI= 2022-09-13T07:28:53.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4tEUGPrb7dgwxhcp7c06mRBnTFI=', 'Content-Length': '0'} 2022-09-13T07:28:53.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab860b2d2c79ff22-0063203135-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.834 DEBUG:boto:path=/yxvvuh-12/%26 2022-09-13T07:28:53.835 DEBUG:boto:auth_path=/yxvvuh-12/%26 2022-09-13T07:28:53.836 DEBUG:boto:Method: GET 2022-09-13T07:28:53.837 DEBUG:boto:Path: /yxvvuh-12/%26 2022-09-13T07:28:53.838 DEBUG:boto:Data: 2022-09-13T07:28:53.839 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.840 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.841 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.842 DEBUG:boto:Params: {} 2022-09-13T07:28:53.843 DEBUG:boto:Token: None 2022-09-13T07:28:53.844 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%26 2022-09-13T07:28:53.845 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4tEUGPrb7dgwxhcp7c06mRBnTFI= 2022-09-13T07:28:53.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4tEUGPrb7dgwxhcp7c06mRBnTFI=', 'Content-Length': '0'} 2022-09-13T07:28:53.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5e79f981ba1b1b6-0063203135-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.850 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.850 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.851 DEBUG:boto:path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:53.851 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:53.852 DEBUG:boto:Method: GET 2022-09-13T07:28:53.852 DEBUG:boto:Path: /yxvvuh-12/?prefix=%26 2022-09-13T07:28:53.853 DEBUG:boto:Data: 2022-09-13T07:28:53.853 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.854 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:53.855 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.855 DEBUG:boto:Params: {} 2022-09-13T07:28:53.856 DEBUG:boto:Token: None 2022-09-13T07:28:53.856 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/ 2022-09-13T07:28:53.857 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0= 2022-09-13T07:28:53.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0=', 'Content-Length': '0'} 2022-09-13T07:28:53.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075623e0134227195-0063203135-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.861 DEBUG:boto:b'yxvvuh-12&1000false&2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.863 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.864 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.864 DEBUG:boto:path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:53.865 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%26 2022-09-13T07:28:53.866 DEBUG:boto:Method: GET 2022-09-13T07:28:53.867 DEBUG:boto:Path: /yxvvuh-12/?prefix=%26 2022-09-13T07:28:53.868 DEBUG:boto:Data: 2022-09-13T07:28:53.869 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.870 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.870 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.871 DEBUG:boto:Params: {} 2022-09-13T07:28:53.871 DEBUG:boto:Token: None 2022-09-13T07:28:53.872 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/ 2022-09-13T07:28:53.872 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0= 2022-09-13T07:28:53.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0=', 'Content-Length': '0'} 2022-09-13T07:28:53.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000da342b57c54789a-0063203135-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.877 DEBUG:boto:b'yxvvuh-12&1000false&2022-09-13T07:28:26.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:53.879 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:53.879 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:53.880 DEBUG:boto:path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.880 DEBUG:boto:auth_path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.881 DEBUG:boto:Method: GET 2022-09-13T07:28:53.881 DEBUG:boto:Path: /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.882 DEBUG:boto:Data: 2022-09-13T07:28:53.882 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.883 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:53.884 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.884 DEBUG:boto:Params: {} 2022-09-13T07:28:53.885 DEBUG:boto:Token: None 2022-09-13T07:28:53.886 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.886 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i+m2YgldH4KujntltZtPaQlNmlw= 2022-09-13T07:28:53.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i+m2YgldH4KujntltZtPaQlNmlw=', 'Content-Length': '0'} 2022-09-13T07:28:53.889 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005dfeee171f88404e-0063203135-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.891 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:53.892 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:53.893 DEBUG:boto:path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.894 DEBUG:boto:auth_path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.895 DEBUG:boto:Method: GET 2022-09-13T07:28:53.896 DEBUG:boto:Path: /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.897 DEBUG:boto:Data: 2022-09-13T07:28:53.898 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.899 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.900 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.901 DEBUG:boto:Params: {} 2022-09-13T07:28:53.902 DEBUG:boto:Token: None 2022-09-13T07:28:53.903 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.904 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i+m2YgldH4KujntltZtPaQlNmlw= 2022-09-13T07:28:53.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i+m2YgldH4KujntltZtPaQlNmlw=', 'Content-Length': '0'} 2022-09-13T07:28:53.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1e064ac1e64d78b-0063203135-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.909 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:53.910 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:53.911 DEBUG:boto:path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.911 DEBUG:boto:auth_path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.912 DEBUG:boto:Method: HEAD 2022-09-13T07:28:53.913 DEBUG:boto:Path: /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.913 DEBUG:boto:Data: 2022-09-13T07:28:53.914 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.915 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:53.915 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.916 DEBUG:boto:Params: {} 2022-09-13T07:28:53.917 DEBUG:boto:Token: None 2022-09-13T07:28:53.917 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.918 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WrhQVgjtkWKL1ZO5lF9SkPJGlwg= 2022-09-13T07:28:53.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WrhQVgjtkWKL1ZO5lF9SkPJGlwg=', 'Content-Length': '0'} 2022-09-13T07:28:53.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b085ab1b6d4c1d6-0063203135-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.922 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:53.923 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:53.924 DEBUG:boto:path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.925 DEBUG:boto:auth_path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.925 DEBUG:boto:Method: HEAD 2022-09-13T07:28:53.927 DEBUG:boto:Path: /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.928 DEBUG:boto:Data: 2022-09-13T07:28:53.928 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.929 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.930 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.931 DEBUG:boto:Params: {} 2022-09-13T07:28:53.932 DEBUG:boto:Token: None 2022-09-13T07:28:53.933 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.934 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WrhQVgjtkWKL1ZO5lF9SkPJGlwg= 2022-09-13T07:28:53.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WrhQVgjtkWKL1ZO5lF9SkPJGlwg=', 'Content-Length': '0'} 2022-09-13T07:28:53.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fb9cdf5201b4031-0063203135-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:53.938 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:53.939 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:53.939 DEBUG:boto:path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.940 DEBUG:boto:auth_path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.940 DEBUG:boto:Method: GET 2022-09-13T07:28:53.941 DEBUG:boto:Path: /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.942 DEBUG:boto:Data: 2022-09-13T07:28:53.942 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.943 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:53.944 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.944 DEBUG:boto:Params: {} 2022-09-13T07:28:53.945 DEBUG:boto:Token: None 2022-09-13T07:28:53.945 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.946 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i+m2YgldH4KujntltZtPaQlNmlw= 2022-09-13T07:28:53.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i+m2YgldH4KujntltZtPaQlNmlw=', 'Content-Length': '0'} 2022-09-13T07:28:53.948 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a593277f29c21ebc-0063203135-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.949 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:53.950 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:53.950 DEBUG:boto:path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.951 DEBUG:boto:auth_path=/yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.952 DEBUG:boto:Method: GET 2022-09-13T07:28:53.952 DEBUG:boto:Path: /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.953 DEBUG:boto:Data: 2022-09-13T07:28:53.953 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.954 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.954 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.955 DEBUG:boto:Params: {} 2022-09-13T07:28:53.956 DEBUG:boto:Token: None 2022-09-13T07:28:53.956 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%3A?versionId=null 2022-09-13T07:28:53.957 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i+m2YgldH4KujntltZtPaQlNmlw= 2022-09-13T07:28:53.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i+m2YgldH4KujntltZtPaQlNmlw=', 'Content-Length': '0'} 2022-09-13T07:28:53.959 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049093c91d6ab4811-0063203135-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.960 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.961 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.961 DEBUG:boto:path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:53.962 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:53.962 DEBUG:boto:Method: GET 2022-09-13T07:28:53.963 DEBUG:boto:Path: /yxvvuh-12/?prefix=%3A 2022-09-13T07:28:53.964 DEBUG:boto:Data: 2022-09-13T07:28:53.964 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.965 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:53.965 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.966 DEBUG:boto:Params: {} 2022-09-13T07:28:53.966 DEBUG:boto:Token: None 2022-09-13T07:28:53.967 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/ 2022-09-13T07:28:53.968 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0= 2022-09-13T07:28:53.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0=', 'Content-Length': '0'} 2022-09-13T07:28:53.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c814c4e476c53545-0063203135-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.972 DEBUG:boto:b'yxvvuh-12:1000false:2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.973 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:53.973 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:53.974 DEBUG:boto:path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:53.974 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:53.975 DEBUG:boto:Method: GET 2022-09-13T07:28:53.976 DEBUG:boto:Path: /yxvvuh-12/?prefix=%3A 2022-09-13T07:28:53.976 DEBUG:boto:Data: 2022-09-13T07:28:53.977 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.978 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:53.978 DEBUG:boto:Port: 8001 2022-09-13T07:28:53.979 DEBUG:boto:Params: {} 2022-09-13T07:28:53.980 DEBUG:boto:Token: None 2022-09-13T07:28:53.980 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/ 2022-09-13T07:28:53.981 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0= 2022-09-13T07:28:53.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siSn7MLutZw3AKs7RJ90iGAqsr0=', 'Content-Length': '0'} 2022-09-13T07:28:53.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000958207fa41ade2cf-0063203135-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.985 DEBUG:boto:b'yxvvuh-12:1000false:2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:53.986 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:53.987 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:53.988 DEBUG:boto:Method: HEAD 2022-09-13T07:28:53.989 DEBUG:boto:Path: /yxvvuh-12/%3A 2022-09-13T07:28:53.989 DEBUG:boto:Data: 2022-09-13T07:28:53.990 DEBUG:boto:Headers: {} 2022-09-13T07:28:53.990 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:53.991 DEBUG:boto:Port: 8000 2022-09-13T07:28:53.991 DEBUG:boto:Params: {} 2022-09-13T07:28:53.992 DEBUG:boto:Token: None 2022-09-13T07:28:53.993 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:53 GMT /yxvvuh-12/%3A 2022-09-13T07:28:53.993 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Z/GYLXw4FoQ+PswH3JbfThFC+xA= 2022-09-13T07:28:53.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Z/GYLXw4FoQ+PswH3JbfThFC+xA=', 'Content-Length': '0'} 2022-09-13T07:28:53.996 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d44d06c4c8be2e33-0063203135-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:53.996 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:53.997 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:53.998 DEBUG:boto:Method: HEAD 2022-09-13T07:28:53.998 DEBUG:boto:Path: /yxvvuh-12/%3A 2022-09-13T07:28:53.999 DEBUG:boto:Data: 2022-09-13T07:28:54.000 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.000 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.002 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.003 DEBUG:boto:Params: {} 2022-09-13T07:28:54.004 DEBUG:boto:Token: None 2022-09-13T07:28:54.006 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/%3A 2022-09-13T07:28:54.007 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Nbdbo+3f9qlDLRya3dh6Mg+9rmQ= 2022-09-13T07:28:54.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Nbdbo+3f9qlDLRya3dh6Mg+9rmQ=', 'Content-Length': '0'} 2022-09-13T07:28:54.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cca2c0856f93597b-0063203136-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-13T07:28:54.013 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:54.014 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:54.015 DEBUG:boto:Method: GET 2022-09-13T07:28:54.016 DEBUG:boto:Path: /yxvvuh-12/%3A 2022-09-13T07:28:54.017 DEBUG:boto:Data: 2022-09-13T07:28:54.018 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.019 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.020 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.021 DEBUG:boto:Params: {} 2022-09-13T07:28:54.022 DEBUG:boto:Token: None 2022-09-13T07:28:54.024 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/%3A 2022-09-13T07:28:54.025 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yZJM48h/YNf/mHC3G5qar76TrHs= 2022-09-13T07:28:54.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yZJM48h/YNf/mHC3G5qar76TrHs=', 'Content-Length': '0'} 2022-09-13T07:28:54.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004af3455026ce606-0063203136-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.030 DEBUG:boto:path=/yxvvuh-12/%3A 2022-09-13T07:28:54.031 DEBUG:boto:auth_path=/yxvvuh-12/%3A 2022-09-13T07:28:54.032 DEBUG:boto:Method: GET 2022-09-13T07:28:54.032 DEBUG:boto:Path: /yxvvuh-12/%3A 2022-09-13T07:28:54.033 DEBUG:boto:Data: 2022-09-13T07:28:54.033 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.034 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.035 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.035 DEBUG:boto:Params: {} 2022-09-13T07:28:54.036 DEBUG:boto:Token: None 2022-09-13T07:28:54.037 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/%3A 2022-09-13T07:28:54.037 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yZJM48h/YNf/mHC3G5qar76TrHs= 2022-09-13T07:28:54.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yZJM48h/YNf/mHC3G5qar76TrHs=', 'Content-Length': '0'} 2022-09-13T07:28:54.040 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8bd7ee57eb97d3c-0063203136-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.086 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:54.087 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:54.089 DEBUG:boto:path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:54.090 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:54.091 DEBUG:boto:Method: GET 2022-09-13T07:28:54.092 DEBUG:boto:Path: /yxvvuh-12/?prefix=%3A 2022-09-13T07:28:54.093 DEBUG:boto:Data: 2022-09-13T07:28:54.094 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.095 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.096 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.097 DEBUG:boto:Params: {} 2022-09-13T07:28:54.098 DEBUG:boto:Token: None 2022-09-13T07:28:54.099 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/ 2022-09-13T07:28:54.100 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks= 2022-09-13T07:28:54.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks=', 'Content-Length': '0'} 2022-09-13T07:28:54.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba04cb32b960a1bd-0063203136-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.106 DEBUG:boto:b'yxvvuh-12:1000false:2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:54.108 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:54.109 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:54.110 DEBUG:boto:path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:54.111 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=%3A 2022-09-13T07:28:54.112 DEBUG:boto:Method: GET 2022-09-13T07:28:54.113 DEBUG:boto:Path: /yxvvuh-12/?prefix=%3A 2022-09-13T07:28:54.114 DEBUG:boto:Data: 2022-09-13T07:28:54.115 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.116 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.117 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.118 DEBUG:boto:Params: {} 2022-09-13T07:28:54.119 DEBUG:boto:Token: None 2022-09-13T07:28:54.121 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/ 2022-09-13T07:28:54.122 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks= 2022-09-13T07:28:54.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks=', 'Content-Length': '0'} 2022-09-13T07:28:54.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000628b3e29bd12ef0-0063203136-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.128 DEBUG:boto:b'yxvvuh-12:1000false:2022-09-13T07:28:26.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:54.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:54.131 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.132 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.133 DEBUG:boto:path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.134 DEBUG:boto:auth_path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.135 DEBUG:boto:Method: GET 2022-09-13T07:28:54.136 DEBUG:boto:Path: /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.138 DEBUG:boto:Data: 2022-09-13T07:28:54.139 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.140 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.141 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.142 DEBUG:boto:Params: {} 2022-09-13T07:28:54.144 DEBUG:boto:Token: None 2022-09-13T07:28:54.145 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.146 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:C93Kzf0xo5NVaJkgHCB4OQc/h1Y= 2022-09-13T07:28:54.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:C93Kzf0xo5NVaJkgHCB4OQc/h1Y=', 'Content-Length': '0'} 2022-09-13T07:28:54.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000863a6d5732d9755f-0063203136-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.151 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.152 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.153 DEBUG:boto:path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.153 DEBUG:boto:auth_path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.154 DEBUG:boto:Method: GET 2022-09-13T07:28:54.154 DEBUG:boto:Path: /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.155 DEBUG:boto:Data: 2022-09-13T07:28:54.156 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.156 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.157 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.158 DEBUG:boto:Params: {} 2022-09-13T07:28:54.159 DEBUG:boto:Token: None 2022-09-13T07:28:54.160 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.160 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:C93Kzf0xo5NVaJkgHCB4OQc/h1Y= 2022-09-13T07:28:54.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:C93Kzf0xo5NVaJkgHCB4OQc/h1Y=', 'Content-Length': '0'} 2022-09-13T07:28:54.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000126a18c29d570442-0063203136-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.164 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.165 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.166 DEBUG:boto:path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.167 DEBUG:boto:auth_path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.168 DEBUG:boto:Method: HEAD 2022-09-13T07:28:54.168 DEBUG:boto:Path: /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.169 DEBUG:boto:Data: 2022-09-13T07:28:54.169 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.170 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.171 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.171 DEBUG:boto:Params: {} 2022-09-13T07:28:54.172 DEBUG:boto:Token: None 2022-09-13T07:28:54.172 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.173 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jaUPs9aXAoRW0xM2bytYLKLwO8Y= 2022-09-13T07:28:54.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jaUPs9aXAoRW0xM2bytYLKLwO8Y=', 'Content-Length': '0'} 2022-09-13T07:28:54.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008674415c5b0f4e0c-0063203136-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.178 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.179 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.180 DEBUG:boto:path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.181 DEBUG:boto:auth_path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.182 DEBUG:boto:Method: HEAD 2022-09-13T07:28:54.183 DEBUG:boto:Path: /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.183 DEBUG:boto:Data: 2022-09-13T07:28:54.184 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.185 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.185 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.186 DEBUG:boto:Params: {} 2022-09-13T07:28:54.187 DEBUG:boto:Token: None 2022-09-13T07:28:54.188 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.189 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jaUPs9aXAoRW0xM2bytYLKLwO8Y= 2022-09-13T07:28:54.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jaUPs9aXAoRW0xM2bytYLKLwO8Y=', 'Content-Length': '0'} 2022-09-13T07:28:54.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a908d8326349601-0063203136-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:54.195 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.196 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.197 DEBUG:boto:path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.198 DEBUG:boto:auth_path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.199 DEBUG:boto:Method: GET 2022-09-13T07:28:54.199 DEBUG:boto:Path: /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.200 DEBUG:boto:Data: 2022-09-13T07:28:54.200 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.201 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.201 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.202 DEBUG:boto:Params: {} 2022-09-13T07:28:54.203 DEBUG:boto:Token: None 2022-09-13T07:28:54.203 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.204 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:C93Kzf0xo5NVaJkgHCB4OQc/h1Y= 2022-09-13T07:28:54.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:C93Kzf0xo5NVaJkgHCB4OQc/h1Y=', 'Content-Length': '0'} 2022-09-13T07:28:54.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff5416abd5ec69c1-0063203136-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.207 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.208 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.209 DEBUG:boto:path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.209 DEBUG:boto:auth_path=/yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.210 DEBUG:boto:Method: GET 2022-09-13T07:28:54.210 DEBUG:boto:Path: /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.211 DEBUG:boto:Data: 2022-09-13T07:28:54.211 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.212 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.213 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.213 DEBUG:boto:Params: {} 2022-09-13T07:28:54.214 DEBUG:boto:Token: None 2022-09-13T07:28:54.215 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/_myobj?versionId=null 2022-09-13T07:28:54.216 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:C93Kzf0xo5NVaJkgHCB4OQc/h1Y= 2022-09-13T07:28:54.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:C93Kzf0xo5NVaJkgHCB4OQc/h1Y=', 'Content-Length': '0'} 2022-09-13T07:28:54.220 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070aa90b7595d775a-0063203136-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.222 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:54.223 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:54.224 DEBUG:boto:path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:54.225 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:54.225 DEBUG:boto:Method: GET 2022-09-13T07:28:54.226 DEBUG:boto:Path: /yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:54.226 DEBUG:boto:Data: 2022-09-13T07:28:54.227 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.227 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.228 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.228 DEBUG:boto:Params: {} 2022-09-13T07:28:54.229 DEBUG:boto:Token: None 2022-09-13T07:28:54.230 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/ 2022-09-13T07:28:54.230 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks= 2022-09-13T07:28:54.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks=', 'Content-Length': '0'} 2022-09-13T07:28:54.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b1d1c10fbc89670-0063203136-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.278 DEBUG:boto:b'yxvvuh-12_myobj1000false_myobj2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:54.279 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:54.280 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:54.280 DEBUG:boto:path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:54.281 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:54.282 DEBUG:boto:Method: GET 2022-09-13T07:28:54.282 DEBUG:boto:Path: /yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:54.283 DEBUG:boto:Data: 2022-09-13T07:28:54.283 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.284 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.284 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.285 DEBUG:boto:Params: {} 2022-09-13T07:28:54.286 DEBUG:boto:Token: None 2022-09-13T07:28:54.287 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/ 2022-09-13T07:28:54.287 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks= 2022-09-13T07:28:54.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks=', 'Content-Length': '0'} 2022-09-13T07:28:54.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d25bfeb7e1fbe97-0063203136-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.292 DEBUG:boto:b'yxvvuh-12_myobj1000false_myobj2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:54.293 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.294 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.294 DEBUG:boto:Method: HEAD 2022-09-13T07:28:54.295 DEBUG:boto:Path: /yxvvuh-12/_myobj 2022-09-13T07:28:54.296 DEBUG:boto:Data: 2022-09-13T07:28:54.297 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.298 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.299 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.300 DEBUG:boto:Params: {} 2022-09-13T07:28:54.302 DEBUG:boto:Token: None 2022-09-13T07:28:54.303 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/_myobj 2022-09-13T07:28:54.304 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FrvuCE3wo5HztxltO/LM9o2Trgw= 2022-09-13T07:28:54.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FrvuCE3wo5HztxltO/LM9o2Trgw=', 'Content-Length': '0'} 2022-09-13T07:28:54.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea0fda5ac5552fe1-0063203136-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.309 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.310 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.311 DEBUG:boto:Method: HEAD 2022-09-13T07:28:54.311 DEBUG:boto:Path: /yxvvuh-12/_myobj 2022-09-13T07:28:54.312 DEBUG:boto:Data: 2022-09-13T07:28:54.313 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.313 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.314 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.314 DEBUG:boto:Params: {} 2022-09-13T07:28:54.315 DEBUG:boto:Token: None 2022-09-13T07:28:54.316 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/_myobj 2022-09-13T07:28:54.316 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FrvuCE3wo5HztxltO/LM9o2Trgw= 2022-09-13T07:28:54.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FrvuCE3wo5HztxltO/LM9o2Trgw=', 'Content-Length': '0'} 2022-09-13T07:28:54.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000567d62fd29e7b517-0063203136-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-13T07:28:54.320 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.321 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.322 DEBUG:boto:Method: GET 2022-09-13T07:28:54.322 DEBUG:boto:Path: /yxvvuh-12/_myobj 2022-09-13T07:28:54.323 DEBUG:boto:Data: 2022-09-13T07:28:54.323 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.324 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.324 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.325 DEBUG:boto:Params: {} 2022-09-13T07:28:54.325 DEBUG:boto:Token: None 2022-09-13T07:28:54.326 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/_myobj 2022-09-13T07:28:54.326 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CVHd6NZAMQlJgWEPCI/gHuOcNeg= 2022-09-13T07:28:54.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CVHd6NZAMQlJgWEPCI/gHuOcNeg=', 'Content-Length': '0'} 2022-09-13T07:28:54.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000011a6be2d6d859c0-0063203136-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.330 DEBUG:boto:path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.331 DEBUG:boto:auth_path=/yxvvuh-12/_myobj 2022-09-13T07:28:54.331 DEBUG:boto:Method: GET 2022-09-13T07:28:54.332 DEBUG:boto:Path: /yxvvuh-12/_myobj 2022-09-13T07:28:54.332 DEBUG:boto:Data: 2022-09-13T07:28:54.333 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.333 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.334 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.334 DEBUG:boto:Params: {} 2022-09-13T07:28:54.335 DEBUG:boto:Token: None 2022-09-13T07:28:54.335 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/_myobj 2022-09-13T07:28:54.336 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CVHd6NZAMQlJgWEPCI/gHuOcNeg= 2022-09-13T07:28:54.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CVHd6NZAMQlJgWEPCI/gHuOcNeg=', 'Content-Length': '0'} 2022-09-13T07:28:54.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b08ab4a102895bd0-0063203136-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.382 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:54.383 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:54.384 DEBUG:boto:path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:54.385 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:54.386 DEBUG:boto:Method: GET 2022-09-13T07:28:54.386 DEBUG:boto:Path: /yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:54.387 DEBUG:boto:Data: 2022-09-13T07:28:54.388 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.389 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.389 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.390 DEBUG:boto:Params: {} 2022-09-13T07:28:54.391 DEBUG:boto:Token: None 2022-09-13T07:28:54.392 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/ 2022-09-13T07:28:54.393 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks= 2022-09-13T07:28:54.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks=', 'Content-Length': '0'} 2022-09-13T07:28:54.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f44a3e089b3814ab-0063203136-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.398 DEBUG:boto:b'yxvvuh-12_myobj1000false_myobj2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:54.399 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:54.400 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:54.400 DEBUG:boto:path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:54.401 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:54.401 DEBUG:boto:Method: GET 2022-09-13T07:28:54.402 DEBUG:boto:Path: /yxvvuh-12/?prefix=_myobj 2022-09-13T07:28:54.402 DEBUG:boto:Data: 2022-09-13T07:28:54.403 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.403 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.404 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.404 DEBUG:boto:Params: {} 2022-09-13T07:28:54.405 DEBUG:boto:Token: None 2022-09-13T07:28:54.406 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/ 2022-09-13T07:28:54.406 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks= 2022-09-13T07:28:54.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks=', 'Content-Length': '0'} 2022-09-13T07:28:54.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e4067db93c3a837-0063203136-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.410 DEBUG:boto:b'yxvvuh-12_myobj1000false_myobj2022-09-13T07:28:26.452Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:54.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:54.412 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:54.412 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:54.413 DEBUG:boto:path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.413 DEBUG:boto:auth_path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.414 DEBUG:boto:Method: GET 2022-09-13T07:28:54.414 DEBUG:boto:Path: /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.415 DEBUG:boto:Data: 2022-09-13T07:28:54.415 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.416 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.416 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.417 DEBUG:boto:Params: {} 2022-09-13T07:28:54.418 DEBUG:boto:Token: None 2022-09-13T07:28:54.419 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.420 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OrcfWjlkFj6qH5Kq3VCeEsV4ppU= 2022-09-13T07:28:54.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OrcfWjlkFj6qH5Kq3VCeEsV4ppU=', 'Content-Length': '0'} 2022-09-13T07:28:54.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dc87ac1d2c57456-0063203136-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.424 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:54.425 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:54.425 DEBUG:boto:path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.426 DEBUG:boto:auth_path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.427 DEBUG:boto:Method: GET 2022-09-13T07:28:54.427 DEBUG:boto:Path: /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.428 DEBUG:boto:Data: 2022-09-13T07:28:54.428 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.429 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.430 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.430 DEBUG:boto:Params: {} 2022-09-13T07:28:54.431 DEBUG:boto:Token: None 2022-09-13T07:28:54.432 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.432 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OrcfWjlkFj6qH5Kq3VCeEsV4ppU= 2022-09-13T07:28:54.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OrcfWjlkFj6qH5Kq3VCeEsV4ppU=', 'Content-Length': '0'} 2022-09-13T07:28:54.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9cae393a71da29c-0063203136-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.436 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:54.436 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:54.437 DEBUG:boto:path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.437 DEBUG:boto:auth_path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.438 DEBUG:boto:Method: HEAD 2022-09-13T07:28:54.438 DEBUG:boto:Path: /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.439 DEBUG:boto:Data: 2022-09-13T07:28:54.440 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.440 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.441 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.441 DEBUG:boto:Params: {} 2022-09-13T07:28:54.442 DEBUG:boto:Token: None 2022-09-13T07:28:54.442 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.443 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PQPS/AawDlnaNvdtcIyhzXPNDe4= 2022-09-13T07:28:54.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PQPS/AawDlnaNvdtcIyhzXPNDe4=', 'Content-Length': '0'} 2022-09-13T07:28:54.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b3ce858ca8174c9-0063203136-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.447 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:54.448 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:54.448 DEBUG:boto:path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.449 DEBUG:boto:auth_path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.449 DEBUG:boto:Method: HEAD 2022-09-13T07:28:54.450 DEBUG:boto:Path: /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.450 DEBUG:boto:Data: 2022-09-13T07:28:54.451 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.451 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.452 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.453 DEBUG:boto:Params: {} 2022-09-13T07:28:54.454 DEBUG:boto:Token: None 2022-09-13T07:28:54.454 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.455 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PQPS/AawDlnaNvdtcIyhzXPNDe4= 2022-09-13T07:28:54.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PQPS/AawDlnaNvdtcIyhzXPNDe4=', 'Content-Length': '0'} 2022-09-13T07:28:54.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0b229c11ee18d67-0063203136-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:54.460 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:54.461 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:54.461 DEBUG:boto:path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.462 DEBUG:boto:auth_path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.462 DEBUG:boto:Method: GET 2022-09-13T07:28:54.463 DEBUG:boto:Path: /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.463 DEBUG:boto:Data: 2022-09-13T07:28:54.464 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.465 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.466 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.466 DEBUG:boto:Params: {} 2022-09-13T07:28:54.467 DEBUG:boto:Token: None 2022-09-13T07:28:54.468 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.468 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OrcfWjlkFj6qH5Kq3VCeEsV4ppU= 2022-09-13T07:28:54.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OrcfWjlkFj6qH5Kq3VCeEsV4ppU=', 'Content-Length': '0'} 2022-09-13T07:28:54.471 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d520587d28c8dd0-0063203136-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.472 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:54.472 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:54.473 DEBUG:boto:path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.474 DEBUG:boto:auth_path=/yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.474 DEBUG:boto:Method: GET 2022-09-13T07:28:54.475 DEBUG:boto:Path: /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.475 DEBUG:boto:Data: 2022-09-13T07:28:54.476 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.476 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.477 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.477 DEBUG:boto:Params: {} 2022-09-13T07:28:54.478 DEBUG:boto:Token: None 2022-09-13T07:28:54.479 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/myobj?versionId=null 2022-09-13T07:28:54.479 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OrcfWjlkFj6qH5Kq3VCeEsV4ppU= 2022-09-13T07:28:54.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OrcfWjlkFj6qH5Kq3VCeEsV4ppU=', 'Content-Length': '0'} 2022-09-13T07:28:54.482 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a56136638cf15a16-0063203136-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.522 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:54.523 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:54.524 DEBUG:boto:path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:54.525 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:54.526 DEBUG:boto:Method: GET 2022-09-13T07:28:54.527 DEBUG:boto:Path: /yxvvuh-12/?prefix=myobj 2022-09-13T07:28:54.528 DEBUG:boto:Data: 2022-09-13T07:28:54.529 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.529 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.530 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.531 DEBUG:boto:Params: {} 2022-09-13T07:28:54.532 DEBUG:boto:Token: None 2022-09-13T07:28:54.533 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/ 2022-09-13T07:28:54.534 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks= 2022-09-13T07:28:54.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks=', 'Content-Length': '0'} 2022-09-13T07:28:54.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020b5cd1efc848b29-0063203136-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.545 DEBUG:boto:b'yxvvuh-12myobj1000falsemyobj2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:54.546 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:54.547 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:54.547 DEBUG:boto:path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:54.548 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:54.548 DEBUG:boto:Method: GET 2022-09-13T07:28:54.549 DEBUG:boto:Path: /yxvvuh-12/?prefix=myobj 2022-09-13T07:28:54.549 DEBUG:boto:Data: 2022-09-13T07:28:54.550 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.551 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.552 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.552 DEBUG:boto:Params: {} 2022-09-13T07:28:54.553 DEBUG:boto:Token: None 2022-09-13T07:28:54.554 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/ 2022-09-13T07:28:54.554 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks= 2022-09-13T07:28:54.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks=', 'Content-Length': '0'} 2022-09-13T07:28:54.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1fe1fa8fdab659d-0063203136-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.559 DEBUG:boto:b'yxvvuh-12myobj1000falsemyobj2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:54.560 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:54.560 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:54.561 DEBUG:boto:Method: HEAD 2022-09-13T07:28:54.561 DEBUG:boto:Path: /yxvvuh-12/myobj 2022-09-13T07:28:54.562 DEBUG:boto:Data: 2022-09-13T07:28:54.562 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.563 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.563 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.564 DEBUG:boto:Params: {} 2022-09-13T07:28:54.565 DEBUG:boto:Token: None 2022-09-13T07:28:54.565 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/myobj 2022-09-13T07:28:54.566 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LRMQ2V6PpsJYxnCEcAqZSkUeg8M= 2022-09-13T07:28:54.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LRMQ2V6PpsJYxnCEcAqZSkUeg8M=', 'Content-Length': '0'} 2022-09-13T07:28:54.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007c30f3a0f2d5e60-0063203136-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.569 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:54.570 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:54.571 DEBUG:boto:Method: HEAD 2022-09-13T07:28:54.571 DEBUG:boto:Path: /yxvvuh-12/myobj 2022-09-13T07:28:54.572 DEBUG:boto:Data: 2022-09-13T07:28:54.572 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.573 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.573 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.574 DEBUG:boto:Params: {} 2022-09-13T07:28:54.574 DEBUG:boto:Token: None 2022-09-13T07:28:54.575 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/myobj 2022-09-13T07:28:54.575 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LRMQ2V6PpsJYxnCEcAqZSkUeg8M= 2022-09-13T07:28:54.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LRMQ2V6PpsJYxnCEcAqZSkUeg8M=', 'Content-Length': '0'} 2022-09-13T07:28:54.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7c23904c155783f-0063203136-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:28:54.581 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:54.581 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:54.582 DEBUG:boto:Method: GET 2022-09-13T07:28:54.583 DEBUG:boto:Path: /yxvvuh-12/myobj 2022-09-13T07:28:54.584 DEBUG:boto:Data: 2022-09-13T07:28:54.584 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.585 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.586 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.587 DEBUG:boto:Params: {} 2022-09-13T07:28:54.587 DEBUG:boto:Token: None 2022-09-13T07:28:54.588 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/myobj 2022-09-13T07:28:54.589 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2HHU4W2Abn0Bfxurd6r4CFkkX/o= 2022-09-13T07:28:54.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2HHU4W2Abn0Bfxurd6r4CFkkX/o=', 'Content-Length': '0'} 2022-09-13T07:28:54.591 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d0e1e6a1fa6b055-0063203136-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.592 DEBUG:boto:path=/yxvvuh-12/myobj 2022-09-13T07:28:54.593 DEBUG:boto:auth_path=/yxvvuh-12/myobj 2022-09-13T07:28:54.593 DEBUG:boto:Method: GET 2022-09-13T07:28:54.594 DEBUG:boto:Path: /yxvvuh-12/myobj 2022-09-13T07:28:54.594 DEBUG:boto:Data: 2022-09-13T07:28:54.595 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.595 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.596 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.596 DEBUG:boto:Params: {} 2022-09-13T07:28:54.597 DEBUG:boto:Token: None 2022-09-13T07:28:54.597 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/myobj 2022-09-13T07:28:54.598 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2HHU4W2Abn0Bfxurd6r4CFkkX/o= 2022-09-13T07:28:54.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2HHU4W2Abn0Bfxurd6r4CFkkX/o=', 'Content-Length': '0'} 2022-09-13T07:28:54.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c4be431a3b70222-0063203136-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.642 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:54.643 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:54.643 DEBUG:boto:path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:54.644 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:54.645 DEBUG:boto:Method: GET 2022-09-13T07:28:54.645 DEBUG:boto:Path: /yxvvuh-12/?prefix=myobj 2022-09-13T07:28:54.646 DEBUG:boto:Data: 2022-09-13T07:28:54.646 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.647 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.647 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.648 DEBUG:boto:Params: {} 2022-09-13T07:28:54.649 DEBUG:boto:Token: None 2022-09-13T07:28:54.649 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/ 2022-09-13T07:28:54.650 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks= 2022-09-13T07:28:54.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks=', 'Content-Length': '0'} 2022-09-13T07:28:54.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a27b3f2488f2a1e7-0063203136-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.654 DEBUG:boto:b'yxvvuh-12myobj1000falsemyobj2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:54.655 DEBUG:boto:path=/yxvvuh-12/ 2022-09-13T07:28:54.656 DEBUG:boto:auth_path=/yxvvuh-12/ 2022-09-13T07:28:54.656 DEBUG:boto:path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:54.657 DEBUG:boto:auth_path=/yxvvuh-12/?prefix=myobj 2022-09-13T07:28:54.657 DEBUG:boto:Method: GET 2022-09-13T07:28:54.658 DEBUG:boto:Path: /yxvvuh-12/?prefix=myobj 2022-09-13T07:28:54.658 DEBUG:boto:Data: 2022-09-13T07:28:54.659 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.659 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.660 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.660 DEBUG:boto:Params: {} 2022-09-13T07:28:54.661 DEBUG:boto:Token: None 2022-09-13T07:28:54.662 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-12/ 2022-09-13T07:28:54.662 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks= 2022-09-13T07:28:54.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZFUpOamzaZ0f4AR5n+5L6LHQks=', 'Content-Length': '0'} 2022-09-13T07:28:54.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000895c4129ea05ac94-0063203136-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.666 DEBUG:boto:b'yxvvuh-12myobj1000falsemyobj2022-09-13T07:28:26.424Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:28:54.667 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-12 zones={test-zone2, test-zone3} 2022-09-13T07:28:54.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2022-09-13T07:28:54.669 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-13 2022-09-13T07:28:54.669 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:28:54.670 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:28:54.670 DEBUG:boto:Method: PUT 2022-09-13T07:28:54.671 DEBUG:boto:Path: /yxvvuh-13/ 2022-09-13T07:28:54.671 DEBUG:boto:Data: 2022-09-13T07:28:54.672 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.672 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:54.673 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.673 DEBUG:boto:Params: {} 2022-09-13T07:28:54.674 DEBUG:boto:Token: None 2022-09-13T07:28:54.675 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-13/ 2022-09-13T07:28:54.675 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:At7iOAxm1tbA1uNBjfnSPz62JXM= 2022-09-13T07:28:54.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:At7iOAxm1tbA1uNBjfnSPz62JXM=', 'Content-Length': '0'} 2022-09-13T07:28:54.694 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006be56360922ad24f-0063203136-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.695 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-14 2022-09-13T07:28:54.696 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:28:54.696 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:28:54.697 DEBUG:boto:Method: PUT 2022-09-13T07:28:54.697 DEBUG:boto:Path: /yxvvuh-14/ 2022-09-13T07:28:54.698 DEBUG:boto:Data: 2022-09-13T07:28:54.699 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.699 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.701 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.702 DEBUG:boto:Params: {} 2022-09-13T07:28:54.703 DEBUG:boto:Token: None 2022-09-13T07:28:54.703 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-14/ 2022-09-13T07:28:54.704 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fr5yWgEtLAzMMnW4ij1BLNQB5CI= 2022-09-13T07:28:54.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fr5yWgEtLAzMMnW4ij1BLNQB5CI=', 'Content-Length': '0'} 2022-09-13T07:28:54.743 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006700c081a8bb9bd4-0063203136-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.744 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-15 2022-09-13T07:28:54.745 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:28:54.745 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:28:54.746 DEBUG:boto:Method: PUT 2022-09-13T07:28:54.747 DEBUG:boto:Path: /yxvvuh-15/ 2022-09-13T07:28:54.748 DEBUG:boto:Data: 2022-09-13T07:28:54.749 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.750 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.751 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.751 DEBUG:boto:Params: {} 2022-09-13T07:28:54.752 DEBUG:boto:Token: None 2022-09-13T07:28:54.753 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-15/ 2022-09-13T07:28:54.754 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z6n411KbpOIHGe41aDk2LUjr2T0= 2022-09-13T07:28:54.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z6n411KbpOIHGe41aDk2LUjr2T0=', 'Content-Length': '0'} 2022-09-13T07:28:54.788 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000013fbf980b7837870-0063203136-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.789 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:28:54.790 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:28:54.791 DEBUG:boto:Method: HEAD 2022-09-13T07:28:54.792 DEBUG:boto:Path: /yxvvuh-13/ 2022-09-13T07:28:54.792 DEBUG:boto:Data: 2022-09-13T07:28:54.793 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.794 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:54.795 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.796 DEBUG:boto:Params: {} 2022-09-13T07:28:54.797 DEBUG:boto:Token: None 2022-09-13T07:28:54.798 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-13/ 2022-09-13T07:28:54.799 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:435AqvKlaAk2YIERYv5BOBvrhVE= 2022-09-13T07:28:54.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:435AqvKlaAk2YIERYv5BOBvrhVE=', 'Content-Length': '0'} 2022-09-13T07:28:54.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080fa9c00064767d7-0063203136-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.807 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:28:54.808 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:28:54.809 DEBUG:boto:Method: PUT 2022-09-13T07:28:54.810 DEBUG:boto:Path: /yxvvuh-13/myobj 2022-09-13T07:28:54.810 DEBUG:boto:Data: 2022-09-13T07:28:54.811 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-13T07:28:54.812 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:28:54.813 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.814 DEBUG:boto:Params: {} 2022-09-13T07:28:54.815 DEBUG:boto:Token: None 2022-09-13T07:28:54.816 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-13/myobj 2022-09-13T07:28:54.817 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iA/BE9xEWOBIZ7igi+dcGZYqctY= 2022-09-13T07:28:54.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iA/BE9xEWOBIZ7igi+dcGZYqctY='} 2022-09-13T07:28:54.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d3d9f765c6b1194-0063203136-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.865 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:28:54.865 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:28:54.866 DEBUG:boto:Method: HEAD 2022-09-13T07:28:54.866 DEBUG:boto:Path: /yxvvuh-14/ 2022-09-13T07:28:54.867 DEBUG:boto:Data: 2022-09-13T07:28:54.867 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.868 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.868 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.869 DEBUG:boto:Params: {} 2022-09-13T07:28:54.869 DEBUG:boto:Token: None 2022-09-13T07:28:54.870 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-14/ 2022-09-13T07:28:54.871 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:sgu/3LqQXpqAFoDY4KuINR0fn9Q= 2022-09-13T07:28:54.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:sgu/3LqQXpqAFoDY4KuINR0fn9Q=', 'Content-Length': '0'} 2022-09-13T07:28:54.879 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b8819e0ba9712ef-0063203136-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.880 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:28:54.881 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:28:54.882 DEBUG:boto:Method: PUT 2022-09-13T07:28:54.883 DEBUG:boto:Path: /yxvvuh-14/myobj 2022-09-13T07:28:54.883 DEBUG:boto:Data: 2022-09-13T07:28:54.884 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-13T07:28:54.885 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:28:54.886 DEBUG:boto:Port: 8000 2022-09-13T07:28:54.887 DEBUG:boto:Params: {} 2022-09-13T07:28:54.888 DEBUG:boto:Token: None 2022-09-13T07:28:54.889 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-14/myobj 2022-09-13T07:28:54.891 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:n5/34EUPKRGwNNjoat31Jqvpdb4= 2022-09-13T07:28:54.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:n5/34EUPKRGwNNjoat31Jqvpdb4='} 2022-09-13T07:28:54.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a145284f01a0c672-0063203136-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.939 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:28:54.940 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:28:54.940 DEBUG:boto:Method: HEAD 2022-09-13T07:28:54.940 DEBUG:boto:Path: /yxvvuh-15/ 2022-09-13T07:28:54.941 DEBUG:boto:Data: 2022-09-13T07:28:54.941 DEBUG:boto:Headers: {} 2022-09-13T07:28:54.942 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.942 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.943 DEBUG:boto:Params: {} 2022-09-13T07:28:54.943 DEBUG:boto:Token: None 2022-09-13T07:28:54.944 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-15/ 2022-09-13T07:28:54.944 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mh4T+oQ+sloamHktv5X0CkcvHsY= 2022-09-13T07:28:54.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mh4T+oQ+sloamHktv5X0CkcvHsY=', 'Content-Length': '0'} 2022-09-13T07:28:54.947 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006196448f9a86ec52-0063203136-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.948 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:28:54.949 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:28:54.949 DEBUG:boto:Method: PUT 2022-09-13T07:28:54.950 DEBUG:boto:Path: /yxvvuh-15/myobj 2022-09-13T07:28:54.950 DEBUG:boto:Data: 2022-09-13T07:28:54.951 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-13T07:28:54.951 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:28:54.952 DEBUG:boto:Port: 8001 2022-09-13T07:28:54.953 DEBUG:boto:Params: {} 2022-09-13T07:28:54.953 DEBUG:boto:Token: None 2022-09-13T07:28:54.954 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 13 Sep 2022 07:28:54 GMT /yxvvuh-15/myobj 2022-09-13T07:28:54.954 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MIjuiSDZDtjRBcXO/JR4dPQo8qw= 2022-09-13T07:28:54.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:28:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MIjuiSDZDtjRBcXO/JR4dPQo8qw='} 2022-09-13T07:28:54.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022e9bb0332433ef9-0063203136-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:28:54.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:54.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:54.964 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:55.017 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:55.018 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:55.063 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:28:55.085 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:28:55.086 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:55.086 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:55.087 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:28:55.088 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:55.089 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:55.089 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:28:55.090 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:28:55.091 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:55.091 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:55.092 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054134.773629_86.1", 2022-09-13T07:28:55.092 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:54.773629Z" 2022-09-13T07:28:55.093 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:28:55.094 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:28:55.094 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:28:55.095 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T07:28:55.095 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:28:55.096 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:28:55.099 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054134.773629_86.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:55.100 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:28:55.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:55.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:28:55.103 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:28:55.159 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:28:55.160 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:28:55.189 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:55.185+0000 7f9a579c8600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:55.190 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:28:55.186+0000 7f9a579c8600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:28:55.201 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:28:55.202 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:28:55.202 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:28:55.202 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:28:55.203 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:28:55.203 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:28:55.203 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:55.204 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:55.204 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:28:55.204 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:55.204 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:28:55.205 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:55.205 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:55.205 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:28:55.206 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:55.206 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:28:55.207 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:55.208 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:28:55.208 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:55.209 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:55.210 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:55.210 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:55.211 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:28:55.211 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:55.212 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:55.212 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:28:55.213 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:55.213 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:55.214 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:55.214 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:28:55.215 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:55.215 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:55.216 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:55.217 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:55.217 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:28:55.218 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:55.218 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:55.219 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054134.727027_82.1", 2022-09-13T07:28:55.220 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:55.220 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:55.221 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:55.221 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:54.727027Z", 2022-09-13T07:28:55.222 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:55.222 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:55.223 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:55.223 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:28:55.224 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:28:55.224 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:28:55.225 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:28:55.226 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:28:55.226 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:28:55.227 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:28:55.228 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:28:55.228 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:28:55.229 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:28:55.229 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:55.230 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:28:55.230 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:28:55.231 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:55.231 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:28:55.232 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:28:55.232 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:28:55.233 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:28:55.234 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:28:55Z" 2022-09-13T07:28:55.234 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:28:55.237 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054134.727027_82.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:54.727027Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:28:55Z" } 2022-09-13T07:28:55.238 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054134.727027_82.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:54.727027Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:28:55.239 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054134.773629_86.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:55.239 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054134.727027_82.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:28:55.240 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663054134.773629_86.1 target=1_1663054134.727027_82.1 2022-09-13T07:29:00.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:29:00.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:29:00.244 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:29:00.301 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:29:00.302 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:29:00.342 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:29:00.339+0000 7f0ed1f3a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:29:00.343 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:29:00.340+0000 7f0ed1f3a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:29:00.358 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:29:00.359 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:29:00.359 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:29:00.360 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:29:00.360 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:29:00.361 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:29:00.361 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:29:00.363 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:29:00.367 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:29:00.367 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:29:00.367 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:29:00.368 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:29:00.368 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:29:00.369 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:29:00.369 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:29:00.370 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:29:00.370 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:29:00.370 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:29:00.370 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:29:00.371 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:29:00.371 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:29:00.371 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:29:00.371 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:29:00.372 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:29:00.372 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:29:00.372 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:29:00.372 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:29:00.373 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:29:00.373 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:29:00.374 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:29:00.374 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:29:00.374 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:29:00.375 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:29:00.375 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:29:00.375 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:29:00.375 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:29:00.376 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:29:00.376 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054134.773629_86.1", 2022-09-13T07:29:00.376 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:29:00.376 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:29:00.377 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:29:00.377 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:54.773629Z", 2022-09-13T07:29:00.377 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:29:00.377 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:29:00.378 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:29:00.378 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:29:00.379 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:29:00.379 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:29:00.379 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:29:00.380 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:29:00.380 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:29:00.380 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:29:00.380 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:29:00.381 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:29:00.381 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:29:00.381 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:29:00.382 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:29:00.382 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:29:00.382 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:29:00.383 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:29:00.383 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:29:00.383 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:29:00.384 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:29:00.384 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:29:00Z" 2022-09-13T07:29:00.385 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:29:00.385 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054134.773629_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:54.773629Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:29:00Z" } 2022-09-13T07:29:00.386 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054134.773629_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:54.773629Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:29:00.387 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054134.773629_86.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:29:00.387 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054134.773629_86.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:29:00.388 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:29:00.388 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:29:00.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:29:00.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:29:00.390 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:29:00.453 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:29:00.453 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:29:00.483 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:00.479+0000 7fa2204d1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:29:00.484 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:00.480+0000 7fa2204d1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:29:00.498 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:29:00.499 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:29:00.499 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:29:00.500 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:29:00.500 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:29:00.501 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:29:00.501 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:29:00.501 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:29:00.502 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:29:00.502 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:29:00.503 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:29:00.503 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:29:00.504 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:29:00.504 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:29:00.504 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:29:00.505 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:29:00.505 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:29:00.505 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:29:00.505 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:29:00.506 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:29:00.506 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:29:00.506 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:29:00.506 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:29:00.507 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:29:00.507 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:29:00.507 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:29:00.507 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:29:00.508 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:29:00.508 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:29:00.508 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:29:00.508 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:29:00.509 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:29:00.509 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:29:00.509 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:29:00.509 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:29:00.510 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:29:00.510 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:29:00.510 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054134.773629_86.1", 2022-09-13T07:29:00.511 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:29:00.511 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:29:00.512 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:29:00.512 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:54.773629Z", 2022-09-13T07:29:00.512 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:29:00.512 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:29:00.513 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:29:00.513 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:29:00.514 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:29:00.514 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:29:00.515 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:29:00.515 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:29:00.516 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:29:00.516 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:29:00.516 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:29:00.516 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:29:00.517 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:29:00.517 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:29:00.518 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:29:00.518 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:29:00.518 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:29:00.518 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:29:00.519 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:29:00.519 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:29:00.519 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:29:00.520 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:29:00Z" 2022-09-13T07:29:00.520 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:29:00.522 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054134.773629_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:54.773629Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:29:00Z" } 2022-09-13T07:29:00.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054134.773629_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:54.773629Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:29:00.524 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054134.773629_86.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:29:00.525 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054134.773629_86.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:29:00.525 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:29:00.526 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:00.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:00.528 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:29:00.589 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:29:00.589 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:29:00.618 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:29:00.615+0000 7faecba0f600 1 waiting to reach incremental sync.. 2022-09-13T07:29:05.619 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:29:05.616+0000 7faecba0f600 1 waiting to reach incremental sync.. 2022-09-13T07:29:10.621 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:29:10.617+0000 7faecba0f600 1 bucket sync caught up with source: 2022-09-13T07:29:10.621 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2022-09-13T07:29:10.622 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2022-09-13T07:29:10.622 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:29:10.617+0000 7faecba0f600 0 bucket checkpoint complete 2022-09-13T07:29:10.636 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-13 zones={test-zone2, test-zone1} 2022-09-13T07:29:10.637 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:10.638 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:10.638 DEBUG:boto:Method: HEAD 2022-09-13T07:29:10.639 DEBUG:boto:Path: /yxvvuh-13/ 2022-09-13T07:29:10.639 DEBUG:boto:Data: 2022-09-13T07:29:10.640 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.640 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:10.641 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.641 DEBUG:boto:Params: {} 2022-09-13T07:29:10.642 DEBUG:boto:Token: None 2022-09-13T07:29:10.642 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/ 2022-09-13T07:29:10.643 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6tkUoUquCs/KyuEHFdyqY25RQFk= 2022-09-13T07:29:10.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6tkUoUquCs/KyuEHFdyqY25RQFk=', 'Content-Length': '0'} 2022-09-13T07:29:10.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023fee62ad776f000-0063203146-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.647 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:10.647 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:10.648 DEBUG:boto:Method: HEAD 2022-09-13T07:29:10.648 DEBUG:boto:Path: /yxvvuh-13/ 2022-09-13T07:29:10.649 DEBUG:boto:Data: 2022-09-13T07:29:10.649 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.650 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:10.650 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.651 DEBUG:boto:Params: {} 2022-09-13T07:29:10.651 DEBUG:boto:Token: None 2022-09-13T07:29:10.652 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/ 2022-09-13T07:29:10.653 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6tkUoUquCs/KyuEHFdyqY25RQFk= 2022-09-13T07:29:10.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6tkUoUquCs/KyuEHFdyqY25RQFk=', 'Content-Length': '0'} 2022-09-13T07:29:10.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000721cde55c21fc28-0063203146-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.657 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:29:10.657 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:10.658 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:10.658 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:10.659 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:10.659 DEBUG:boto:Method: GET 2022-09-13T07:29:10.660 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:10.660 DEBUG:boto:Data: 2022-09-13T07:29:10.661 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.661 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:10.662 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.662 DEBUG:boto:Params: {} 2022-09-13T07:29:10.663 DEBUG:boto:Token: None 2022-09-13T07:29:10.664 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/?versions 2022-09-13T07:29:10.664 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:G8CqJ2q+dqHtxNvfUFBBf1rPAQI= 2022-09-13T07:29:10.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:G8CqJ2q+dqHtxNvfUFBBf1rPAQI=', 'Content-Length': '0'} 2022-09-13T07:29:10.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000413654a62706b378-0063203146-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.669 DEBUG:boto:b'yxvvuh-13999falsemyobjnulltrue2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:10.670 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:29:10.670 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:29:10.671 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:10.671 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:10.672 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:10.672 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:10.673 DEBUG:boto:Method: GET 2022-09-13T07:29:10.673 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:10.674 DEBUG:boto:Data: 2022-09-13T07:29:10.674 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.675 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:10.675 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.676 DEBUG:boto:Params: {} 2022-09-13T07:29:10.677 DEBUG:boto:Token: None 2022-09-13T07:29:10.677 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/?versions 2022-09-13T07:29:10.678 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:G8CqJ2q+dqHtxNvfUFBBf1rPAQI= 2022-09-13T07:29:10.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:G8CqJ2q+dqHtxNvfUFBBf1rPAQI=', 'Content-Length': '0'} 2022-09-13T07:29:10.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016cd13a60855f905-0063203146-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.681 DEBUG:boto:b'yxvvuh-13999falsemyobjnulltrue2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:10.682 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:29:10.683 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:10.684 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:10.684 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:10.685 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:10.685 DEBUG:boto:Method: GET 2022-09-13T07:29:10.686 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:10.686 DEBUG:boto:Data: 2022-09-13T07:29:10.687 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.688 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:10.688 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.689 DEBUG:boto:Params: {} 2022-09-13T07:29:10.689 DEBUG:boto:Token: None 2022-09-13T07:29:10.690 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/?versions 2022-09-13T07:29:10.690 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:G8CqJ2q+dqHtxNvfUFBBf1rPAQI= 2022-09-13T07:29:10.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:G8CqJ2q+dqHtxNvfUFBBf1rPAQI=', 'Content-Length': '0'} 2022-09-13T07:29:10.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e490fa5542059827-0063203146-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.695 DEBUG:boto:b'yxvvuh-13999falsemyobjnulltrue2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:10.695 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:10.696 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:10.696 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:10.697 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:10.698 DEBUG:boto:Method: GET 2022-09-13T07:29:10.699 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:10.699 DEBUG:boto:Data: 2022-09-13T07:29:10.700 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.701 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:10.701 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.702 DEBUG:boto:Params: {} 2022-09-13T07:29:10.702 DEBUG:boto:Token: None 2022-09-13T07:29:10.703 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/?versions 2022-09-13T07:29:10.704 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:G8CqJ2q+dqHtxNvfUFBBf1rPAQI= 2022-09-13T07:29:10.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:G8CqJ2q+dqHtxNvfUFBBf1rPAQI=', 'Content-Length': '0'} 2022-09-13T07:29:10.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dc5014bea9daabf-0063203146-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.707 DEBUG:boto:b'yxvvuh-13999falsemyobjnulltrue2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:10.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:10.709 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:10.709 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:10.710 DEBUG:boto:path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.711 DEBUG:boto:auth_path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.711 DEBUG:boto:Method: GET 2022-09-13T07:29:10.712 DEBUG:boto:Path: /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.712 DEBUG:boto:Data: 2022-09-13T07:29:10.713 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.713 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:10.714 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.714 DEBUG:boto:Params: {} 2022-09-13T07:29:10.715 DEBUG:boto:Token: None 2022-09-13T07:29:10.715 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.716 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZoNPvMAOQu1CQWWMp4E75PWpCI8= 2022-09-13T07:29:10.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZoNPvMAOQu1CQWWMp4E75PWpCI8=', 'Content-Length': '0'} 2022-09-13T07:29:10.719 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f58c55c0ca765acb-0063203146-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.720 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:10.720 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:10.721 DEBUG:boto:path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.721 DEBUG:boto:auth_path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.722 DEBUG:boto:Method: GET 2022-09-13T07:29:10.722 DEBUG:boto:Path: /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.723 DEBUG:boto:Data: 2022-09-13T07:29:10.723 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.724 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:10.724 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.725 DEBUG:boto:Params: {} 2022-09-13T07:29:10.725 DEBUG:boto:Token: None 2022-09-13T07:29:10.726 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.727 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZoNPvMAOQu1CQWWMp4E75PWpCI8= 2022-09-13T07:29:10.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZoNPvMAOQu1CQWWMp4E75PWpCI8=', 'Content-Length': '0'} 2022-09-13T07:29:10.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005b4a44e164bb00c-0063203146-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.730 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:10.731 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:10.731 DEBUG:boto:path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.732 DEBUG:boto:auth_path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.732 DEBUG:boto:Method: HEAD 2022-09-13T07:29:10.733 DEBUG:boto:Path: /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.733 DEBUG:boto:Data: 2022-09-13T07:29:10.734 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.734 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:10.735 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.735 DEBUG:boto:Params: {} 2022-09-13T07:29:10.736 DEBUG:boto:Token: None 2022-09-13T07:29:10.737 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.737 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UgF7Yla7VEw0TAVLKROUQRiplM4= 2022-09-13T07:29:10.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UgF7Yla7VEw0TAVLKROUQRiplM4=', 'Content-Length': '0'} 2022-09-13T07:29:10.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002201e8daceb5c4d6-0063203146-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.741 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:10.741 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:10.742 DEBUG:boto:path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.742 DEBUG:boto:auth_path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.743 DEBUG:boto:Method: HEAD 2022-09-13T07:29:10.743 DEBUG:boto:Path: /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.744 DEBUG:boto:Data: 2022-09-13T07:29:10.744 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.744 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:10.745 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.745 DEBUG:boto:Params: {} 2022-09-13T07:29:10.746 DEBUG:boto:Token: None 2022-09-13T07:29:10.747 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.747 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UgF7Yla7VEw0TAVLKROUQRiplM4= 2022-09-13T07:29:10.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UgF7Yla7VEw0TAVLKROUQRiplM4=', 'Content-Length': '0'} 2022-09-13T07:29:10.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043d1f8b60cf852c5-0063203146-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:10.751 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:10.752 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:10.752 DEBUG:boto:path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.753 DEBUG:boto:auth_path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.754 DEBUG:boto:Method: GET 2022-09-13T07:29:10.754 DEBUG:boto:Path: /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.755 DEBUG:boto:Data: 2022-09-13T07:29:10.755 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.756 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:10.756 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.757 DEBUG:boto:Params: {} 2022-09-13T07:29:10.757 DEBUG:boto:Token: None 2022-09-13T07:29:10.758 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.759 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZoNPvMAOQu1CQWWMp4E75PWpCI8= 2022-09-13T07:29:10.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZoNPvMAOQu1CQWWMp4E75PWpCI8=', 'Content-Length': '0'} 2022-09-13T07:29:10.761 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005150439a93ccf429-0063203146-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.802 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:10.803 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:10.803 DEBUG:boto:path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.804 DEBUG:boto:auth_path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.804 DEBUG:boto:Method: GET 2022-09-13T07:29:10.805 DEBUG:boto:Path: /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.805 DEBUG:boto:Data: 2022-09-13T07:29:10.806 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.806 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:10.807 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.807 DEBUG:boto:Params: {} 2022-09-13T07:29:10.808 DEBUG:boto:Token: None 2022-09-13T07:29:10.809 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:10.809 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZoNPvMAOQu1CQWWMp4E75PWpCI8= 2022-09-13T07:29:10.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZoNPvMAOQu1CQWWMp4E75PWpCI8=', 'Content-Length': '0'} 2022-09-13T07:29:10.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ee4099e5ed7a71f-0063203146-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.813 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:10.813 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:10.814 DEBUG:boto:path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:10.814 DEBUG:boto:auth_path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:10.815 DEBUG:boto:Method: GET 2022-09-13T07:29:10.815 DEBUG:boto:Path: /yxvvuh-13/?prefix=myobj 2022-09-13T07:29:10.816 DEBUG:boto:Data: 2022-09-13T07:29:10.816 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.817 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:10.817 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.818 DEBUG:boto:Params: {} 2022-09-13T07:29:10.818 DEBUG:boto:Token: None 2022-09-13T07:29:10.819 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/ 2022-09-13T07:29:10.819 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:17ym4YB4tNt1tu4ACoP68VfE5yI= 2022-09-13T07:29:10.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:17ym4YB4tNt1tu4ACoP68VfE5yI=', 'Content-Length': '0'} 2022-09-13T07:29:10.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049421911fff0c6f1-0063203146-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.823 DEBUG:boto:b'yxvvuh-13myobj1000falsemyobj2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:10.824 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:10.825 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:10.825 DEBUG:boto:path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:10.826 DEBUG:boto:auth_path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:10.826 DEBUG:boto:Method: GET 2022-09-13T07:29:10.827 DEBUG:boto:Path: /yxvvuh-13/?prefix=myobj 2022-09-13T07:29:10.827 DEBUG:boto:Data: 2022-09-13T07:29:10.828 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.828 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:10.829 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.829 DEBUG:boto:Params: {} 2022-09-13T07:29:10.830 DEBUG:boto:Token: None 2022-09-13T07:29:10.830 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/ 2022-09-13T07:29:10.831 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:17ym4YB4tNt1tu4ACoP68VfE5yI= 2022-09-13T07:29:10.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:17ym4YB4tNt1tu4ACoP68VfE5yI=', 'Content-Length': '0'} 2022-09-13T07:29:10.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004015ba71a4b141a3-0063203146-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.835 DEBUG:boto:b'yxvvuh-13myobj1000falsemyobj2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:10.836 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:10.836 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:10.837 DEBUG:boto:Method: HEAD 2022-09-13T07:29:10.838 DEBUG:boto:Path: /yxvvuh-13/myobj 2022-09-13T07:29:10.839 DEBUG:boto:Data: 2022-09-13T07:29:10.839 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.840 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:10.841 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.841 DEBUG:boto:Params: {} 2022-09-13T07:29:10.842 DEBUG:boto:Token: None 2022-09-13T07:29:10.842 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/myobj 2022-09-13T07:29:10.843 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:morS6Cf4c5WUbgAumx6srrWEiXE= 2022-09-13T07:29:10.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:morS6Cf4c5WUbgAumx6srrWEiXE=', 'Content-Length': '0'} 2022-09-13T07:29:10.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089720ed51891a118-0063203146-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.847 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:10.847 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:10.848 DEBUG:boto:Method: HEAD 2022-09-13T07:29:10.848 DEBUG:boto:Path: /yxvvuh-13/myobj 2022-09-13T07:29:10.849 DEBUG:boto:Data: 2022-09-13T07:29:10.850 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.851 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:10.852 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.853 DEBUG:boto:Params: {} 2022-09-13T07:29:10.854 DEBUG:boto:Token: None 2022-09-13T07:29:10.855 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/myobj 2022-09-13T07:29:10.856 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:morS6Cf4c5WUbgAumx6srrWEiXE= 2022-09-13T07:29:10.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:morS6Cf4c5WUbgAumx6srrWEiXE=', 'Content-Length': '0'} 2022-09-13T07:29:10.860 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d1cd3cecab896c2-0063203146-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:10.861 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:10.862 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:10.862 DEBUG:boto:Method: GET 2022-09-13T07:29:10.863 DEBUG:boto:Path: /yxvvuh-13/myobj 2022-09-13T07:29:10.865 DEBUG:boto:Data: 2022-09-13T07:29:10.866 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.866 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:10.867 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.867 DEBUG:boto:Params: {} 2022-09-13T07:29:10.868 DEBUG:boto:Token: None 2022-09-13T07:29:10.869 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/myobj 2022-09-13T07:29:10.869 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:47X1dPRGcjUFPJW5dUvyL98lSmI= 2022-09-13T07:29:10.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:47X1dPRGcjUFPJW5dUvyL98lSmI=', 'Content-Length': '0'} 2022-09-13T07:29:10.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002175dc18fbb30fd8-0063203146-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.914 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:10.915 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:10.916 DEBUG:boto:Method: GET 2022-09-13T07:29:10.917 DEBUG:boto:Path: /yxvvuh-13/myobj 2022-09-13T07:29:10.918 DEBUG:boto:Data: 2022-09-13T07:29:10.919 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.920 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:10.921 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.922 DEBUG:boto:Params: {} 2022-09-13T07:29:10.923 DEBUG:boto:Token: None 2022-09-13T07:29:10.924 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/myobj 2022-09-13T07:29:10.925 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:47X1dPRGcjUFPJW5dUvyL98lSmI= 2022-09-13T07:29:10.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:47X1dPRGcjUFPJW5dUvyL98lSmI=', 'Content-Length': '0'} 2022-09-13T07:29:10.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca250da2e4b3b9f4-0063203146-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.931 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:10.932 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:10.933 DEBUG:boto:path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:10.934 DEBUG:boto:auth_path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:10.935 DEBUG:boto:Method: GET 2022-09-13T07:29:10.936 DEBUG:boto:Path: /yxvvuh-13/?prefix=myobj 2022-09-13T07:29:10.937 DEBUG:boto:Data: 2022-09-13T07:29:10.938 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.939 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:10.940 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.941 DEBUG:boto:Params: {} 2022-09-13T07:29:10.942 DEBUG:boto:Token: None 2022-09-13T07:29:10.943 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/ 2022-09-13T07:29:10.945 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:17ym4YB4tNt1tu4ACoP68VfE5yI= 2022-09-13T07:29:10.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:17ym4YB4tNt1tu4ACoP68VfE5yI=', 'Content-Length': '0'} 2022-09-13T07:29:10.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091531f988b348c30-0063203146-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.950 DEBUG:boto:b'yxvvuh-13myobj1000falsemyobj2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:10.952 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:10.953 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:10.954 DEBUG:boto:path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:10.955 DEBUG:boto:auth_path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:10.956 DEBUG:boto:Method: GET 2022-09-13T07:29:10.957 DEBUG:boto:Path: /yxvvuh-13/?prefix=myobj 2022-09-13T07:29:10.958 DEBUG:boto:Data: 2022-09-13T07:29:10.959 DEBUG:boto:Headers: {} 2022-09-13T07:29:10.960 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:10.961 DEBUG:boto:Port: 8000 2022-09-13T07:29:10.962 DEBUG:boto:Params: {} 2022-09-13T07:29:10.963 DEBUG:boto:Token: None 2022-09-13T07:29:10.965 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:10 GMT /yxvvuh-13/ 2022-09-13T07:29:10.966 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:17ym4YB4tNt1tu4ACoP68VfE5yI= 2022-09-13T07:29:10.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:17ym4YB4tNt1tu4ACoP68VfE5yI=', 'Content-Length': '0'} 2022-09-13T07:29:10.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052e5823067f7cf25-0063203146-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:10.971 DEBUG:boto:b'yxvvuh-13myobj1000falsemyobj2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:10.972 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-13 zones={test-zone2, test-zone1} 2022-09-13T07:29:10.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:10.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:10.973 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:29:11.039 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:29:11.040 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:29:11.087 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:11.084+0000 7f80ef532600 1 waiting to reach incremental sync.. 2022-09-13T07:29:16.089 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:16.086+0000 7f80ef532600 1 bucket sync caught up with source: 2022-09-13T07:29:16.090 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2022-09-13T07:29:16.090 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2022-09-13T07:29:16.090 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:16.086+0000 7f80ef532600 0 bucket checkpoint complete 2022-09-13T07:29:16.108 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-13 zones={test-zone3, test-zone1} 2022-09-13T07:29:16.109 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:16.110 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:16.110 DEBUG:boto:Method: HEAD 2022-09-13T07:29:16.111 DEBUG:boto:Path: /yxvvuh-13/ 2022-09-13T07:29:16.111 DEBUG:boto:Data: 2022-09-13T07:29:16.112 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.112 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:16.113 DEBUG:boto:Port: 8001 2022-09-13T07:29:16.113 DEBUG:boto:Params: {} 2022-09-13T07:29:16.114 DEBUG:boto:Token: None 2022-09-13T07:29:16.114 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/ 2022-09-13T07:29:16.115 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WQt5B0AAju8PzHHHm0WSE1yZOO8= 2022-09-13T07:29:16.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WQt5B0AAju8PzHHHm0WSE1yZOO8=', 'Content-Length': '0'} 2022-09-13T07:29:16.119 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b20be4370d3ae0b7-006320314c-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.120 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:16.120 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:16.121 DEBUG:boto:Method: HEAD 2022-09-13T07:29:16.121 DEBUG:boto:Path: /yxvvuh-13/ 2022-09-13T07:29:16.122 DEBUG:boto:Data: 2022-09-13T07:29:16.122 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.123 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.123 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.124 DEBUG:boto:Params: {} 2022-09-13T07:29:16.125 DEBUG:boto:Token: None 2022-09-13T07:29:16.125 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/ 2022-09-13T07:29:16.126 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WQt5B0AAju8PzHHHm0WSE1yZOO8= 2022-09-13T07:29:16.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WQt5B0AAju8PzHHHm0WSE1yZOO8=', 'Content-Length': '0'} 2022-09-13T07:29:16.130 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073975caf5e30580a-006320314c-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.131 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:29:16.131 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:16.132 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:16.133 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:16.133 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:16.134 DEBUG:boto:Method: GET 2022-09-13T07:29:16.134 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:16.135 DEBUG:boto:Data: 2022-09-13T07:29:16.135 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.137 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:16.137 DEBUG:boto:Port: 8001 2022-09-13T07:29:16.138 DEBUG:boto:Params: {} 2022-09-13T07:29:16.139 DEBUG:boto:Token: None 2022-09-13T07:29:16.140 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/?versions 2022-09-13T07:29:16.140 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5g3gSsgP3BRTTb6y4qTc3RfAXeI= 2022-09-13T07:29:16.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5g3gSsgP3BRTTb6y4qTc3RfAXeI=', 'Content-Length': '0'} 2022-09-13T07:29:16.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003d46b60ce59b0f8-006320314c-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.190 DEBUG:boto:b'yxvvuh-13999falsemyobjnulltrue2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:16.192 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:29:16.193 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:29:16.194 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:16.195 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:16.195 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:16.196 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:16.197 DEBUG:boto:Method: GET 2022-09-13T07:29:16.198 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:16.199 DEBUG:boto:Data: 2022-09-13T07:29:16.200 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.201 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.202 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.202 DEBUG:boto:Params: {} 2022-09-13T07:29:16.203 DEBUG:boto:Token: None 2022-09-13T07:29:16.203 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/?versions 2022-09-13T07:29:16.204 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5g3gSsgP3BRTTb6y4qTc3RfAXeI= 2022-09-13T07:29:16.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5g3gSsgP3BRTTb6y4qTc3RfAXeI=', 'Content-Length': '0'} 2022-09-13T07:29:16.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042085349d907fa12-006320314c-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.208 DEBUG:boto:b'yxvvuh-13999falsemyobjnulltrue2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:16.209 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:29:16.210 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:16.210 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:16.211 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:16.211 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:16.212 DEBUG:boto:Method: GET 2022-09-13T07:29:16.213 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:16.214 DEBUG:boto:Data: 2022-09-13T07:29:16.214 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.215 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:16.215 DEBUG:boto:Port: 8001 2022-09-13T07:29:16.216 DEBUG:boto:Params: {} 2022-09-13T07:29:16.216 DEBUG:boto:Token: None 2022-09-13T07:29:16.217 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/?versions 2022-09-13T07:29:16.217 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5g3gSsgP3BRTTb6y4qTc3RfAXeI= 2022-09-13T07:29:16.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5g3gSsgP3BRTTb6y4qTc3RfAXeI=', 'Content-Length': '0'} 2022-09-13T07:29:16.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c0f208d9a08d3f3-006320314c-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.222 DEBUG:boto:b'yxvvuh-13999falsemyobjnulltrue2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:16.224 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:16.225 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:16.226 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:16.227 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:16.227 DEBUG:boto:Method: GET 2022-09-13T07:29:16.228 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:16.228 DEBUG:boto:Data: 2022-09-13T07:29:16.229 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.229 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.230 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.230 DEBUG:boto:Params: {} 2022-09-13T07:29:16.231 DEBUG:boto:Token: None 2022-09-13T07:29:16.232 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/?versions 2022-09-13T07:29:16.232 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5g3gSsgP3BRTTb6y4qTc3RfAXeI= 2022-09-13T07:29:16.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5g3gSsgP3BRTTb6y4qTc3RfAXeI=', 'Content-Length': '0'} 2022-09-13T07:29:16.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022939b0a070195a7-006320314c-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.278 DEBUG:boto:b'yxvvuh-13999falsemyobjnulltrue2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:16.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:16.281 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:16.282 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:16.283 DEBUG:boto:path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.284 DEBUG:boto:auth_path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.285 DEBUG:boto:Method: GET 2022-09-13T07:29:16.287 DEBUG:boto:Path: /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.287 DEBUG:boto:Data: 2022-09-13T07:29:16.288 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.288 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:16.289 DEBUG:boto:Port: 8001 2022-09-13T07:29:16.289 DEBUG:boto:Params: {} 2022-09-13T07:29:16.290 DEBUG:boto:Token: None 2022-09-13T07:29:16.291 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.292 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zVPvDbtoADFtKsNqpGhMHPQO3v8= 2022-09-13T07:29:16.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zVPvDbtoADFtKsNqpGhMHPQO3v8=', 'Content-Length': '0'} 2022-09-13T07:29:16.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef14ce26c18511ff-006320314c-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.296 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:16.296 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:16.297 DEBUG:boto:path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.297 DEBUG:boto:auth_path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.298 DEBUG:boto:Method: GET 2022-09-13T07:29:16.299 DEBUG:boto:Path: /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.299 DEBUG:boto:Data: 2022-09-13T07:29:16.300 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.300 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.301 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.301 DEBUG:boto:Params: {} 2022-09-13T07:29:16.302 DEBUG:boto:Token: None 2022-09-13T07:29:16.303 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.303 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zVPvDbtoADFtKsNqpGhMHPQO3v8= 2022-09-13T07:29:16.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zVPvDbtoADFtKsNqpGhMHPQO3v8=', 'Content-Length': '0'} 2022-09-13T07:29:16.306 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fea5de1bfc9e5f6e-006320314c-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.307 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:16.307 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:16.308 DEBUG:boto:path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.308 DEBUG:boto:auth_path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.309 DEBUG:boto:Method: HEAD 2022-09-13T07:29:16.309 DEBUG:boto:Path: /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.310 DEBUG:boto:Data: 2022-09-13T07:29:16.311 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.311 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:16.312 DEBUG:boto:Port: 8001 2022-09-13T07:29:16.312 DEBUG:boto:Params: {} 2022-09-13T07:29:16.313 DEBUG:boto:Token: None 2022-09-13T07:29:16.314 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.314 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1l9vvQKp6lwzyy3rHT5bfWQ7vdk= 2022-09-13T07:29:16.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1l9vvQKp6lwzyy3rHT5bfWQ7vdk=', 'Content-Length': '0'} 2022-09-13T07:29:16.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002230b8945089bd25-006320314c-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.317 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:16.318 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:16.318 DEBUG:boto:path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.319 DEBUG:boto:auth_path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.319 DEBUG:boto:Method: HEAD 2022-09-13T07:29:16.320 DEBUG:boto:Path: /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.320 DEBUG:boto:Data: 2022-09-13T07:29:16.321 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.322 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.322 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.323 DEBUG:boto:Params: {} 2022-09-13T07:29:16.323 DEBUG:boto:Token: None 2022-09-13T07:29:16.324 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.325 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1l9vvQKp6lwzyy3rHT5bfWQ7vdk= 2022-09-13T07:29:16.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1l9vvQKp6lwzyy3rHT5bfWQ7vdk=', 'Content-Length': '0'} 2022-09-13T07:29:16.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e44abb52bb7919a6-006320314c-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:16.333 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:16.334 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:16.334 DEBUG:boto:path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.335 DEBUG:boto:auth_path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.335 DEBUG:boto:Method: GET 2022-09-13T07:29:16.336 DEBUG:boto:Path: /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.336 DEBUG:boto:Data: 2022-09-13T07:29:16.338 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.339 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:16.340 DEBUG:boto:Port: 8001 2022-09-13T07:29:16.341 DEBUG:boto:Params: {} 2022-09-13T07:29:16.342 DEBUG:boto:Token: None 2022-09-13T07:29:16.343 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.344 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zVPvDbtoADFtKsNqpGhMHPQO3v8= 2022-09-13T07:29:16.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zVPvDbtoADFtKsNqpGhMHPQO3v8=', 'Content-Length': '0'} 2022-09-13T07:29:16.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004491f52502a418a0-006320314c-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.349 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:16.350 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:16.351 DEBUG:boto:path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.351 DEBUG:boto:auth_path=/yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.352 DEBUG:boto:Method: GET 2022-09-13T07:29:16.352 DEBUG:boto:Path: /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.353 DEBUG:boto:Data: 2022-09-13T07:29:16.353 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.354 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.354 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.355 DEBUG:boto:Params: {} 2022-09-13T07:29:16.356 DEBUG:boto:Token: None 2022-09-13T07:29:16.356 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/myobj?versionId=null 2022-09-13T07:29:16.357 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zVPvDbtoADFtKsNqpGhMHPQO3v8= 2022-09-13T07:29:16.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zVPvDbtoADFtKsNqpGhMHPQO3v8=', 'Content-Length': '0'} 2022-09-13T07:29:16.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df56de2b82d0324c-006320314c-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.360 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:16.361 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:16.361 DEBUG:boto:path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:16.362 DEBUG:boto:auth_path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:16.363 DEBUG:boto:Method: GET 2022-09-13T07:29:16.363 DEBUG:boto:Path: /yxvvuh-13/?prefix=myobj 2022-09-13T07:29:16.364 DEBUG:boto:Data: 2022-09-13T07:29:16.364 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.365 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:16.365 DEBUG:boto:Port: 8001 2022-09-13T07:29:16.366 DEBUG:boto:Params: {} 2022-09-13T07:29:16.366 DEBUG:boto:Token: None 2022-09-13T07:29:16.367 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/ 2022-09-13T07:29:16.368 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rO4AsNiYUV5Vra7rBYmF2vrOs94= 2022-09-13T07:29:16.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rO4AsNiYUV5Vra7rBYmF2vrOs94=', 'Content-Length': '0'} 2022-09-13T07:29:16.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002df0cb477fa4e45-006320314c-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.372 DEBUG:boto:b'yxvvuh-13myobj1000falsemyobj2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:16.373 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:16.373 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:16.374 DEBUG:boto:path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:16.374 DEBUG:boto:auth_path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:16.375 DEBUG:boto:Method: GET 2022-09-13T07:29:16.375 DEBUG:boto:Path: /yxvvuh-13/?prefix=myobj 2022-09-13T07:29:16.376 DEBUG:boto:Data: 2022-09-13T07:29:16.376 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.377 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.377 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.378 DEBUG:boto:Params: {} 2022-09-13T07:29:16.379 DEBUG:boto:Token: None 2022-09-13T07:29:16.379 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/ 2022-09-13T07:29:16.380 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rO4AsNiYUV5Vra7rBYmF2vrOs94= 2022-09-13T07:29:16.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rO4AsNiYUV5Vra7rBYmF2vrOs94=', 'Content-Length': '0'} 2022-09-13T07:29:16.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064804bcbb3e0baa7-006320314c-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.387 DEBUG:boto:b'yxvvuh-13myobj1000falsemyobj2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:16.388 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:16.389 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:16.390 DEBUG:boto:Method: HEAD 2022-09-13T07:29:16.391 DEBUG:boto:Path: /yxvvuh-13/myobj 2022-09-13T07:29:16.392 DEBUG:boto:Data: 2022-09-13T07:29:16.393 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.394 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:16.395 DEBUG:boto:Port: 8001 2022-09-13T07:29:16.396 DEBUG:boto:Params: {} 2022-09-13T07:29:16.397 DEBUG:boto:Token: None 2022-09-13T07:29:16.398 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/myobj 2022-09-13T07:29:16.399 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Irm5XA/hw+rRUWB5LIqCsjHPOX8= 2022-09-13T07:29:16.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Irm5XA/hw+rRUWB5LIqCsjHPOX8=', 'Content-Length': '0'} 2022-09-13T07:29:16.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f9bdf76a66941f1-006320314c-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.404 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:16.405 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:16.406 DEBUG:boto:Method: HEAD 2022-09-13T07:29:16.407 DEBUG:boto:Path: /yxvvuh-13/myobj 2022-09-13T07:29:16.408 DEBUG:boto:Data: 2022-09-13T07:29:16.409 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.411 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.412 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.413 DEBUG:boto:Params: {} 2022-09-13T07:29:16.414 DEBUG:boto:Token: None 2022-09-13T07:29:16.415 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/myobj 2022-09-13T07:29:16.416 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Irm5XA/hw+rRUWB5LIqCsjHPOX8= 2022-09-13T07:29:16.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Irm5XA/hw+rRUWB5LIqCsjHPOX8=', 'Content-Length': '0'} 2022-09-13T07:29:16.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061afd88538a251e3-006320314c-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:16.422 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:16.423 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:16.424 DEBUG:boto:Method: GET 2022-09-13T07:29:16.425 DEBUG:boto:Path: /yxvvuh-13/myobj 2022-09-13T07:29:16.426 DEBUG:boto:Data: 2022-09-13T07:29:16.427 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.428 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:16.430 DEBUG:boto:Port: 8001 2022-09-13T07:29:16.431 DEBUG:boto:Params: {} 2022-09-13T07:29:16.432 DEBUG:boto:Token: None 2022-09-13T07:29:16.434 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/myobj 2022-09-13T07:29:16.435 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:waaRJk3bsmc/6MEwaQNolofMx8g= 2022-09-13T07:29:16.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:waaRJk3bsmc/6MEwaQNolofMx8g=', 'Content-Length': '0'} 2022-09-13T07:29:16.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df54255f1d7621ac-006320314c-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.482 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:16.483 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:16.484 DEBUG:boto:Method: GET 2022-09-13T07:29:16.485 DEBUG:boto:Path: /yxvvuh-13/myobj 2022-09-13T07:29:16.485 DEBUG:boto:Data: 2022-09-13T07:29:16.486 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.486 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.487 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.487 DEBUG:boto:Params: {} 2022-09-13T07:29:16.488 DEBUG:boto:Token: None 2022-09-13T07:29:16.489 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/myobj 2022-09-13T07:29:16.489 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:waaRJk3bsmc/6MEwaQNolofMx8g= 2022-09-13T07:29:16.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:waaRJk3bsmc/6MEwaQNolofMx8g=', 'Content-Length': '0'} 2022-09-13T07:29:16.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e64d185705c14c83-006320314c-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.493 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:16.494 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:16.495 DEBUG:boto:path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:16.495 DEBUG:boto:auth_path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:16.496 DEBUG:boto:Method: GET 2022-09-13T07:29:16.496 DEBUG:boto:Path: /yxvvuh-13/?prefix=myobj 2022-09-13T07:29:16.497 DEBUG:boto:Data: 2022-09-13T07:29:16.497 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.498 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:16.498 DEBUG:boto:Port: 8001 2022-09-13T07:29:16.499 DEBUG:boto:Params: {} 2022-09-13T07:29:16.499 DEBUG:boto:Token: None 2022-09-13T07:29:16.500 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/ 2022-09-13T07:29:16.501 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rO4AsNiYUV5Vra7rBYmF2vrOs94= 2022-09-13T07:29:16.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rO4AsNiYUV5Vra7rBYmF2vrOs94=', 'Content-Length': '0'} 2022-09-13T07:29:16.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044ab872e9fc32bf9-006320314c-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.505 DEBUG:boto:b'yxvvuh-13myobj1000falsemyobj2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:16.506 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:16.507 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:16.507 DEBUG:boto:path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:16.508 DEBUG:boto:auth_path=/yxvvuh-13/?prefix=myobj 2022-09-13T07:29:16.508 DEBUG:boto:Method: GET 2022-09-13T07:29:16.509 DEBUG:boto:Path: /yxvvuh-13/?prefix=myobj 2022-09-13T07:29:16.510 DEBUG:boto:Data: 2022-09-13T07:29:16.510 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.511 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.512 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.512 DEBUG:boto:Params: {} 2022-09-13T07:29:16.513 DEBUG:boto:Token: None 2022-09-13T07:29:16.513 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-13/ 2022-09-13T07:29:16.514 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rO4AsNiYUV5Vra7rBYmF2vrOs94= 2022-09-13T07:29:16.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rO4AsNiYUV5Vra7rBYmF2vrOs94=', 'Content-Length': '0'} 2022-09-13T07:29:16.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007729cfddd77672a5-006320314c-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.518 DEBUG:boto:b'yxvvuh-13myobj1000falsemyobj2022-09-13T07:28:54.819Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:16.519 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-13 zones={test-zone3, test-zone1} 2022-09-13T07:29:16.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:16.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:16.521 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:29:16.589 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:29:16.589 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:29:16.629 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:16.625+0000 7faa5f0d9600 1 bucket sync caught up with source: 2022-09-13T07:29:16.630 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2022-09-13T07:29:16.631 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2022-09-13T07:29:16.631 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:16.625+0000 7faa5f0d9600 0 bucket checkpoint complete 2022-09-13T07:29:16.642 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-14 zones={test-zone1, test-zone2} 2022-09-13T07:29:16.643 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:16.643 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:16.644 DEBUG:boto:Method: HEAD 2022-09-13T07:29:16.644 DEBUG:boto:Path: /yxvvuh-14/ 2022-09-13T07:29:16.645 DEBUG:boto:Data: 2022-09-13T07:29:16.645 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.646 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.647 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.647 DEBUG:boto:Params: {} 2022-09-13T07:29:16.648 DEBUG:boto:Token: None 2022-09-13T07:29:16.648 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/ 2022-09-13T07:29:16.649 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:u+psIZ7gwVq7mK/CLJ49MgdRiu8= 2022-09-13T07:29:16.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:u+psIZ7gwVq7mK/CLJ49MgdRiu8=', 'Content-Length': '0'} 2022-09-13T07:29:16.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021aad800a7eb5779-006320314c-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.653 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:16.653 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:16.654 DEBUG:boto:Method: HEAD 2022-09-13T07:29:16.654 DEBUG:boto:Path: /yxvvuh-14/ 2022-09-13T07:29:16.655 DEBUG:boto:Data: 2022-09-13T07:29:16.656 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.656 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:16.657 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.657 DEBUG:boto:Params: {} 2022-09-13T07:29:16.658 DEBUG:boto:Token: None 2022-09-13T07:29:16.659 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/ 2022-09-13T07:29:16.659 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:u+psIZ7gwVq7mK/CLJ49MgdRiu8= 2022-09-13T07:29:16.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:u+psIZ7gwVq7mK/CLJ49MgdRiu8=', 'Content-Length': '0'} 2022-09-13T07:29:16.662 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000707cdee16a278d4f-006320314c-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.663 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:29:16.664 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:16.664 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:16.665 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:16.667 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:16.667 DEBUG:boto:Method: GET 2022-09-13T07:29:16.668 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:16.668 DEBUG:boto:Data: 2022-09-13T07:29:16.669 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.669 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.670 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.670 DEBUG:boto:Params: {} 2022-09-13T07:29:16.671 DEBUG:boto:Token: None 2022-09-13T07:29:16.672 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/?versions 2022-09-13T07:29:16.673 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NilunrOx831gNzOWjpiKPoimzsk= 2022-09-13T07:29:16.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NilunrOx831gNzOWjpiKPoimzsk=', 'Content-Length': '0'} 2022-09-13T07:29:16.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025e12c8a2ce2280a-006320314c-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.677 DEBUG:boto:b'yxvvuh-14999falsemyobjnulltrue2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:16.677 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:29:16.678 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:29:16.678 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:16.679 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:16.679 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:16.680 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:16.681 DEBUG:boto:Method: GET 2022-09-13T07:29:16.682 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:16.682 DEBUG:boto:Data: 2022-09-13T07:29:16.683 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.683 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:16.684 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.684 DEBUG:boto:Params: {} 2022-09-13T07:29:16.685 DEBUG:boto:Token: None 2022-09-13T07:29:16.686 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/?versions 2022-09-13T07:29:16.686 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NilunrOx831gNzOWjpiKPoimzsk= 2022-09-13T07:29:16.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NilunrOx831gNzOWjpiKPoimzsk=', 'Content-Length': '0'} 2022-09-13T07:29:16.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051d3f0af9b6ebbc2-006320314c-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.690 DEBUG:boto:b'yxvvuh-14999falsemyobjnulltrue2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:16.691 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:29:16.692 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:16.693 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:16.694 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:16.695 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:16.696 DEBUG:boto:Method: GET 2022-09-13T07:29:16.697 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:16.697 DEBUG:boto:Data: 2022-09-13T07:29:16.698 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.699 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.700 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.701 DEBUG:boto:Params: {} 2022-09-13T07:29:16.702 DEBUG:boto:Token: None 2022-09-13T07:29:16.703 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/?versions 2022-09-13T07:29:16.704 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NilunrOx831gNzOWjpiKPoimzsk= 2022-09-13T07:29:16.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NilunrOx831gNzOWjpiKPoimzsk=', 'Content-Length': '0'} 2022-09-13T07:29:16.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3717c40d166a793-006320314c-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.710 DEBUG:boto:b'yxvvuh-14999falsemyobjnulltrue2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:16.711 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:16.712 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:16.713 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:16.714 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:16.715 DEBUG:boto:Method: GET 2022-09-13T07:29:16.716 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:16.717 DEBUG:boto:Data: 2022-09-13T07:29:16.718 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.719 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:16.720 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.720 DEBUG:boto:Params: {} 2022-09-13T07:29:16.721 DEBUG:boto:Token: None 2022-09-13T07:29:16.723 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/?versions 2022-09-13T07:29:16.724 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NilunrOx831gNzOWjpiKPoimzsk= 2022-09-13T07:29:16.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NilunrOx831gNzOWjpiKPoimzsk=', 'Content-Length': '0'} 2022-09-13T07:29:16.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e261e5138888ec7b-006320314c-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.729 DEBUG:boto:b'yxvvuh-14999falsemyobjnulltrue2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:16.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:16.731 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:16.732 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:16.733 DEBUG:boto:path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.734 DEBUG:boto:auth_path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.735 DEBUG:boto:Method: GET 2022-09-13T07:29:16.736 DEBUG:boto:Path: /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.737 DEBUG:boto:Data: 2022-09-13T07:29:16.738 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.739 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.740 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.741 DEBUG:boto:Params: {} 2022-09-13T07:29:16.742 DEBUG:boto:Token: None 2022-09-13T07:29:16.743 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.744 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9iLpOUEKukSJLdjfiV6F1D/wNJE= 2022-09-13T07:29:16.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9iLpOUEKukSJLdjfiV6F1D/wNJE=', 'Content-Length': '0'} 2022-09-13T07:29:16.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000755c17fd0ce726c1-006320314c-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.750 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:16.751 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:16.752 DEBUG:boto:path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.753 DEBUG:boto:auth_path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.754 DEBUG:boto:Method: GET 2022-09-13T07:29:16.755 DEBUG:boto:Path: /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.756 DEBUG:boto:Data: 2022-09-13T07:29:16.757 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.758 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:16.759 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.760 DEBUG:boto:Params: {} 2022-09-13T07:29:16.761 DEBUG:boto:Token: None 2022-09-13T07:29:16.762 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.763 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9iLpOUEKukSJLdjfiV6F1D/wNJE= 2022-09-13T07:29:16.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9iLpOUEKukSJLdjfiV6F1D/wNJE=', 'Content-Length': '0'} 2022-09-13T07:29:16.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a2d2346aa19e406-006320314c-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.769 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:16.770 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:16.771 DEBUG:boto:path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.771 DEBUG:boto:auth_path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.772 DEBUG:boto:Method: HEAD 2022-09-13T07:29:16.773 DEBUG:boto:Path: /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.774 DEBUG:boto:Data: 2022-09-13T07:29:16.788 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.789 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.790 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.791 DEBUG:boto:Params: {} 2022-09-13T07:29:16.792 DEBUG:boto:Token: None 2022-09-13T07:29:16.793 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.794 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8ZZuB5rcihYg0pdC4qKmkxS57nw= 2022-09-13T07:29:16.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8ZZuB5rcihYg0pdC4qKmkxS57nw=', 'Content-Length': '0'} 2022-09-13T07:29:16.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2bcc1f1a1d1586c-006320314c-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.800 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:16.801 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:16.802 DEBUG:boto:path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.802 DEBUG:boto:auth_path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.803 DEBUG:boto:Method: HEAD 2022-09-13T07:29:16.804 DEBUG:boto:Path: /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.805 DEBUG:boto:Data: 2022-09-13T07:29:16.806 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.807 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:16.808 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.809 DEBUG:boto:Params: {} 2022-09-13T07:29:16.810 DEBUG:boto:Token: None 2022-09-13T07:29:16.811 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.812 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8ZZuB5rcihYg0pdC4qKmkxS57nw= 2022-09-13T07:29:16.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8ZZuB5rcihYg0pdC4qKmkxS57nw=', 'Content-Length': '0'} 2022-09-13T07:29:16.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006056499f009f5005-006320314c-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:16.818 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:16.819 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:16.820 DEBUG:boto:path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.820 DEBUG:boto:auth_path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.822 DEBUG:boto:Method: GET 2022-09-13T07:29:16.823 DEBUG:boto:Path: /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.823 DEBUG:boto:Data: 2022-09-13T07:29:16.824 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.825 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.826 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.827 DEBUG:boto:Params: {} 2022-09-13T07:29:16.828 DEBUG:boto:Token: None 2022-09-13T07:29:16.830 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.831 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9iLpOUEKukSJLdjfiV6F1D/wNJE= 2022-09-13T07:29:16.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9iLpOUEKukSJLdjfiV6F1D/wNJE=', 'Content-Length': '0'} 2022-09-13T07:29:16.835 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aab4a62849413bd9-006320314c-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.836 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:16.837 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:16.838 DEBUG:boto:path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.839 DEBUG:boto:auth_path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.840 DEBUG:boto:Method: GET 2022-09-13T07:29:16.841 DEBUG:boto:Path: /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.842 DEBUG:boto:Data: 2022-09-13T07:29:16.843 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.843 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:16.844 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.845 DEBUG:boto:Params: {} 2022-09-13T07:29:16.846 DEBUG:boto:Token: None 2022-09-13T07:29:16.847 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:16.849 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9iLpOUEKukSJLdjfiV6F1D/wNJE= 2022-09-13T07:29:16.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9iLpOUEKukSJLdjfiV6F1D/wNJE=', 'Content-Length': '0'} 2022-09-13T07:29:16.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb5dd010aa9aabe3-006320314c-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.853 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:16.854 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:16.856 DEBUG:boto:path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:16.857 DEBUG:boto:auth_path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:16.858 DEBUG:boto:Method: GET 2022-09-13T07:29:16.859 DEBUG:boto:Path: /yxvvuh-14/?prefix=myobj 2022-09-13T07:29:16.860 DEBUG:boto:Data: 2022-09-13T07:29:16.861 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.862 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.863 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.864 DEBUG:boto:Params: {} 2022-09-13T07:29:16.865 DEBUG:boto:Token: None 2022-09-13T07:29:16.866 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/ 2022-09-13T07:29:16.867 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yY7KipzopYl9rjZZlMmQeyUUUz8= 2022-09-13T07:29:16.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yY7KipzopYl9rjZZlMmQeyUUUz8=', 'Content-Length': '0'} 2022-09-13T07:29:16.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000535325baa258b379-006320314c-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.873 DEBUG:boto:b'yxvvuh-14myobj1000falsemyobj2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:16.874 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:16.875 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:16.875 DEBUG:boto:path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:16.876 DEBUG:boto:auth_path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:16.876 DEBUG:boto:Method: GET 2022-09-13T07:29:16.877 DEBUG:boto:Path: /yxvvuh-14/?prefix=myobj 2022-09-13T07:29:16.877 DEBUG:boto:Data: 2022-09-13T07:29:16.878 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.878 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:16.879 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.879 DEBUG:boto:Params: {} 2022-09-13T07:29:16.880 DEBUG:boto:Token: None 2022-09-13T07:29:16.880 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/ 2022-09-13T07:29:16.881 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yY7KipzopYl9rjZZlMmQeyUUUz8= 2022-09-13T07:29:16.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yY7KipzopYl9rjZZlMmQeyUUUz8=', 'Content-Length': '0'} 2022-09-13T07:29:16.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053484ea01323f645-006320314c-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.885 DEBUG:boto:b'yxvvuh-14myobj1000falsemyobj2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:16.886 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:16.886 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:16.887 DEBUG:boto:Method: HEAD 2022-09-13T07:29:16.887 DEBUG:boto:Path: /yxvvuh-14/myobj 2022-09-13T07:29:16.888 DEBUG:boto:Data: 2022-09-13T07:29:16.888 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.889 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.889 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.889 DEBUG:boto:Params: {} 2022-09-13T07:29:16.890 DEBUG:boto:Token: None 2022-09-13T07:29:16.891 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/myobj 2022-09-13T07:29:16.891 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cy5ZCq28i7J/1OO0tuwUCQhf3mM= 2022-09-13T07:29:16.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cy5ZCq28i7J/1OO0tuwUCQhf3mM=', 'Content-Length': '0'} 2022-09-13T07:29:16.894 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b9552e24293e8db-006320314c-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.894 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:16.895 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:16.895 DEBUG:boto:Method: HEAD 2022-09-13T07:29:16.896 DEBUG:boto:Path: /yxvvuh-14/myobj 2022-09-13T07:29:16.896 DEBUG:boto:Data: 2022-09-13T07:29:16.897 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.897 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:16.898 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.898 DEBUG:boto:Params: {} 2022-09-13T07:29:16.899 DEBUG:boto:Token: None 2022-09-13T07:29:16.899 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/myobj 2022-09-13T07:29:16.900 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cy5ZCq28i7J/1OO0tuwUCQhf3mM= 2022-09-13T07:29:16.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cy5ZCq28i7J/1OO0tuwUCQhf3mM=', 'Content-Length': '0'} 2022-09-13T07:29:16.906 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6876428eb67ad73-006320314c-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:16.910 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:16.910 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:16.911 DEBUG:boto:Method: GET 2022-09-13T07:29:16.911 DEBUG:boto:Path: /yxvvuh-14/myobj 2022-09-13T07:29:16.912 DEBUG:boto:Data: 2022-09-13T07:29:16.912 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.913 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.913 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.914 DEBUG:boto:Params: {} 2022-09-13T07:29:16.915 DEBUG:boto:Token: None 2022-09-13T07:29:16.916 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/myobj 2022-09-13T07:29:16.917 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ePXc4kiYd3XlsteirR+vzWuxtjQ= 2022-09-13T07:29:16.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ePXc4kiYd3XlsteirR+vzWuxtjQ=', 'Content-Length': '0'} 2022-09-13T07:29:16.923 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000457d3a752b1e12b1-006320314c-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.924 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:16.924 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:16.925 DEBUG:boto:Method: GET 2022-09-13T07:29:16.925 DEBUG:boto:Path: /yxvvuh-14/myobj 2022-09-13T07:29:16.926 DEBUG:boto:Data: 2022-09-13T07:29:16.927 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.927 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:16.928 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.928 DEBUG:boto:Params: {} 2022-09-13T07:29:16.929 DEBUG:boto:Token: None 2022-09-13T07:29:16.929 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/myobj 2022-09-13T07:29:16.930 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ePXc4kiYd3XlsteirR+vzWuxtjQ= 2022-09-13T07:29:16.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ePXc4kiYd3XlsteirR+vzWuxtjQ=', 'Content-Length': '0'} 2022-09-13T07:29:16.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ccad9dd801fa14c-006320314c-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.978 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:16.979 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:16.980 DEBUG:boto:path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:16.981 DEBUG:boto:auth_path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:16.981 DEBUG:boto:Method: GET 2022-09-13T07:29:16.982 DEBUG:boto:Path: /yxvvuh-14/?prefix=myobj 2022-09-13T07:29:16.983 DEBUG:boto:Data: 2022-09-13T07:29:16.984 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.984 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:16.985 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.985 DEBUG:boto:Params: {} 2022-09-13T07:29:16.986 DEBUG:boto:Token: None 2022-09-13T07:29:16.987 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/ 2022-09-13T07:29:16.987 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yY7KipzopYl9rjZZlMmQeyUUUz8= 2022-09-13T07:29:16.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yY7KipzopYl9rjZZlMmQeyUUUz8=', 'Content-Length': '0'} 2022-09-13T07:29:16.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000670b1c13f8960bb8-006320314c-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:16.991 DEBUG:boto:b'yxvvuh-14myobj1000falsemyobj2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:16.992 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:16.993 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:16.993 DEBUG:boto:path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:16.994 DEBUG:boto:auth_path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:16.994 DEBUG:boto:Method: GET 2022-09-13T07:29:16.995 DEBUG:boto:Path: /yxvvuh-14/?prefix=myobj 2022-09-13T07:29:16.995 DEBUG:boto:Data: 2022-09-13T07:29:16.996 DEBUG:boto:Headers: {} 2022-09-13T07:29:16.996 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:16.997 DEBUG:boto:Port: 8000 2022-09-13T07:29:16.997 DEBUG:boto:Params: {} 2022-09-13T07:29:16.998 DEBUG:boto:Token: None 2022-09-13T07:29:16.999 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:16 GMT /yxvvuh-14/ 2022-09-13T07:29:16.999 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yY7KipzopYl9rjZZlMmQeyUUUz8= 2022-09-13T07:29:17.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:16 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yY7KipzopYl9rjZZlMmQeyUUUz8=', 'Content-Length': '0'} 2022-09-13T07:29:17.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd2c3e6cd2bea1bd-006320314c-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.003 DEBUG:boto:b'yxvvuh-14myobj1000falsemyobj2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.004 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-14 zones={test-zone1, test-zone2} 2022-09-13T07:29:17.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:17.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:17.006 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:29:17.069 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:29:17.069 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:29:17.104 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:17.100+0000 7f6856aa4600 1 bucket sync caught up with source: 2022-09-13T07:29:17.104 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2022-09-13T07:29:17.105 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2022-09-13T07:29:17.105 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:17.100+0000 7f6856aa4600 0 bucket checkpoint complete 2022-09-13T07:29:17.116 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-14 zones={test-zone3, test-zone2} 2022-09-13T07:29:17.116 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:17.117 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:17.117 DEBUG:boto:Method: HEAD 2022-09-13T07:29:17.118 DEBUG:boto:Path: /yxvvuh-14/ 2022-09-13T07:29:17.119 DEBUG:boto:Data: 2022-09-13T07:29:17.119 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.120 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.121 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.122 DEBUG:boto:Params: {} 2022-09-13T07:29:17.123 DEBUG:boto:Token: None 2022-09-13T07:29:17.124 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/ 2022-09-13T07:29:17.125 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6Wns4ckAZVMnNZI4TKE9oKMFRBk= 2022-09-13T07:29:17.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6Wns4ckAZVMnNZI4TKE9oKMFRBk=', 'Content-Length': '0'} 2022-09-13T07:29:17.129 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b32ff48455ff40ca-006320314d-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.131 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:17.132 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:17.133 DEBUG:boto:Method: HEAD 2022-09-13T07:29:17.134 DEBUG:boto:Path: /yxvvuh-14/ 2022-09-13T07:29:17.135 DEBUG:boto:Data: 2022-09-13T07:29:17.136 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.137 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:17.137 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.139 DEBUG:boto:Params: {} 2022-09-13T07:29:17.140 DEBUG:boto:Token: None 2022-09-13T07:29:17.141 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/ 2022-09-13T07:29:17.142 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6Wns4ckAZVMnNZI4TKE9oKMFRBk= 2022-09-13T07:29:17.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6Wns4ckAZVMnNZI4TKE9oKMFRBk=', 'Content-Length': '0'} 2022-09-13T07:29:17.146 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e48abdfd0f7ac3d-006320314d-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.147 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:29:17.148 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:17.149 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:17.150 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:17.151 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:17.151 DEBUG:boto:Method: GET 2022-09-13T07:29:17.153 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:17.153 DEBUG:boto:Data: 2022-09-13T07:29:17.154 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.155 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.156 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.157 DEBUG:boto:Params: {} 2022-09-13T07:29:17.158 DEBUG:boto:Token: None 2022-09-13T07:29:17.159 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/?versions 2022-09-13T07:29:17.160 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siWCIN9JoXQkjd9C9ekThauXjIo= 2022-09-13T07:29:17.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siWCIN9JoXQkjd9C9ekThauXjIo=', 'Content-Length': '0'} 2022-09-13T07:29:17.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ef0548f2cbea3d6-006320314d-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.163 DEBUG:boto:b'yxvvuh-14999falsemyobjnulltrue2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.164 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:29:17.164 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:29:17.165 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:17.165 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:17.166 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:17.167 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:17.168 DEBUG:boto:Method: GET 2022-09-13T07:29:17.168 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:17.169 DEBUG:boto:Data: 2022-09-13T07:29:17.169 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.170 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:17.171 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.171 DEBUG:boto:Params: {} 2022-09-13T07:29:17.172 DEBUG:boto:Token: None 2022-09-13T07:29:17.172 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/?versions 2022-09-13T07:29:17.173 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siWCIN9JoXQkjd9C9ekThauXjIo= 2022-09-13T07:29:17.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siWCIN9JoXQkjd9C9ekThauXjIo=', 'Content-Length': '0'} 2022-09-13T07:29:17.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060e63148cdc40417-006320314d-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.177 DEBUG:boto:b'yxvvuh-14999falsemyobjnulltrue2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.178 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:29:17.179 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:17.179 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:17.180 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:17.180 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:17.181 DEBUG:boto:Method: GET 2022-09-13T07:29:17.181 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:17.182 DEBUG:boto:Data: 2022-09-13T07:29:17.182 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.183 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.183 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.183 DEBUG:boto:Params: {} 2022-09-13T07:29:17.184 DEBUG:boto:Token: None 2022-09-13T07:29:17.185 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/?versions 2022-09-13T07:29:17.186 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siWCIN9JoXQkjd9C9ekThauXjIo= 2022-09-13T07:29:17.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siWCIN9JoXQkjd9C9ekThauXjIo=', 'Content-Length': '0'} 2022-09-13T07:29:17.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b35cff426f8fa35-006320314d-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.190 DEBUG:boto:b'yxvvuh-14999falsemyobjnulltrue2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.191 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:17.192 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:17.192 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:17.193 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:17.193 DEBUG:boto:Method: GET 2022-09-13T07:29:17.194 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:17.194 DEBUG:boto:Data: 2022-09-13T07:29:17.195 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.196 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:17.196 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.197 DEBUG:boto:Params: {} 2022-09-13T07:29:17.197 DEBUG:boto:Token: None 2022-09-13T07:29:17.198 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/?versions 2022-09-13T07:29:17.198 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:siWCIN9JoXQkjd9C9ekThauXjIo= 2022-09-13T07:29:17.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:siWCIN9JoXQkjd9C9ekThauXjIo=', 'Content-Length': '0'} 2022-09-13T07:29:17.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d0fbd6b947c7baf-006320314d-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.202 DEBUG:boto:b'yxvvuh-14999falsemyobjnulltrue2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:17.203 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:17.204 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:17.204 DEBUG:boto:path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.205 DEBUG:boto:auth_path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.205 DEBUG:boto:Method: GET 2022-09-13T07:29:17.206 DEBUG:boto:Path: /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.207 DEBUG:boto:Data: 2022-09-13T07:29:17.208 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.208 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.209 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.209 DEBUG:boto:Params: {} 2022-09-13T07:29:17.210 DEBUG:boto:Token: None 2022-09-13T07:29:17.210 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.211 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i6T6U+rBP92piHAVO4dFXpxB6Lw= 2022-09-13T07:29:17.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i6T6U+rBP92piHAVO4dFXpxB6Lw=', 'Content-Length': '0'} 2022-09-13T07:29:17.213 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007188a04abb867361-006320314d-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.214 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:17.215 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:17.215 DEBUG:boto:path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.216 DEBUG:boto:auth_path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.216 DEBUG:boto:Method: GET 2022-09-13T07:29:17.217 DEBUG:boto:Path: /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.217 DEBUG:boto:Data: 2022-09-13T07:29:17.218 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.218 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:17.219 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.219 DEBUG:boto:Params: {} 2022-09-13T07:29:17.220 DEBUG:boto:Token: None 2022-09-13T07:29:17.220 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.221 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i6T6U+rBP92piHAVO4dFXpxB6Lw= 2022-09-13T07:29:17.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i6T6U+rBP92piHAVO4dFXpxB6Lw=', 'Content-Length': '0'} 2022-09-13T07:29:17.224 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b496d4a56291173-006320314d-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.225 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:17.225 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:17.226 DEBUG:boto:path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.226 DEBUG:boto:auth_path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.227 DEBUG:boto:Method: HEAD 2022-09-13T07:29:17.227 DEBUG:boto:Path: /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.228 DEBUG:boto:Data: 2022-09-13T07:29:17.228 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.229 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.229 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.230 DEBUG:boto:Params: {} 2022-09-13T07:29:17.230 DEBUG:boto:Token: None 2022-09-13T07:29:17.231 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.232 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fiXw/yA1E7dtcrg6El7yFHTIXxM= 2022-09-13T07:29:17.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fiXw/yA1E7dtcrg6El7yFHTIXxM=', 'Content-Length': '0'} 2022-09-13T07:29:17.234 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c9742cf6aeb4685-006320314d-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.235 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:17.235 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:17.236 DEBUG:boto:path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.236 DEBUG:boto:auth_path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.237 DEBUG:boto:Method: HEAD 2022-09-13T07:29:17.237 DEBUG:boto:Path: /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.238 DEBUG:boto:Data: 2022-09-13T07:29:17.238 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.239 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:17.239 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.240 DEBUG:boto:Params: {} 2022-09-13T07:29:17.241 DEBUG:boto:Token: None 2022-09-13T07:29:17.242 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.242 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fiXw/yA1E7dtcrg6El7yFHTIXxM= 2022-09-13T07:29:17.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fiXw/yA1E7dtcrg6El7yFHTIXxM=', 'Content-Length': '0'} 2022-09-13T07:29:17.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036cf460be8c31efc-006320314d-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:17.246 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:17.247 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:17.248 DEBUG:boto:path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.249 DEBUG:boto:auth_path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.250 DEBUG:boto:Method: GET 2022-09-13T07:29:17.250 DEBUG:boto:Path: /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.251 DEBUG:boto:Data: 2022-09-13T07:29:17.254 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.255 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.256 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.256 DEBUG:boto:Params: {} 2022-09-13T07:29:17.257 DEBUG:boto:Token: None 2022-09-13T07:29:17.259 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.260 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i6T6U+rBP92piHAVO4dFXpxB6Lw= 2022-09-13T07:29:17.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i6T6U+rBP92piHAVO4dFXpxB6Lw=', 'Content-Length': '0'} 2022-09-13T07:29:17.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000553f72909bf86d4e-006320314d-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.306 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:17.307 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:17.307 DEBUG:boto:path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.308 DEBUG:boto:auth_path=/yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.308 DEBUG:boto:Method: GET 2022-09-13T07:29:17.309 DEBUG:boto:Path: /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.309 DEBUG:boto:Data: 2022-09-13T07:29:17.310 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.311 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:17.311 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.312 DEBUG:boto:Params: {} 2022-09-13T07:29:17.312 DEBUG:boto:Token: None 2022-09-13T07:29:17.313 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/myobj?versionId=null 2022-09-13T07:29:17.314 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i6T6U+rBP92piHAVO4dFXpxB6Lw= 2022-09-13T07:29:17.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i6T6U+rBP92piHAVO4dFXpxB6Lw=', 'Content-Length': '0'} 2022-09-13T07:29:17.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbbb62b727b13d4c-006320314d-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.321 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:17.322 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:17.322 DEBUG:boto:path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:17.323 DEBUG:boto:auth_path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:17.323 DEBUG:boto:Method: GET 2022-09-13T07:29:17.324 DEBUG:boto:Path: /yxvvuh-14/?prefix=myobj 2022-09-13T07:29:17.324 DEBUG:boto:Data: 2022-09-13T07:29:17.325 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.326 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.327 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.328 DEBUG:boto:Params: {} 2022-09-13T07:29:17.329 DEBUG:boto:Token: None 2022-09-13T07:29:17.330 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/ 2022-09-13T07:29:17.331 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8ewdkMmTSeAG/JmmTsifOgf057w= 2022-09-13T07:29:17.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8ewdkMmTSeAG/JmmTsifOgf057w=', 'Content-Length': '0'} 2022-09-13T07:29:17.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dfb5e2dabdcded4-006320314d-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.336 DEBUG:boto:b'yxvvuh-14myobj1000falsemyobj2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.338 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:17.339 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:17.340 DEBUG:boto:path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:17.341 DEBUG:boto:auth_path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:17.342 DEBUG:boto:Method: GET 2022-09-13T07:29:17.343 DEBUG:boto:Path: /yxvvuh-14/?prefix=myobj 2022-09-13T07:29:17.344 DEBUG:boto:Data: 2022-09-13T07:29:17.345 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.346 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:17.347 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.348 DEBUG:boto:Params: {} 2022-09-13T07:29:17.348 DEBUG:boto:Token: None 2022-09-13T07:29:17.350 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/ 2022-09-13T07:29:17.350 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8ewdkMmTSeAG/JmmTsifOgf057w= 2022-09-13T07:29:17.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8ewdkMmTSeAG/JmmTsifOgf057w=', 'Content-Length': '0'} 2022-09-13T07:29:17.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000415d71acb62f6b01-006320314d-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.355 DEBUG:boto:b'yxvvuh-14myobj1000falsemyobj2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.356 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:17.357 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:17.357 DEBUG:boto:Method: HEAD 2022-09-13T07:29:17.358 DEBUG:boto:Path: /yxvvuh-14/myobj 2022-09-13T07:29:17.359 DEBUG:boto:Data: 2022-09-13T07:29:17.359 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.360 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.360 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.361 DEBUG:boto:Params: {} 2022-09-13T07:29:17.361 DEBUG:boto:Token: None 2022-09-13T07:29:17.362 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/myobj 2022-09-13T07:29:17.363 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kZJQFR7mMGbcWRrQ2JG3VsmLPjs= 2022-09-13T07:29:17.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kZJQFR7mMGbcWRrQ2JG3VsmLPjs=', 'Content-Length': '0'} 2022-09-13T07:29:17.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c5a8c00e7d25034-006320314d-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.366 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:17.366 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:17.367 DEBUG:boto:Method: HEAD 2022-09-13T07:29:17.367 DEBUG:boto:Path: /yxvvuh-14/myobj 2022-09-13T07:29:17.368 DEBUG:boto:Data: 2022-09-13T07:29:17.369 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.369 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:17.370 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.370 DEBUG:boto:Params: {} 2022-09-13T07:29:17.371 DEBUG:boto:Token: None 2022-09-13T07:29:17.372 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/myobj 2022-09-13T07:29:17.373 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kZJQFR7mMGbcWRrQ2JG3VsmLPjs= 2022-09-13T07:29:17.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kZJQFR7mMGbcWRrQ2JG3VsmLPjs=', 'Content-Length': '0'} 2022-09-13T07:29:17.377 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c09c1ad60ede99ea-006320314d-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:17.380 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:17.381 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:17.382 DEBUG:boto:Method: GET 2022-09-13T07:29:17.383 DEBUG:boto:Path: /yxvvuh-14/myobj 2022-09-13T07:29:17.384 DEBUG:boto:Data: 2022-09-13T07:29:17.385 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.386 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.387 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.388 DEBUG:boto:Params: {} 2022-09-13T07:29:17.389 DEBUG:boto:Token: None 2022-09-13T07:29:17.390 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/myobj 2022-09-13T07:29:17.391 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cpnlXmsgA9gJZUWtT0rswK1PGPw= 2022-09-13T07:29:17.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cpnlXmsgA9gJZUWtT0rswK1PGPw=', 'Content-Length': '0'} 2022-09-13T07:29:17.395 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c511fb112ab0a72-006320314d-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.397 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:17.398 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:17.399 DEBUG:boto:Method: GET 2022-09-13T07:29:17.400 DEBUG:boto:Path: /yxvvuh-14/myobj 2022-09-13T07:29:17.401 DEBUG:boto:Data: 2022-09-13T07:29:17.402 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.403 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:17.404 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.405 DEBUG:boto:Params: {} 2022-09-13T07:29:17.406 DEBUG:boto:Token: None 2022-09-13T07:29:17.408 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/myobj 2022-09-13T07:29:17.409 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cpnlXmsgA9gJZUWtT0rswK1PGPw= 2022-09-13T07:29:17.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cpnlXmsgA9gJZUWtT0rswK1PGPw=', 'Content-Length': '0'} 2022-09-13T07:29:17.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6632d58f8e37597-006320314d-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.454 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:17.455 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:17.456 DEBUG:boto:path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:17.457 DEBUG:boto:auth_path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:17.458 DEBUG:boto:Method: GET 2022-09-13T07:29:17.459 DEBUG:boto:Path: /yxvvuh-14/?prefix=myobj 2022-09-13T07:29:17.460 DEBUG:boto:Data: 2022-09-13T07:29:17.461 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.462 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.463 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.464 DEBUG:boto:Params: {} 2022-09-13T07:29:17.465 DEBUG:boto:Token: None 2022-09-13T07:29:17.466 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/ 2022-09-13T07:29:17.467 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8ewdkMmTSeAG/JmmTsifOgf057w= 2022-09-13T07:29:17.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8ewdkMmTSeAG/JmmTsifOgf057w=', 'Content-Length': '0'} 2022-09-13T07:29:17.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e4a580ccc8c7b7a-006320314d-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.473 DEBUG:boto:b'yxvvuh-14myobj1000falsemyobj2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.475 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:17.476 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:17.477 DEBUG:boto:path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:17.478 DEBUG:boto:auth_path=/yxvvuh-14/?prefix=myobj 2022-09-13T07:29:17.479 DEBUG:boto:Method: GET 2022-09-13T07:29:17.480 DEBUG:boto:Path: /yxvvuh-14/?prefix=myobj 2022-09-13T07:29:17.480 DEBUG:boto:Data: 2022-09-13T07:29:17.481 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.483 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:17.483 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.484 DEBUG:boto:Params: {} 2022-09-13T07:29:17.484 DEBUG:boto:Token: None 2022-09-13T07:29:17.485 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-14/ 2022-09-13T07:29:17.486 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8ewdkMmTSeAG/JmmTsifOgf057w= 2022-09-13T07:29:17.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8ewdkMmTSeAG/JmmTsifOgf057w=', 'Content-Length': '0'} 2022-09-13T07:29:17.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000981533394f59a2a9-006320314d-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.490 DEBUG:boto:b'yxvvuh-14myobj1000falsemyobj2022-09-13T07:28:54.893Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.491 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-14 zones={test-zone3, test-zone2} 2022-09-13T07:29:17.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:17.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:17.492 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:29:17.554 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:29:17.554 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:29:17.589 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:17.585+0000 7f2eed2a6600 1 bucket sync caught up with source: 2022-09-13T07:29:17.589 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:29:17.589 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:29:17.590 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:17.585+0000 7f2eed2a6600 0 bucket checkpoint complete 2022-09-13T07:29:17.602 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-15 zones={test-zone1, test-zone3} 2022-09-13T07:29:17.603 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:17.604 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:17.605 DEBUG:boto:Method: HEAD 2022-09-13T07:29:17.606 DEBUG:boto:Path: /yxvvuh-15/ 2022-09-13T07:29:17.607 DEBUG:boto:Data: 2022-09-13T07:29:17.608 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.608 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:17.609 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.609 DEBUG:boto:Params: {} 2022-09-13T07:29:17.610 DEBUG:boto:Token: None 2022-09-13T07:29:17.610 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/ 2022-09-13T07:29:17.611 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uJUKizs6Tqup+QIVfF0uwL/IlnQ= 2022-09-13T07:29:17.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uJUKizs6Tqup+QIVfF0uwL/IlnQ=', 'Content-Length': '0'} 2022-09-13T07:29:17.614 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036476b18bc737b33-006320314d-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.614 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:17.615 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:17.615 DEBUG:boto:Method: HEAD 2022-09-13T07:29:17.616 DEBUG:boto:Path: /yxvvuh-15/ 2022-09-13T07:29:17.616 DEBUG:boto:Data: 2022-09-13T07:29:17.617 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.617 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.618 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.618 DEBUG:boto:Params: {} 2022-09-13T07:29:17.619 DEBUG:boto:Token: None 2022-09-13T07:29:17.619 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/ 2022-09-13T07:29:17.620 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uJUKizs6Tqup+QIVfF0uwL/IlnQ= 2022-09-13T07:29:17.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uJUKizs6Tqup+QIVfF0uwL/IlnQ=', 'Content-Length': '0'} 2022-09-13T07:29:17.623 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006973aa592b9c7d22-006320314d-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.623 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:29:17.624 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:17.625 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:17.625 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:17.625 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:17.626 DEBUG:boto:Method: GET 2022-09-13T07:29:17.627 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:17.627 DEBUG:boto:Data: 2022-09-13T07:29:17.628 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.628 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:17.629 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.629 DEBUG:boto:Params: {} 2022-09-13T07:29:17.630 DEBUG:boto:Token: None 2022-09-13T07:29:17.630 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/?versions 2022-09-13T07:29:17.639 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TdqC07qtywVR+GK+DXSgV3/1j3A= 2022-09-13T07:29:17.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TdqC07qtywVR+GK+DXSgV3/1j3A=', 'Content-Length': '0'} 2022-09-13T07:29:17.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cfaab328612b03f-006320314d-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.644 DEBUG:boto:b'yxvvuh-15999falsemyobjnulltrue2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.645 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:29:17.646 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:29:17.647 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:17.648 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:17.649 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:17.650 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:17.651 DEBUG:boto:Method: GET 2022-09-13T07:29:17.652 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:17.652 DEBUG:boto:Data: 2022-09-13T07:29:17.653 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.654 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.655 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.656 DEBUG:boto:Params: {} 2022-09-13T07:29:17.657 DEBUG:boto:Token: None 2022-09-13T07:29:17.658 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/?versions 2022-09-13T07:29:17.659 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TdqC07qtywVR+GK+DXSgV3/1j3A= 2022-09-13T07:29:17.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TdqC07qtywVR+GK+DXSgV3/1j3A=', 'Content-Length': '0'} 2022-09-13T07:29:17.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb30b28e07761ad6-006320314d-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.706 DEBUG:boto:b'yxvvuh-15999falsemyobjnulltrue2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.707 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:29:17.708 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:17.708 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:17.709 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:17.709 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:17.710 DEBUG:boto:Method: GET 2022-09-13T07:29:17.711 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:17.711 DEBUG:boto:Data: 2022-09-13T07:29:17.712 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.712 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:17.715 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.715 DEBUG:boto:Params: {} 2022-09-13T07:29:17.716 DEBUG:boto:Token: None 2022-09-13T07:29:17.717 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/?versions 2022-09-13T07:29:17.717 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TdqC07qtywVR+GK+DXSgV3/1j3A= 2022-09-13T07:29:17.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TdqC07qtywVR+GK+DXSgV3/1j3A=', 'Content-Length': '0'} 2022-09-13T07:29:17.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d4833d009631bde-006320314d-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.721 DEBUG:boto:b'yxvvuh-15999falsemyobjnulltrue2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.722 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:17.723 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:17.724 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:17.724 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:17.725 DEBUG:boto:Method: GET 2022-09-13T07:29:17.725 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:17.726 DEBUG:boto:Data: 2022-09-13T07:29:17.727 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.727 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.728 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.728 DEBUG:boto:Params: {} 2022-09-13T07:29:17.729 DEBUG:boto:Token: None 2022-09-13T07:29:17.729 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/?versions 2022-09-13T07:29:17.730 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TdqC07qtywVR+GK+DXSgV3/1j3A= 2022-09-13T07:29:17.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TdqC07qtywVR+GK+DXSgV3/1j3A=', 'Content-Length': '0'} 2022-09-13T07:29:17.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013a70baa2941b2f2-006320314d-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.735 DEBUG:boto:b'yxvvuh-15999falsemyobjnulltrue2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:17.737 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:17.738 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:17.739 DEBUG:boto:path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.740 DEBUG:boto:auth_path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.741 DEBUG:boto:Method: GET 2022-09-13T07:29:17.742 DEBUG:boto:Path: /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.743 DEBUG:boto:Data: 2022-09-13T07:29:17.743 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.744 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:17.744 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.745 DEBUG:boto:Params: {} 2022-09-13T07:29:17.746 DEBUG:boto:Token: None 2022-09-13T07:29:17.746 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.747 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:o0lUM+9K2y53NGkW2hATwHefaY0= 2022-09-13T07:29:17.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:o0lUM+9K2y53NGkW2hATwHefaY0=', 'Content-Length': '0'} 2022-09-13T07:29:17.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb1682d419775c05-006320314d-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.751 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:17.752 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:17.752 DEBUG:boto:path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.753 DEBUG:boto:auth_path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.754 DEBUG:boto:Method: GET 2022-09-13T07:29:17.754 DEBUG:boto:Path: /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.755 DEBUG:boto:Data: 2022-09-13T07:29:17.755 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.756 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.756 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.757 DEBUG:boto:Params: {} 2022-09-13T07:29:17.757 DEBUG:boto:Token: None 2022-09-13T07:29:17.758 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.758 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:o0lUM+9K2y53NGkW2hATwHefaY0= 2022-09-13T07:29:17.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:o0lUM+9K2y53NGkW2hATwHefaY0=', 'Content-Length': '0'} 2022-09-13T07:29:17.761 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef2bab4cfb4bb2b8-006320314d-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.762 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:17.762 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:17.763 DEBUG:boto:path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.764 DEBUG:boto:auth_path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.764 DEBUG:boto:Method: HEAD 2022-09-13T07:29:17.765 DEBUG:boto:Path: /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.765 DEBUG:boto:Data: 2022-09-13T07:29:17.766 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.766 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:17.767 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.767 DEBUG:boto:Params: {} 2022-09-13T07:29:17.768 DEBUG:boto:Token: None 2022-09-13T07:29:17.768 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.769 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T0f4bEljcb9bDy4w7qMDTLi4xzk= 2022-09-13T07:29:17.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T0f4bEljcb9bDy4w7qMDTLi4xzk=', 'Content-Length': '0'} 2022-09-13T07:29:17.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009101652121297ed1-006320314d-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.772 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:17.773 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:17.773 DEBUG:boto:path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.774 DEBUG:boto:auth_path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.775 DEBUG:boto:Method: HEAD 2022-09-13T07:29:17.776 DEBUG:boto:Path: /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.777 DEBUG:boto:Data: 2022-09-13T07:29:17.779 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.780 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.781 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.782 DEBUG:boto:Params: {} 2022-09-13T07:29:17.783 DEBUG:boto:Token: None 2022-09-13T07:29:17.784 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.785 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T0f4bEljcb9bDy4w7qMDTLi4xzk= 2022-09-13T07:29:17.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T0f4bEljcb9bDy4w7qMDTLi4xzk=', 'Content-Length': '0'} 2022-09-13T07:29:17.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4a643700c9538fc-006320314d-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:17.790 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:17.790 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:17.791 DEBUG:boto:path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.791 DEBUG:boto:auth_path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.792 DEBUG:boto:Method: GET 2022-09-13T07:29:17.792 DEBUG:boto:Path: /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.793 DEBUG:boto:Data: 2022-09-13T07:29:17.793 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.794 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:17.794 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.795 DEBUG:boto:Params: {} 2022-09-13T07:29:17.795 DEBUG:boto:Token: None 2022-09-13T07:29:17.796 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.797 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:o0lUM+9K2y53NGkW2hATwHefaY0= 2022-09-13T07:29:17.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:o0lUM+9K2y53NGkW2hATwHefaY0=', 'Content-Length': '0'} 2022-09-13T07:29:17.799 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dd3a5240cafdab5-006320314d-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.842 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:17.844 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:17.844 DEBUG:boto:path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.845 DEBUG:boto:auth_path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.846 DEBUG:boto:Method: GET 2022-09-13T07:29:17.847 DEBUG:boto:Path: /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.848 DEBUG:boto:Data: 2022-09-13T07:29:17.849 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.851 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.851 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.852 DEBUG:boto:Params: {} 2022-09-13T07:29:17.853 DEBUG:boto:Token: None 2022-09-13T07:29:17.854 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:17.855 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:o0lUM+9K2y53NGkW2hATwHefaY0= 2022-09-13T07:29:17.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:o0lUM+9K2y53NGkW2hATwHefaY0=', 'Content-Length': '0'} 2022-09-13T07:29:17.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002271e9dd3f9a5782-006320314d-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.859 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:17.859 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:17.860 DEBUG:boto:path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:17.860 DEBUG:boto:auth_path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:17.861 DEBUG:boto:Method: GET 2022-09-13T07:29:17.861 DEBUG:boto:Path: /yxvvuh-15/?prefix=myobj 2022-09-13T07:29:17.862 DEBUG:boto:Data: 2022-09-13T07:29:17.863 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.864 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:17.864 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.865 DEBUG:boto:Params: {} 2022-09-13T07:29:17.865 DEBUG:boto:Token: None 2022-09-13T07:29:17.866 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/ 2022-09-13T07:29:17.867 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+f+UDWkBtiguxAjDj84ZVikBwhw= 2022-09-13T07:29:17.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+f+UDWkBtiguxAjDj84ZVikBwhw=', 'Content-Length': '0'} 2022-09-13T07:29:17.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050a60272433ea5eb-006320314d-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.872 DEBUG:boto:b'yxvvuh-15myobj1000falsemyobj2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.874 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:17.875 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:17.875 DEBUG:boto:path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:17.876 DEBUG:boto:auth_path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:17.876 DEBUG:boto:Method: GET 2022-09-13T07:29:17.877 DEBUG:boto:Path: /yxvvuh-15/?prefix=myobj 2022-09-13T07:29:17.877 DEBUG:boto:Data: 2022-09-13T07:29:17.878 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.878 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.879 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.879 DEBUG:boto:Params: {} 2022-09-13T07:29:17.880 DEBUG:boto:Token: None 2022-09-13T07:29:17.881 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/ 2022-09-13T07:29:17.881 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+f+UDWkBtiguxAjDj84ZVikBwhw= 2022-09-13T07:29:17.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+f+UDWkBtiguxAjDj84ZVikBwhw=', 'Content-Length': '0'} 2022-09-13T07:29:17.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbe592d9ea8562a6-006320314d-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.885 DEBUG:boto:b'yxvvuh-15myobj1000falsemyobj2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.886 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:17.887 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:17.887 DEBUG:boto:Method: HEAD 2022-09-13T07:29:17.888 DEBUG:boto:Path: /yxvvuh-15/myobj 2022-09-13T07:29:17.888 DEBUG:boto:Data: 2022-09-13T07:29:17.889 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.890 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:17.891 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.891 DEBUG:boto:Params: {} 2022-09-13T07:29:17.892 DEBUG:boto:Token: None 2022-09-13T07:29:17.892 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/myobj 2022-09-13T07:29:17.893 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tLrPr0y6MPKJ9oqP+HAFmdgMqbE= 2022-09-13T07:29:17.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tLrPr0y6MPKJ9oqP+HAFmdgMqbE=', 'Content-Length': '0'} 2022-09-13T07:29:17.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000099f8b4a6c65ae8e-006320314d-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.904 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:17.905 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:17.905 DEBUG:boto:Method: HEAD 2022-09-13T07:29:17.906 DEBUG:boto:Path: /yxvvuh-15/myobj 2022-09-13T07:29:17.906 DEBUG:boto:Data: 2022-09-13T07:29:17.907 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.907 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.908 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.908 DEBUG:boto:Params: {} 2022-09-13T07:29:17.909 DEBUG:boto:Token: None 2022-09-13T07:29:17.909 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/myobj 2022-09-13T07:29:17.910 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tLrPr0y6MPKJ9oqP+HAFmdgMqbE= 2022-09-13T07:29:17.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tLrPr0y6MPKJ9oqP+HAFmdgMqbE=', 'Content-Length': '0'} 2022-09-13T07:29:17.912 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e88254c967135f60-006320314d-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:17.913 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:17.914 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:17.915 DEBUG:boto:Method: GET 2022-09-13T07:29:17.915 DEBUG:boto:Path: /yxvvuh-15/myobj 2022-09-13T07:29:17.916 DEBUG:boto:Data: 2022-09-13T07:29:17.916 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.917 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:17.921 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.922 DEBUG:boto:Params: {} 2022-09-13T07:29:17.923 DEBUG:boto:Token: None 2022-09-13T07:29:17.923 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/myobj 2022-09-13T07:29:17.924 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9Dk7zhL8Ns5SReasiKuLtwa1VQs= 2022-09-13T07:29:17.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9Dk7zhL8Ns5SReasiKuLtwa1VQs=', 'Content-Length': '0'} 2022-09-13T07:29:17.927 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4d7c85971e9212e-006320314d-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.970 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:17.971 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:17.971 DEBUG:boto:Method: GET 2022-09-13T07:29:17.972 DEBUG:boto:Path: /yxvvuh-15/myobj 2022-09-13T07:29:17.972 DEBUG:boto:Data: 2022-09-13T07:29:17.973 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.973 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:17.974 DEBUG:boto:Port: 8001 2022-09-13T07:29:17.974 DEBUG:boto:Params: {} 2022-09-13T07:29:17.975 DEBUG:boto:Token: None 2022-09-13T07:29:17.976 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/myobj 2022-09-13T07:29:17.976 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9Dk7zhL8Ns5SReasiKuLtwa1VQs= 2022-09-13T07:29:17.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9Dk7zhL8Ns5SReasiKuLtwa1VQs=', 'Content-Length': '0'} 2022-09-13T07:29:17.979 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3fd2cbf9517b3aa-006320314d-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.979 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:17.980 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:17.981 DEBUG:boto:path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:17.981 DEBUG:boto:auth_path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:17.982 DEBUG:boto:Method: GET 2022-09-13T07:29:17.982 DEBUG:boto:Path: /yxvvuh-15/?prefix=myobj 2022-09-13T07:29:17.983 DEBUG:boto:Data: 2022-09-13T07:29:17.983 DEBUG:boto:Headers: {} 2022-09-13T07:29:17.984 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:17.985 DEBUG:boto:Port: 8000 2022-09-13T07:29:17.986 DEBUG:boto:Params: {} 2022-09-13T07:29:17.987 DEBUG:boto:Token: None 2022-09-13T07:29:17.988 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:17 GMT /yxvvuh-15/ 2022-09-13T07:29:17.989 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+f+UDWkBtiguxAjDj84ZVikBwhw= 2022-09-13T07:29:17.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+f+UDWkBtiguxAjDj84ZVikBwhw=', 'Content-Length': '0'} 2022-09-13T07:29:17.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095fc1e40de5e672d-006320314d-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:17.994 DEBUG:boto:b'yxvvuh-15myobj1000falsemyobj2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:17.995 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:17.996 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:17.996 DEBUG:boto:path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:17.997 DEBUG:boto:auth_path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:17.998 DEBUG:boto:Method: GET 2022-09-13T07:29:17.998 DEBUG:boto:Path: /yxvvuh-15/?prefix=myobj 2022-09-13T07:29:17.999 DEBUG:boto:Data: 2022-09-13T07:29:18.000 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.001 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:18.002 DEBUG:boto:Port: 8001 2022-09-13T07:29:18.003 DEBUG:boto:Params: {} 2022-09-13T07:29:18.004 DEBUG:boto:Token: None 2022-09-13T07:29:18.005 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/ 2022-09-13T07:29:18.006 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ammnEv+NHtor2oP/ckOXsDYSZfQ= 2022-09-13T07:29:18.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ammnEv+NHtor2oP/ckOXsDYSZfQ=', 'Content-Length': '0'} 2022-09-13T07:29:18.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d3cdf7279f07dd3-006320314e-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.054 DEBUG:boto:b'yxvvuh-15myobj1000falsemyobj2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:18.055 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-15 zones={test-zone1, test-zone3} 2022-09-13T07:29:18.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:18.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:18.057 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:29:18.122 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:29:18.123 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:29:18.163 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:29:18.160+0000 7fea52178600 1 bucket sync caught up with source: 2022-09-13T07:29:18.164 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:29:18.169 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:29:18.169 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:29:18.160+0000 7fea52178600 0 bucket checkpoint complete 2022-09-13T07:29:18.179 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-15 zones={test-zone2, test-zone3} 2022-09-13T07:29:18.179 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:18.180 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:18.180 DEBUG:boto:Method: HEAD 2022-09-13T07:29:18.181 DEBUG:boto:Path: /yxvvuh-15/ 2022-09-13T07:29:18.181 DEBUG:boto:Data: 2022-09-13T07:29:18.182 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.182 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:18.183 DEBUG:boto:Port: 8000 2022-09-13T07:29:18.184 DEBUG:boto:Params: {} 2022-09-13T07:29:18.185 DEBUG:boto:Token: None 2022-09-13T07:29:18.185 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/ 2022-09-13T07:29:18.186 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Wa1V1Hle3aJ6g5mm80QfPxsQZCw= 2022-09-13T07:29:18.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Wa1V1Hle3aJ6g5mm80QfPxsQZCw=', 'Content-Length': '0'} 2022-09-13T07:29:18.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009648cd28b209ae8c-006320314e-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.191 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:18.192 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:18.193 DEBUG:boto:Method: HEAD 2022-09-13T07:29:18.193 DEBUG:boto:Path: /yxvvuh-15/ 2022-09-13T07:29:18.194 DEBUG:boto:Data: 2022-09-13T07:29:18.195 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.195 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:18.196 DEBUG:boto:Port: 8001 2022-09-13T07:29:18.196 DEBUG:boto:Params: {} 2022-09-13T07:29:18.197 DEBUG:boto:Token: None 2022-09-13T07:29:18.197 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/ 2022-09-13T07:29:18.198 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Wa1V1Hle3aJ6g5mm80QfPxsQZCw= 2022-09-13T07:29:18.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Wa1V1Hle3aJ6g5mm80QfPxsQZCw=', 'Content-Length': '0'} 2022-09-13T07:29:18.203 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027127a543ff103a7-006320314e-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.204 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:29:18.205 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:18.206 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:18.207 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:18.208 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:18.209 DEBUG:boto:Method: GET 2022-09-13T07:29:18.210 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:18.211 DEBUG:boto:Data: 2022-09-13T07:29:18.212 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.212 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:18.213 DEBUG:boto:Port: 8000 2022-09-13T07:29:18.214 DEBUG:boto:Params: {} 2022-09-13T07:29:18.215 DEBUG:boto:Token: None 2022-09-13T07:29:18.216 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/?versions 2022-09-13T07:29:18.217 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/N263u65qGfUDoH/ljAAVjlE78I= 2022-09-13T07:29:18.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/N263u65qGfUDoH/ljAAVjlE78I=', 'Content-Length': '0'} 2022-09-13T07:29:18.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b037443b37e6abfe-006320314e-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.222 DEBUG:boto:b'yxvvuh-15999falsemyobjnulltrue2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:18.223 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:29:18.224 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:29:18.224 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:18.225 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:18.225 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:18.226 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:18.226 DEBUG:boto:Method: GET 2022-09-13T07:29:18.227 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:18.228 DEBUG:boto:Data: 2022-09-13T07:29:18.228 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.229 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:18.229 DEBUG:boto:Port: 8001 2022-09-13T07:29:18.230 DEBUG:boto:Params: {} 2022-09-13T07:29:18.230 DEBUG:boto:Token: None 2022-09-13T07:29:18.231 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/?versions 2022-09-13T07:29:18.232 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/N263u65qGfUDoH/ljAAVjlE78I= 2022-09-13T07:29:18.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/N263u65qGfUDoH/ljAAVjlE78I=', 'Content-Length': '0'} 2022-09-13T07:29:18.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5fdb966f13a14a3-006320314e-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.235 DEBUG:boto:b'yxvvuh-15999falsemyobjnulltrue2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:18.236 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-13T07:29:18.237 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:18.237 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:18.238 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:18.238 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:18.239 DEBUG:boto:Method: GET 2022-09-13T07:29:18.239 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:18.240 DEBUG:boto:Data: 2022-09-13T07:29:18.241 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.241 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:18.242 DEBUG:boto:Port: 8000 2022-09-13T07:29:18.242 DEBUG:boto:Params: {} 2022-09-13T07:29:18.243 DEBUG:boto:Token: None 2022-09-13T07:29:18.243 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/?versions 2022-09-13T07:29:18.244 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/N263u65qGfUDoH/ljAAVjlE78I= 2022-09-13T07:29:18.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/N263u65qGfUDoH/ljAAVjlE78I=', 'Content-Length': '0'} 2022-09-13T07:29:18.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba60761fbc57519d-006320314e-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.248 DEBUG:boto:b'yxvvuh-15999falsemyobjnulltrue2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:18.249 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:18.249 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:18.250 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:18.250 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:18.251 DEBUG:boto:Method: GET 2022-09-13T07:29:18.251 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:29:18.252 DEBUG:boto:Data: 2022-09-13T07:29:18.252 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.253 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:18.253 DEBUG:boto:Port: 8001 2022-09-13T07:29:18.254 DEBUG:boto:Params: {} 2022-09-13T07:29:18.255 DEBUG:boto:Token: None 2022-09-13T07:29:18.255 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/?versions 2022-09-13T07:29:18.256 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/N263u65qGfUDoH/ljAAVjlE78I= 2022-09-13T07:29:18.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/N263u65qGfUDoH/ljAAVjlE78I=', 'Content-Length': '0'} 2022-09-13T07:29:18.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d327aefcd8077536-006320314e-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.260 DEBUG:boto:b'yxvvuh-15999falsemyobjnulltrue2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:18.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:18.261 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:18.262 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:18.262 DEBUG:boto:path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.263 DEBUG:boto:auth_path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.263 DEBUG:boto:Method: GET 2022-09-13T07:29:18.264 DEBUG:boto:Path: /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.264 DEBUG:boto:Data: 2022-09-13T07:29:18.265 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.265 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:18.266 DEBUG:boto:Port: 8000 2022-09-13T07:29:18.266 DEBUG:boto:Params: {} 2022-09-13T07:29:18.267 DEBUG:boto:Token: None 2022-09-13T07:29:18.268 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.268 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4TT6ywS09JUqaE42qYS8HCUJA9U= 2022-09-13T07:29:18.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4TT6ywS09JUqaE42qYS8HCUJA9U=', 'Content-Length': '0'} 2022-09-13T07:29:18.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c92c666ddf67aeb-006320314e-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.272 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:18.272 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:18.273 DEBUG:boto:path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.273 DEBUG:boto:auth_path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.274 DEBUG:boto:Method: GET 2022-09-13T07:29:18.275 DEBUG:boto:Path: /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.275 DEBUG:boto:Data: 2022-09-13T07:29:18.276 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.276 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:18.277 DEBUG:boto:Port: 8001 2022-09-13T07:29:18.277 DEBUG:boto:Params: {} 2022-09-13T07:29:18.278 DEBUG:boto:Token: None 2022-09-13T07:29:18.278 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.279 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4TT6ywS09JUqaE42qYS8HCUJA9U= 2022-09-13T07:29:18.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4TT6ywS09JUqaE42qYS8HCUJA9U=', 'Content-Length': '0'} 2022-09-13T07:29:18.282 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7fc46b2471d979a-006320314e-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.283 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:18.284 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:18.285 DEBUG:boto:path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.285 DEBUG:boto:auth_path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.286 DEBUG:boto:Method: HEAD 2022-09-13T07:29:18.286 DEBUG:boto:Path: /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.287 DEBUG:boto:Data: 2022-09-13T07:29:18.287 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.288 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:18.289 DEBUG:boto:Port: 8000 2022-09-13T07:29:18.289 DEBUG:boto:Params: {} 2022-09-13T07:29:18.290 DEBUG:boto:Token: None 2022-09-13T07:29:18.290 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.291 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wvzfzEYd6cSNvKgxUpkMTJt4yRU= 2022-09-13T07:29:18.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wvzfzEYd6cSNvKgxUpkMTJt4yRU=', 'Content-Length': '0'} 2022-09-13T07:29:18.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000621a410fcae369d9-006320314e-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.294 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:18.295 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:18.295 DEBUG:boto:path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.296 DEBUG:boto:auth_path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.296 DEBUG:boto:Method: HEAD 2022-09-13T07:29:18.297 DEBUG:boto:Path: /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.297 DEBUG:boto:Data: 2022-09-13T07:29:18.298 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.298 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:18.299 DEBUG:boto:Port: 8001 2022-09-13T07:29:18.299 DEBUG:boto:Params: {} 2022-09-13T07:29:18.300 DEBUG:boto:Token: None 2022-09-13T07:29:18.301 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.302 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wvzfzEYd6cSNvKgxUpkMTJt4yRU= 2022-09-13T07:29:18.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wvzfzEYd6cSNvKgxUpkMTJt4yRU=', 'Content-Length': '0'} 2022-09-13T07:29:18.306 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055ecd65b882827ae-006320314e-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:18.308 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:18.309 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:18.310 DEBUG:boto:path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.311 DEBUG:boto:auth_path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.312 DEBUG:boto:Method: GET 2022-09-13T07:29:18.313 DEBUG:boto:Path: /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.314 DEBUG:boto:Data: 2022-09-13T07:29:18.315 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.316 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:18.317 DEBUG:boto:Port: 8000 2022-09-13T07:29:18.318 DEBUG:boto:Params: {} 2022-09-13T07:29:18.319 DEBUG:boto:Token: None 2022-09-13T07:29:18.320 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.321 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4TT6ywS09JUqaE42qYS8HCUJA9U= 2022-09-13T07:29:18.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4TT6ywS09JUqaE42qYS8HCUJA9U=', 'Content-Length': '0'} 2022-09-13T07:29:18.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064380a0ca558fc27-006320314e-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.325 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:18.326 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:18.326 DEBUG:boto:path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.327 DEBUG:boto:auth_path=/yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.327 DEBUG:boto:Method: GET 2022-09-13T07:29:18.328 DEBUG:boto:Path: /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.329 DEBUG:boto:Data: 2022-09-13T07:29:18.329 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.330 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:18.330 DEBUG:boto:Port: 8001 2022-09-13T07:29:18.331 DEBUG:boto:Params: {} 2022-09-13T07:29:18.332 DEBUG:boto:Token: None 2022-09-13T07:29:18.333 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/myobj?versionId=null 2022-09-13T07:29:18.334 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4TT6ywS09JUqaE42qYS8HCUJA9U= 2022-09-13T07:29:18.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4TT6ywS09JUqaE42qYS8HCUJA9U=', 'Content-Length': '0'} 2022-09-13T07:29:18.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba80d01253b86957-006320314e-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.378 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:18.379 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:18.379 DEBUG:boto:path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:18.380 DEBUG:boto:auth_path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:18.381 DEBUG:boto:Method: GET 2022-09-13T07:29:18.381 DEBUG:boto:Path: /yxvvuh-15/?prefix=myobj 2022-09-13T07:29:18.382 DEBUG:boto:Data: 2022-09-13T07:29:18.382 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.383 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:18.384 DEBUG:boto:Port: 8000 2022-09-13T07:29:18.384 DEBUG:boto:Params: {} 2022-09-13T07:29:18.385 DEBUG:boto:Token: None 2022-09-13T07:29:18.386 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/ 2022-09-13T07:29:18.386 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ammnEv+NHtor2oP/ckOXsDYSZfQ= 2022-09-13T07:29:18.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ammnEv+NHtor2oP/ckOXsDYSZfQ=', 'Content-Length': '0'} 2022-09-13T07:29:18.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea6bc2a667c53a38-006320314e-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.391 DEBUG:boto:b'yxvvuh-15myobj1000falsemyobj2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:18.392 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:18.393 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:18.393 DEBUG:boto:path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:18.394 DEBUG:boto:auth_path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:18.395 DEBUG:boto:Method: GET 2022-09-13T07:29:18.395 DEBUG:boto:Path: /yxvvuh-15/?prefix=myobj 2022-09-13T07:29:18.396 DEBUG:boto:Data: 2022-09-13T07:29:18.396 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.397 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:18.397 DEBUG:boto:Port: 8001 2022-09-13T07:29:18.398 DEBUG:boto:Params: {} 2022-09-13T07:29:18.398 DEBUG:boto:Token: None 2022-09-13T07:29:18.399 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/ 2022-09-13T07:29:18.400 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ammnEv+NHtor2oP/ckOXsDYSZfQ= 2022-09-13T07:29:18.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ammnEv+NHtor2oP/ckOXsDYSZfQ=', 'Content-Length': '0'} 2022-09-13T07:29:18.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e801c2747707934a-006320314e-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.403 DEBUG:boto:b'yxvvuh-15myobj1000falsemyobj2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:18.404 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:18.405 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:18.406 DEBUG:boto:Method: HEAD 2022-09-13T07:29:18.407 DEBUG:boto:Path: /yxvvuh-15/myobj 2022-09-13T07:29:18.407 DEBUG:boto:Data: 2022-09-13T07:29:18.408 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.408 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:18.409 DEBUG:boto:Port: 8000 2022-09-13T07:29:18.409 DEBUG:boto:Params: {} 2022-09-13T07:29:18.410 DEBUG:boto:Token: None 2022-09-13T07:29:18.411 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/myobj 2022-09-13T07:29:18.411 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Mu1hj1y54ZLAhG8cAiBTmITI4Aw= 2022-09-13T07:29:18.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Mu1hj1y54ZLAhG8cAiBTmITI4Aw=', 'Content-Length': '0'} 2022-09-13T07:29:18.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2cb0d1a96691dbe-006320314e-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.415 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:18.415 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:18.416 DEBUG:boto:Method: HEAD 2022-09-13T07:29:18.416 DEBUG:boto:Path: /yxvvuh-15/myobj 2022-09-13T07:29:18.417 DEBUG:boto:Data: 2022-09-13T07:29:18.417 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.418 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:18.418 DEBUG:boto:Port: 8001 2022-09-13T07:29:18.419 DEBUG:boto:Params: {} 2022-09-13T07:29:18.419 DEBUG:boto:Token: None 2022-09-13T07:29:18.420 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/myobj 2022-09-13T07:29:18.420 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Mu1hj1y54ZLAhG8cAiBTmITI4Aw= 2022-09-13T07:29:18.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Mu1hj1y54ZLAhG8cAiBTmITI4Aw=', 'Content-Length': '0'} 2022-09-13T07:29:18.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067c2dfb836d185dc-006320314e-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-13T07:29:18.429 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:18.429 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:18.430 DEBUG:boto:Method: GET 2022-09-13T07:29:18.430 DEBUG:boto:Path: /yxvvuh-15/myobj 2022-09-13T07:29:18.431 DEBUG:boto:Data: 2022-09-13T07:29:18.431 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.432 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:18.432 DEBUG:boto:Port: 8000 2022-09-13T07:29:18.433 DEBUG:boto:Params: {} 2022-09-13T07:29:18.433 DEBUG:boto:Token: None 2022-09-13T07:29:18.434 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/myobj 2022-09-13T07:29:18.435 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iuWesOExu+h1aSoytlkM9oGFez4= 2022-09-13T07:29:18.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iuWesOExu+h1aSoytlkM9oGFez4=', 'Content-Length': '0'} 2022-09-13T07:29:18.437 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093e1a598b6f98c75-006320314e-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.438 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:18.439 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:18.439 DEBUG:boto:Method: GET 2022-09-13T07:29:18.440 DEBUG:boto:Path: /yxvvuh-15/myobj 2022-09-13T07:29:18.440 DEBUG:boto:Data: 2022-09-13T07:29:18.441 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.442 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:18.442 DEBUG:boto:Port: 8001 2022-09-13T07:29:18.444 DEBUG:boto:Params: {} 2022-09-13T07:29:18.444 DEBUG:boto:Token: None 2022-09-13T07:29:18.445 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/myobj 2022-09-13T07:29:18.446 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iuWesOExu+h1aSoytlkM9oGFez4= 2022-09-13T07:29:18.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iuWesOExu+h1aSoytlkM9oGFez4=', 'Content-Length': '0'} 2022-09-13T07:29:18.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003332e81ed97fadc6-006320314e-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.494 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:18.495 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:18.496 DEBUG:boto:path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:18.497 DEBUG:boto:auth_path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:18.498 DEBUG:boto:Method: GET 2022-09-13T07:29:18.499 DEBUG:boto:Path: /yxvvuh-15/?prefix=myobj 2022-09-13T07:29:18.501 DEBUG:boto:Data: 2022-09-13T07:29:18.502 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.502 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:18.503 DEBUG:boto:Port: 8000 2022-09-13T07:29:18.503 DEBUG:boto:Params: {} 2022-09-13T07:29:18.504 DEBUG:boto:Token: None 2022-09-13T07:29:18.505 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/ 2022-09-13T07:29:18.506 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ammnEv+NHtor2oP/ckOXsDYSZfQ= 2022-09-13T07:29:18.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ammnEv+NHtor2oP/ckOXsDYSZfQ=', 'Content-Length': '0'} 2022-09-13T07:29:18.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c31fd0e6af1baec3-006320314e-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.511 DEBUG:boto:b'yxvvuh-15myobj1000falsemyobj2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:18.512 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:18.512 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:18.513 DEBUG:boto:path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:18.513 DEBUG:boto:auth_path=/yxvvuh-15/?prefix=myobj 2022-09-13T07:29:18.514 DEBUG:boto:Method: GET 2022-09-13T07:29:18.514 DEBUG:boto:Path: /yxvvuh-15/?prefix=myobj 2022-09-13T07:29:18.515 DEBUG:boto:Data: 2022-09-13T07:29:18.515 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.516 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:18.516 DEBUG:boto:Port: 8001 2022-09-13T07:29:18.517 DEBUG:boto:Params: {} 2022-09-13T07:29:18.517 DEBUG:boto:Token: None 2022-09-13T07:29:18.518 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-15/ 2022-09-13T07:29:18.519 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ammnEv+NHtor2oP/ckOXsDYSZfQ= 2022-09-13T07:29:18.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ammnEv+NHtor2oP/ckOXsDYSZfQ=', 'Content-Length': '0'} 2022-09-13T07:29:18.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000474b6689b1454ff8-006320314e-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.523 DEBUG:boto:b'yxvvuh-15myobj1000falsemyobj2022-09-13T07:28:54.956Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:29:18.524 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-15 zones={test-zone2, test-zone3} 2022-09-13T07:29:18.524 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:18.525 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:18.525 DEBUG:boto:Method: HEAD 2022-09-13T07:29:18.526 DEBUG:boto:Path: /yxvvuh-13/ 2022-09-13T07:29:18.526 DEBUG:boto:Data: 2022-09-13T07:29:18.527 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.527 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:18.528 DEBUG:boto:Port: 8000 2022-09-13T07:29:18.528 DEBUG:boto:Params: {} 2022-09-13T07:29:18.529 DEBUG:boto:Token: None 2022-09-13T07:29:18.529 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-13/ 2022-09-13T07:29:18.530 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2qdXzfUfzesuoWvceqhkVTbAar4= 2022-09-13T07:29:18.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2qdXzfUfzesuoWvceqhkVTbAar4=', 'Content-Length': '0'} 2022-09-13T07:29:18.533 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000209ed457b492cc8c-006320314e-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.534 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:18.534 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:18.535 DEBUG:boto:Method: HEAD 2022-09-13T07:29:18.535 DEBUG:boto:Path: /yxvvuh-13/myobj 2022-09-13T07:29:18.536 DEBUG:boto:Data: 2022-09-13T07:29:18.537 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.537 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:18.538 DEBUG:boto:Port: 8000 2022-09-13T07:29:18.539 DEBUG:boto:Params: {} 2022-09-13T07:29:18.540 DEBUG:boto:Token: None 2022-09-13T07:29:18.541 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-13/myobj 2022-09-13T07:29:18.542 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bjbU7JgxRouhdrO19NRHCdsUStM= 2022-09-13T07:29:18.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjbU7JgxRouhdrO19NRHCdsUStM=', 'Content-Length': '0'} 2022-09-13T07:29:18.546 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064437843020cbb74-006320314e-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.547 DEBUG:boto:path=/yxvvuh-13/myobj 2022-09-13T07:29:18.548 DEBUG:boto:auth_path=/yxvvuh-13/myobj 2022-09-13T07:29:18.549 DEBUG:boto:Method: DELETE 2022-09-13T07:29:18.549 DEBUG:boto:Path: /yxvvuh-13/myobj 2022-09-13T07:29:18.550 DEBUG:boto:Data: 2022-09-13T07:29:18.551 DEBUG:boto:Headers: {} 2022-09-13T07:29:18.551 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:18.552 DEBUG:boto:Port: 8000 2022-09-13T07:29:18.552 DEBUG:boto:Params: {} 2022-09-13T07:29:18.553 DEBUG:boto:Token: None 2022-09-13T07:29:18.554 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:29:18 GMT /yxvvuh-13/myobj 2022-09-13T07:29:18.555 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:L9clHNBckZoFWlCSbAzCCCCkgdI= 2022-09-13T07:29:18.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:18 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:L9clHNBckZoFWlCSbAzCCCCkgdI=', 'Content-Length': '0'} 2022-09-13T07:29:18.560 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d8778bab3f0a909a-006320314e-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:29:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:18.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:18.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:18.562 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:29:18.621 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:29:18.622 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:29:18.662 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:29:18.658+0000 7f6df7824600 1 waiting for incremental sync to catch up: 2022-09-13T07:29:18.662 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2022-09-13T07:29:18.663 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2022-09-13T07:29:23.664 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:29:23.660+0000 7f6df7824600 1 waiting for incremental sync to catch up: 2022-09-13T07:29:23.664 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2022-09-13T07:29:23.665 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2022-09-13T07:29:28.665 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:29:28.662+0000 7f6df7824600 1 bucket sync caught up with source: 2022-09-13T07:29:28.666 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2022-09-13T07:29:28.666 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2022-09-13T07:29:28.667 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:29:28.662+0000 7f6df7824600 0 bucket checkpoint complete 2022-09-13T07:29:28.686 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-13 zones={test-zone2, test-zone1} 2022-09-13T07:29:28.687 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:28.687 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:28.688 DEBUG:boto:Method: HEAD 2022-09-13T07:29:28.689 DEBUG:boto:Path: /yxvvuh-13/ 2022-09-13T07:29:28.689 DEBUG:boto:Data: 2022-09-13T07:29:28.690 DEBUG:boto:Headers: {} 2022-09-13T07:29:28.691 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:28.692 DEBUG:boto:Port: 8000 2022-09-13T07:29:28.692 DEBUG:boto:Params: {} 2022-09-13T07:29:28.697 DEBUG:boto:Token: None 2022-09-13T07:29:28.698 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:28 GMT /yxvvuh-13/ 2022-09-13T07:29:28.699 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LoFVF6tz61STsK62D9QfXqKZDsY= 2022-09-13T07:29:28.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LoFVF6tz61STsK62D9QfXqKZDsY=', 'Content-Length': '0'} 2022-09-13T07:29:28.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e67bd6f57ecf8204-0063203158-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:28.706 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:28.706 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:28.707 DEBUG:boto:Method: HEAD 2022-09-13T07:29:28.707 DEBUG:boto:Path: /yxvvuh-13/ 2022-09-13T07:29:28.708 DEBUG:boto:Data: 2022-09-13T07:29:28.709 DEBUG:boto:Headers: {} 2022-09-13T07:29:28.709 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:28.710 DEBUG:boto:Port: 8000 2022-09-13T07:29:28.710 DEBUG:boto:Params: {} 2022-09-13T07:29:28.711 DEBUG:boto:Token: None 2022-09-13T07:29:28.711 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:28 GMT /yxvvuh-13/ 2022-09-13T07:29:28.712 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LoFVF6tz61STsK62D9QfXqKZDsY= 2022-09-13T07:29:28.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LoFVF6tz61STsK62D9QfXqKZDsY=', 'Content-Length': '0'} 2022-09-13T07:29:28.716 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e9a20a77743ea6d-0063203158-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:28.717 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:29:28.717 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:28.718 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:28.718 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:28.719 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:28.719 DEBUG:boto:Method: GET 2022-09-13T07:29:28.720 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:28.720 DEBUG:boto:Data: 2022-09-13T07:29:28.721 DEBUG:boto:Headers: {} 2022-09-13T07:29:28.722 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:28.722 DEBUG:boto:Port: 8000 2022-09-13T07:29:28.723 DEBUG:boto:Params: {} 2022-09-13T07:29:28.724 DEBUG:boto:Token: None 2022-09-13T07:29:28.725 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:28 GMT /yxvvuh-13/?versions 2022-09-13T07:29:28.725 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TlMvAK5bTuaOmdI+e30Huogf30s= 2022-09-13T07:29:28.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TlMvAK5bTuaOmdI+e30Huogf30s=', 'Content-Length': '0'} 2022-09-13T07:29:28.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087e0a163b986e82d-0063203158-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:28.734 DEBUG:boto:b'yxvvuh-13999false' 2022-09-13T07:29:28.735 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:29:28.735 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:28.736 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:28.736 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:28.737 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:28.737 DEBUG:boto:Method: GET 2022-09-13T07:29:28.738 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:28.738 DEBUG:boto:Data: 2022-09-13T07:29:28.739 DEBUG:boto:Headers: {} 2022-09-13T07:29:28.740 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:28.741 DEBUG:boto:Port: 8000 2022-09-13T07:29:28.741 DEBUG:boto:Params: {} 2022-09-13T07:29:28.742 DEBUG:boto:Token: None 2022-09-13T07:29:28.743 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:28 GMT /yxvvuh-13/?versions 2022-09-13T07:29:28.743 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TlMvAK5bTuaOmdI+e30Huogf30s= 2022-09-13T07:29:28.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TlMvAK5bTuaOmdI+e30Huogf30s=', 'Content-Length': '0'} 2022-09-13T07:29:28.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a4722515f20f2d8-0063203158-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:28.747 DEBUG:boto:b'yxvvuh-13999false' 2022-09-13T07:29:28.748 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:28.749 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:28.749 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:28.750 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:28.750 DEBUG:boto:Method: GET 2022-09-13T07:29:28.751 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:28.751 DEBUG:boto:Data: 2022-09-13T07:29:28.752 DEBUG:boto:Headers: {} 2022-09-13T07:29:28.752 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:28.753 DEBUG:boto:Port: 8000 2022-09-13T07:29:28.753 DEBUG:boto:Params: {} 2022-09-13T07:29:28.754 DEBUG:boto:Token: None 2022-09-13T07:29:28.754 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:28 GMT /yxvvuh-13/?versions 2022-09-13T07:29:28.755 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TlMvAK5bTuaOmdI+e30Huogf30s= 2022-09-13T07:29:28.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TlMvAK5bTuaOmdI+e30Huogf30s=', 'Content-Length': '0'} 2022-09-13T07:29:28.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081dc8857e3868089-0063203158-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:28.802 DEBUG:boto:b'yxvvuh-13999false' 2022-09-13T07:29:28.805 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:28.805 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:28.806 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:28.806 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:28.807 DEBUG:boto:Method: GET 2022-09-13T07:29:28.807 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:28.808 DEBUG:boto:Data: 2022-09-13T07:29:28.809 DEBUG:boto:Headers: {} 2022-09-13T07:29:28.809 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:28.810 DEBUG:boto:Port: 8000 2022-09-13T07:29:28.810 DEBUG:boto:Params: {} 2022-09-13T07:29:28.811 DEBUG:boto:Token: None 2022-09-13T07:29:28.811 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:28 GMT /yxvvuh-13/?versions 2022-09-13T07:29:28.812 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TlMvAK5bTuaOmdI+e30Huogf30s= 2022-09-13T07:29:28.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TlMvAK5bTuaOmdI+e30Huogf30s=', 'Content-Length': '0'} 2022-09-13T07:29:28.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d0c81fccc226994-0063203158-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:28.816 DEBUG:boto:b'yxvvuh-13999false' 2022-09-13T07:29:28.817 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-13 zones={test-zone2, test-zone1} 2022-09-13T07:29:28.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:28.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:28.818 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:29:28.884 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:29:28.884 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:29:28.920 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:28.917+0000 7f8eb8454600 1 waiting for incremental sync to catch up: 2022-09-13T07:29:28.921 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2022-09-13T07:29:28.921 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2022-09-13T07:29:33.922 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:33.919+0000 7f8eb8454600 1 bucket sync caught up with source: 2022-09-13T07:29:33.922 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2022-09-13T07:29:33.923 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2022-09-13T07:29:33.923 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:33.919+0000 7f8eb8454600 0 bucket checkpoint complete 2022-09-13T07:29:33.940 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-13 zones={test-zone3, test-zone1} 2022-09-13T07:29:33.941 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:33.942 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:33.943 DEBUG:boto:Method: HEAD 2022-09-13T07:29:33.943 DEBUG:boto:Path: /yxvvuh-13/ 2022-09-13T07:29:33.944 DEBUG:boto:Data: 2022-09-13T07:29:33.945 DEBUG:boto:Headers: {} 2022-09-13T07:29:33.945 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:33.946 DEBUG:boto:Port: 8001 2022-09-13T07:29:33.948 DEBUG:boto:Params: {} 2022-09-13T07:29:33.949 DEBUG:boto:Token: None 2022-09-13T07:29:33.950 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:33 GMT /yxvvuh-13/ 2022-09-13T07:29:33.951 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ve0wiaAXTAUr/hxs6XkmiA31vS8= 2022-09-13T07:29:33.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ve0wiaAXTAUr/hxs6XkmiA31vS8=', 'Content-Length': '0'} 2022-09-13T07:29:33.956 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d558f908b32b02ee-006320315d-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:33.959 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:33.960 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:33.961 DEBUG:boto:Method: HEAD 2022-09-13T07:29:33.961 DEBUG:boto:Path: /yxvvuh-13/ 2022-09-13T07:29:33.963 DEBUG:boto:Data: 2022-09-13T07:29:33.964 DEBUG:boto:Headers: {} 2022-09-13T07:29:33.965 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:33.966 DEBUG:boto:Port: 8000 2022-09-13T07:29:33.967 DEBUG:boto:Params: {} 2022-09-13T07:29:33.968 DEBUG:boto:Token: None 2022-09-13T07:29:33.969 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:33 GMT /yxvvuh-13/ 2022-09-13T07:29:33.970 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ve0wiaAXTAUr/hxs6XkmiA31vS8= 2022-09-13T07:29:33.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ve0wiaAXTAUr/hxs6XkmiA31vS8=', 'Content-Length': '0'} 2022-09-13T07:29:33.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018deb158a52029e3-006320315d-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:33.976 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:29:33.977 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:33.978 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:33.979 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:33.980 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:33.981 DEBUG:boto:Method: GET 2022-09-13T07:29:33.981 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:33.982 DEBUG:boto:Data: 2022-09-13T07:29:33.983 DEBUG:boto:Headers: {} 2022-09-13T07:29:33.984 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:33.985 DEBUG:boto:Port: 8001 2022-09-13T07:29:33.986 DEBUG:boto:Params: {} 2022-09-13T07:29:33.987 DEBUG:boto:Token: None 2022-09-13T07:29:33.988 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:33 GMT /yxvvuh-13/?versions 2022-09-13T07:29:33.989 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OoKjeH2vyEWCunccNA+YK7mSSkQ= 2022-09-13T07:29:33.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OoKjeH2vyEWCunccNA+YK7mSSkQ=', 'Content-Length': '0'} 2022-09-13T07:29:33.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8c20268af53380e-006320315d-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:33.997 DEBUG:boto:b'yxvvuh-13999false' 2022-09-13T07:29:33.998 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:29:33.999 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:34.000 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:34.001 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:34.002 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:34.002 DEBUG:boto:Method: GET 2022-09-13T07:29:34.003 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:34.003 DEBUG:boto:Data: 2022-09-13T07:29:34.004 DEBUG:boto:Headers: {} 2022-09-13T07:29:34.004 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:34.005 DEBUG:boto:Port: 8000 2022-09-13T07:29:34.006 DEBUG:boto:Params: {} 2022-09-13T07:29:34.006 DEBUG:boto:Token: None 2022-09-13T07:29:34.007 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:34 GMT /yxvvuh-13/?versions 2022-09-13T07:29:34.007 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cinW1fZyaUpHeJbtwIuwt/Tkf3M= 2022-09-13T07:29:34.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cinW1fZyaUpHeJbtwIuwt/Tkf3M=', 'Content-Length': '0'} 2022-09-13T07:29:34.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4fad0c089558f8f-006320315e-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:34.014 DEBUG:boto:b'yxvvuh-13999false' 2022-09-13T07:29:34.015 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:34.016 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:34.016 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:34.017 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:34.017 DEBUG:boto:Method: GET 2022-09-13T07:29:34.018 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:34.018 DEBUG:boto:Data: 2022-09-13T07:29:34.019 DEBUG:boto:Headers: {} 2022-09-13T07:29:34.020 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:34.020 DEBUG:boto:Port: 8001 2022-09-13T07:29:34.021 DEBUG:boto:Params: {} 2022-09-13T07:29:34.021 DEBUG:boto:Token: None 2022-09-13T07:29:34.022 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:34 GMT /yxvvuh-13/?versions 2022-09-13T07:29:34.023 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cinW1fZyaUpHeJbtwIuwt/Tkf3M= 2022-09-13T07:29:34.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cinW1fZyaUpHeJbtwIuwt/Tkf3M=', 'Content-Length': '0'} 2022-09-13T07:29:34.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5712a79bb752a82-006320315e-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:34.030 DEBUG:boto:b'yxvvuh-13999false' 2022-09-13T07:29:34.031 DEBUG:boto:path=/yxvvuh-13/ 2022-09-13T07:29:34.032 DEBUG:boto:auth_path=/yxvvuh-13/ 2022-09-13T07:29:34.032 DEBUG:boto:path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:34.033 DEBUG:boto:auth_path=/yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:34.033 DEBUG:boto:Method: GET 2022-09-13T07:29:34.034 DEBUG:boto:Path: /yxvvuh-13/?versions&max-keys=999 2022-09-13T07:29:34.035 DEBUG:boto:Data: 2022-09-13T07:29:34.036 DEBUG:boto:Headers: {} 2022-09-13T07:29:34.036 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:34.037 DEBUG:boto:Port: 8000 2022-09-13T07:29:34.037 DEBUG:boto:Params: {} 2022-09-13T07:29:34.038 DEBUG:boto:Token: None 2022-09-13T07:29:34.039 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:34 GMT /yxvvuh-13/?versions 2022-09-13T07:29:34.040 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cinW1fZyaUpHeJbtwIuwt/Tkf3M= 2022-09-13T07:29:34.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cinW1fZyaUpHeJbtwIuwt/Tkf3M=', 'Content-Length': '0'} 2022-09-13T07:29:34.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055ecc5d1ffad036f-006320315e-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:34.093 DEBUG:boto:b'yxvvuh-13999false' 2022-09-13T07:29:34.095 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-13 zones={test-zone3, test-zone1} 2022-09-13T07:29:34.095 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:34.096 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:34.096 DEBUG:boto:Method: HEAD 2022-09-13T07:29:34.097 DEBUG:boto:Path: /yxvvuh-14/ 2022-09-13T07:29:34.097 DEBUG:boto:Data: 2022-09-13T07:29:34.098 DEBUG:boto:Headers: {} 2022-09-13T07:29:34.099 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:34.099 DEBUG:boto:Port: 8000 2022-09-13T07:29:34.100 DEBUG:boto:Params: {} 2022-09-13T07:29:34.100 DEBUG:boto:Token: None 2022-09-13T07:29:34.101 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:34 GMT /yxvvuh-14/ 2022-09-13T07:29:34.102 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hp/1csnKVQeMmrayTT4cBek+jIE= 2022-09-13T07:29:34.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hp/1csnKVQeMmrayTT4cBek+jIE=', 'Content-Length': '0'} 2022-09-13T07:29:34.105 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000518fd11c2b9626f-006320315e-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:34.106 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:34.107 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:34.107 DEBUG:boto:Method: HEAD 2022-09-13T07:29:34.108 DEBUG:boto:Path: /yxvvuh-14/myobj 2022-09-13T07:29:34.108 DEBUG:boto:Data: 2022-09-13T07:29:34.108 DEBUG:boto:Headers: {} 2022-09-13T07:29:34.109 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:34.110 DEBUG:boto:Port: 8000 2022-09-13T07:29:34.110 DEBUG:boto:Params: {} 2022-09-13T07:29:34.111 DEBUG:boto:Token: None 2022-09-13T07:29:34.111 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:34 GMT /yxvvuh-14/myobj 2022-09-13T07:29:34.112 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Lrdk21MqNQl/rFnYmsX5A+4mPwk= 2022-09-13T07:29:34.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Lrdk21MqNQl/rFnYmsX5A+4mPwk=', 'Content-Length': '0'} 2022-09-13T07:29:34.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a7f3c6484b0a871-006320315e-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:34.115 DEBUG:boto:path=/yxvvuh-14/myobj 2022-09-13T07:29:34.116 DEBUG:boto:auth_path=/yxvvuh-14/myobj 2022-09-13T07:29:34.116 DEBUG:boto:Method: DELETE 2022-09-13T07:29:34.117 DEBUG:boto:Path: /yxvvuh-14/myobj 2022-09-13T07:29:34.117 DEBUG:boto:Data: 2022-09-13T07:29:34.118 DEBUG:boto:Headers: {} 2022-09-13T07:29:34.118 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:34.119 DEBUG:boto:Port: 8000 2022-09-13T07:29:34.119 DEBUG:boto:Params: {} 2022-09-13T07:29:34.120 DEBUG:boto:Token: None 2022-09-13T07:29:34.120 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:29:34 GMT /yxvvuh-14/myobj 2022-09-13T07:29:34.121 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nxRgQJiIZ2X0ea3/6rIlziArriM= 2022-09-13T07:29:34.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nxRgQJiIZ2X0ea3/6rIlziArriM=', 'Content-Length': '0'} 2022-09-13T07:29:34.128 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ea27d39b10cf3e1f-006320315e-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:29:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:34.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:34.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:34.130 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:29:34.200 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:29:34.200 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:29:34.250 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:34.247+0000 7fc718fdd600 1 waiting for incremental sync to catch up: 2022-09-13T07:29:34.250 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2022-09-13T07:29:34.250 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2022-09-13T07:29:39.251 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:39.249+0000 7fc718fdd600 1 waiting for incremental sync to catch up: 2022-09-13T07:29:39.252 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2022-09-13T07:29:39.252 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2022-09-13T07:29:44.253 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:44.250+0000 7fc718fdd600 1 waiting for incremental sync to catch up: 2022-09-13T07:29:44.254 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2022-09-13T07:29:44.254 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2022-09-13T07:29:49.254 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:49.252+0000 7fc718fdd600 1 bucket sync caught up with source: 2022-09-13T07:29:49.255 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2022-09-13T07:29:49.256 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2022-09-13T07:29:49.256 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:49.252+0000 7fc718fdd600 0 bucket checkpoint complete 2022-09-13T07:29:49.278 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-14 zones={test-zone1, test-zone2} 2022-09-13T07:29:49.279 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:49.279 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:49.280 DEBUG:boto:Method: HEAD 2022-09-13T07:29:49.280 DEBUG:boto:Path: /yxvvuh-14/ 2022-09-13T07:29:49.281 DEBUG:boto:Data: 2022-09-13T07:29:49.281 DEBUG:boto:Headers: {} 2022-09-13T07:29:49.282 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:49.283 DEBUG:boto:Port: 8000 2022-09-13T07:29:49.284 DEBUG:boto:Params: {} 2022-09-13T07:29:49.284 DEBUG:boto:Token: None 2022-09-13T07:29:49.285 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:49 GMT /yxvvuh-14/ 2022-09-13T07:29:49.286 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pksb9+QtPBcqAkUC4lpwgv4Iz1s= 2022-09-13T07:29:49.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pksb9+QtPBcqAkUC4lpwgv4Iz1s=', 'Content-Length': '0'} 2022-09-13T07:29:49.290 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c85ec82a6bf80e7-006320316d-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:49.291 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:49.292 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:49.292 DEBUG:boto:Method: HEAD 2022-09-13T07:29:49.293 DEBUG:boto:Path: /yxvvuh-14/ 2022-09-13T07:29:49.293 DEBUG:boto:Data: 2022-09-13T07:29:49.294 DEBUG:boto:Headers: {} 2022-09-13T07:29:49.295 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:49.295 DEBUG:boto:Port: 8000 2022-09-13T07:29:49.296 DEBUG:boto:Params: {} 2022-09-13T07:29:49.296 DEBUG:boto:Token: None 2022-09-13T07:29:49.297 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:49 GMT /yxvvuh-14/ 2022-09-13T07:29:49.297 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pksb9+QtPBcqAkUC4lpwgv4Iz1s= 2022-09-13T07:29:49.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pksb9+QtPBcqAkUC4lpwgv4Iz1s=', 'Content-Length': '0'} 2022-09-13T07:29:49.301 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a6c218cb86d98fb-006320316d-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:49.301 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:29:49.302 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:49.303 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:49.303 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:49.304 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:49.304 DEBUG:boto:Method: GET 2022-09-13T07:29:49.305 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:49.305 DEBUG:boto:Data: 2022-09-13T07:29:49.306 DEBUG:boto:Headers: {} 2022-09-13T07:29:49.306 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:49.307 DEBUG:boto:Port: 8000 2022-09-13T07:29:49.307 DEBUG:boto:Params: {} 2022-09-13T07:29:49.308 DEBUG:boto:Token: None 2022-09-13T07:29:49.308 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:49 GMT /yxvvuh-14/?versions 2022-09-13T07:29:49.309 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vKYP9J8PSpYjbfRXC7FxNERgZeI= 2022-09-13T07:29:49.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vKYP9J8PSpYjbfRXC7FxNERgZeI=', 'Content-Length': '0'} 2022-09-13T07:29:49.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006734c551d63b7029-006320316d-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:49.313 DEBUG:boto:b'yxvvuh-14999false' 2022-09-13T07:29:49.314 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:29:49.314 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:49.315 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:49.315 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:49.316 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:49.316 DEBUG:boto:Method: GET 2022-09-13T07:29:49.317 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:49.317 DEBUG:boto:Data: 2022-09-13T07:29:49.318 DEBUG:boto:Headers: {} 2022-09-13T07:29:49.318 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:49.319 DEBUG:boto:Port: 8000 2022-09-13T07:29:49.319 DEBUG:boto:Params: {} 2022-09-13T07:29:49.320 DEBUG:boto:Token: None 2022-09-13T07:29:49.320 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:49 GMT /yxvvuh-14/?versions 2022-09-13T07:29:49.321 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vKYP9J8PSpYjbfRXC7FxNERgZeI= 2022-09-13T07:29:49.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vKYP9J8PSpYjbfRXC7FxNERgZeI=', 'Content-Length': '0'} 2022-09-13T07:29:49.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af3896c81b86aebb-006320316d-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:49.325 DEBUG:boto:b'yxvvuh-14999false' 2022-09-13T07:29:49.326 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:49.326 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:49.327 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:49.327 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:49.328 DEBUG:boto:Method: GET 2022-09-13T07:29:49.328 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:49.329 DEBUG:boto:Data: 2022-09-13T07:29:49.329 DEBUG:boto:Headers: {} 2022-09-13T07:29:49.330 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:29:49.330 DEBUG:boto:Port: 8000 2022-09-13T07:29:49.331 DEBUG:boto:Params: {} 2022-09-13T07:29:49.331 DEBUG:boto:Token: None 2022-09-13T07:29:49.332 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:49 GMT /yxvvuh-14/?versions 2022-09-13T07:29:49.333 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vKYP9J8PSpYjbfRXC7FxNERgZeI= 2022-09-13T07:29:49.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vKYP9J8PSpYjbfRXC7FxNERgZeI=', 'Content-Length': '0'} 2022-09-13T07:29:49.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003752fd76f5b1e571-006320316d-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:49.336 DEBUG:boto:b'yxvvuh-14999false' 2022-09-13T07:29:49.337 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:49.337 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:49.338 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:49.338 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:49.339 DEBUG:boto:Method: GET 2022-09-13T07:29:49.340 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:49.340 DEBUG:boto:Data: 2022-09-13T07:29:49.341 DEBUG:boto:Headers: {} 2022-09-13T07:29:49.341 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:49.342 DEBUG:boto:Port: 8000 2022-09-13T07:29:49.342 DEBUG:boto:Params: {} 2022-09-13T07:29:49.343 DEBUG:boto:Token: None 2022-09-13T07:29:49.343 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:49 GMT /yxvvuh-14/?versions 2022-09-13T07:29:49.344 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vKYP9J8PSpYjbfRXC7FxNERgZeI= 2022-09-13T07:29:49.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vKYP9J8PSpYjbfRXC7FxNERgZeI=', 'Content-Length': '0'} 2022-09-13T07:29:49.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e0079670a437f41-006320316d-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:49.347 DEBUG:boto:b'yxvvuh-14999false' 2022-09-13T07:29:49.348 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-14 zones={test-zone1, test-zone2} 2022-09-13T07:29:49.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:49.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:49.350 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:29:49.416 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:29:49.417 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:29:49.469 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:49.466+0000 7f98ccc7d600 1 waiting for incremental sync to catch up: 2022-09-13T07:29:49.469 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2022-09-13T07:29:49.469 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2022-09-13T07:29:54.471 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:54.468+0000 7f98ccc7d600 1 bucket sync caught up with source: 2022-09-13T07:29:54.471 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2022-09-13T07:29:54.471 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2022-09-13T07:29:54.472 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:54.468+0000 7f98ccc7d600 0 bucket checkpoint complete 2022-09-13T07:29:54.488 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-14 zones={test-zone3, test-zone2} 2022-09-13T07:29:54.489 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:54.489 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:54.490 DEBUG:boto:Method: HEAD 2022-09-13T07:29:54.490 DEBUG:boto:Path: /yxvvuh-14/ 2022-09-13T07:29:54.491 DEBUG:boto:Data: 2022-09-13T07:29:54.491 DEBUG:boto:Headers: {} 2022-09-13T07:29:54.492 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:54.492 DEBUG:boto:Port: 8001 2022-09-13T07:29:54.493 DEBUG:boto:Params: {} 2022-09-13T07:29:54.493 DEBUG:boto:Token: None 2022-09-13T07:29:54.494 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:54 GMT /yxvvuh-14/ 2022-09-13T07:29:54.495 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zJHlsXWxqcywuIwbK1cVC/hjWlY= 2022-09-13T07:29:54.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zJHlsXWxqcywuIwbK1cVC/hjWlY=', 'Content-Length': '0'} 2022-09-13T07:29:54.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000838bf4a624a7fc75-0063203172-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:54.498 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:54.499 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:54.499 DEBUG:boto:Method: HEAD 2022-09-13T07:29:54.500 DEBUG:boto:Path: /yxvvuh-14/ 2022-09-13T07:29:54.500 DEBUG:boto:Data: 2022-09-13T07:29:54.501 DEBUG:boto:Headers: {} 2022-09-13T07:29:54.501 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:54.502 DEBUG:boto:Port: 8000 2022-09-13T07:29:54.502 DEBUG:boto:Params: {} 2022-09-13T07:29:54.503 DEBUG:boto:Token: None 2022-09-13T07:29:54.504 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:54 GMT /yxvvuh-14/ 2022-09-13T07:29:54.504 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zJHlsXWxqcywuIwbK1cVC/hjWlY= 2022-09-13T07:29:54.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zJHlsXWxqcywuIwbK1cVC/hjWlY=', 'Content-Length': '0'} 2022-09-13T07:29:54.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe73f52be3fe39db-0063203172-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:54.508 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:29:54.509 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:54.509 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:54.510 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:54.510 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:54.511 DEBUG:boto:Method: GET 2022-09-13T07:29:54.512 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:54.512 DEBUG:boto:Data: 2022-09-13T07:29:54.513 DEBUG:boto:Headers: {} 2022-09-13T07:29:54.513 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:54.513 DEBUG:boto:Port: 8001 2022-09-13T07:29:54.514 DEBUG:boto:Params: {} 2022-09-13T07:29:54.515 DEBUG:boto:Token: None 2022-09-13T07:29:54.515 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:54 GMT /yxvvuh-14/?versions 2022-09-13T07:29:54.516 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImPr/7w/ffsEi04jE/yIHUZlsYA= 2022-09-13T07:29:54.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImPr/7w/ffsEi04jE/yIHUZlsYA=', 'Content-Length': '0'} 2022-09-13T07:29:54.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcf1561e1cd855e8-0063203172-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:54.562 DEBUG:boto:b'yxvvuh-14999false' 2022-09-13T07:29:54.563 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:29:54.564 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:54.565 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:54.566 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:54.567 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:54.568 DEBUG:boto:Method: GET 2022-09-13T07:29:54.568 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:54.569 DEBUG:boto:Data: 2022-09-13T07:29:54.570 DEBUG:boto:Headers: {} 2022-09-13T07:29:54.571 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:54.572 DEBUG:boto:Port: 8000 2022-09-13T07:29:54.573 DEBUG:boto:Params: {} 2022-09-13T07:29:54.574 DEBUG:boto:Token: None 2022-09-13T07:29:54.575 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:54 GMT /yxvvuh-14/?versions 2022-09-13T07:29:54.576 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImPr/7w/ffsEi04jE/yIHUZlsYA= 2022-09-13T07:29:54.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImPr/7w/ffsEi04jE/yIHUZlsYA=', 'Content-Length': '0'} 2022-09-13T07:29:54.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf44ebad2e890c19-0063203172-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:54.582 DEBUG:boto:b'yxvvuh-14999false' 2022-09-13T07:29:54.583 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:54.584 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:54.585 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:54.586 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:54.586 DEBUG:boto:Method: GET 2022-09-13T07:29:54.587 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:54.588 DEBUG:boto:Data: 2022-09-13T07:29:54.589 DEBUG:boto:Headers: {} 2022-09-13T07:29:54.590 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:54.591 DEBUG:boto:Port: 8001 2022-09-13T07:29:54.592 DEBUG:boto:Params: {} 2022-09-13T07:29:54.593 DEBUG:boto:Token: None 2022-09-13T07:29:54.594 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:54 GMT /yxvvuh-14/?versions 2022-09-13T07:29:54.595 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImPr/7w/ffsEi04jE/yIHUZlsYA= 2022-09-13T07:29:54.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImPr/7w/ffsEi04jE/yIHUZlsYA=', 'Content-Length': '0'} 2022-09-13T07:29:54.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012dfcb6bf9c9663a-0063203172-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:54.601 DEBUG:boto:b'yxvvuh-14999false' 2022-09-13T07:29:54.602 DEBUG:boto:path=/yxvvuh-14/ 2022-09-13T07:29:54.603 DEBUG:boto:auth_path=/yxvvuh-14/ 2022-09-13T07:29:54.604 DEBUG:boto:path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:54.604 DEBUG:boto:auth_path=/yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:54.605 DEBUG:boto:Method: GET 2022-09-13T07:29:54.606 DEBUG:boto:Path: /yxvvuh-14/?versions&max-keys=999 2022-09-13T07:29:54.607 DEBUG:boto:Data: 2022-09-13T07:29:54.607 DEBUG:boto:Headers: {} 2022-09-13T07:29:54.608 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:29:54.608 DEBUG:boto:Port: 8000 2022-09-13T07:29:54.609 DEBUG:boto:Params: {} 2022-09-13T07:29:54.609 DEBUG:boto:Token: None 2022-09-13T07:29:54.610 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:29:54 GMT /yxvvuh-14/?versions 2022-09-13T07:29:54.610 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImPr/7w/ffsEi04jE/yIHUZlsYA= 2022-09-13T07:29:54.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImPr/7w/ffsEi04jE/yIHUZlsYA=', 'Content-Length': '0'} 2022-09-13T07:29:54.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fbe6b049dcfe740-0063203172-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:29:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:54.614 DEBUG:boto:b'yxvvuh-14999false' 2022-09-13T07:29:54.615 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-14 zones={test-zone3, test-zone2} 2022-09-13T07:29:54.615 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:29:54.616 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:29:54.616 DEBUG:boto:Method: HEAD 2022-09-13T07:29:54.616 DEBUG:boto:Path: /yxvvuh-15/ 2022-09-13T07:29:54.617 DEBUG:boto:Data: 2022-09-13T07:29:54.617 DEBUG:boto:Headers: {} 2022-09-13T07:29:54.618 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:54.618 DEBUG:boto:Port: 8001 2022-09-13T07:29:54.619 DEBUG:boto:Params: {} 2022-09-13T07:29:54.619 DEBUG:boto:Token: None 2022-09-13T07:29:54.620 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:54 GMT /yxvvuh-15/ 2022-09-13T07:29:54.620 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/xMf/ziLBGh2ILZm2FW8pXyqDGY= 2022-09-13T07:29:54.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/xMf/ziLBGh2ILZm2FW8pXyqDGY=', 'Content-Length': '0'} 2022-09-13T07:29:54.623 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc914e64dd222d66-0063203172-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:29:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:54.624 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:54.624 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:54.625 DEBUG:boto:Method: HEAD 2022-09-13T07:29:54.625 DEBUG:boto:Path: /yxvvuh-15/myobj 2022-09-13T07:29:54.626 DEBUG:boto:Data: 2022-09-13T07:29:54.626 DEBUG:boto:Headers: {} 2022-09-13T07:29:54.627 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:54.627 DEBUG:boto:Port: 8001 2022-09-13T07:29:54.628 DEBUG:boto:Params: {} 2022-09-13T07:29:54.628 DEBUG:boto:Token: None 2022-09-13T07:29:54.629 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:29:54 GMT /yxvvuh-15/myobj 2022-09-13T07:29:54.629 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KGS3umjvAKnq6qmw0B0L8D1ViCU= 2022-09-13T07:29:54.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KGS3umjvAKnq6qmw0B0L8D1ViCU=', 'Content-Length': '0'} 2022-09-13T07:29:54.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:28:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039865f3c63943b4a-0063203172-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:29:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:54.632 DEBUG:boto:path=/yxvvuh-15/myobj 2022-09-13T07:29:54.633 DEBUG:boto:auth_path=/yxvvuh-15/myobj 2022-09-13T07:29:54.633 DEBUG:boto:Method: DELETE 2022-09-13T07:29:54.634 DEBUG:boto:Path: /yxvvuh-15/myobj 2022-09-13T07:29:54.634 DEBUG:boto:Data: 2022-09-13T07:29:54.635 DEBUG:boto:Headers: {} 2022-09-13T07:29:54.635 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:29:54.636 DEBUG:boto:Port: 8001 2022-09-13T07:29:54.636 DEBUG:boto:Params: {} 2022-09-13T07:29:54.637 DEBUG:boto:Token: None 2022-09-13T07:29:54.637 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:29:54 GMT /yxvvuh-15/myobj 2022-09-13T07:29:54.638 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OxIe/6LN7StDqgmG4jRhxYl2RQU= 2022-09-13T07:29:54.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:29:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OxIe/6LN7StDqgmG4jRhxYl2RQU=', 'Content-Length': '0'} 2022-09-13T07:29:54.645 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bee1844793e3e5cc-0063203172-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:29:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:29:54.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:54.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:29:54.647 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:29:54.711 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:29:54.711 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:29:54.752 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:54.749+0000 7fa56a016600 1 waiting for incremental sync to catch up: 2022-09-13T07:29:54.752 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:29:54.752 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:29:59.753 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:29:59.749+0000 7fa56a016600 1 waiting for incremental sync to catch up: 2022-09-13T07:29:59.753 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:29:59.754 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:30:04.755 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:30:04.751+0000 7fa56a016600 1 waiting for incremental sync to catch up: 2022-09-13T07:30:04.755 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:30:04.756 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:30:09.756 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:30:09.753+0000 7fa56a016600 1 bucket sync caught up with source: 2022-09-13T07:30:09.757 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:30:09.757 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:30:09.757 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:30:09.753+0000 7fa56a016600 0 bucket checkpoint complete 2022-09-13T07:30:09.778 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-15 zones={test-zone1, test-zone3} 2022-09-13T07:30:09.779 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:30:09.780 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:30:09.780 DEBUG:boto:Method: HEAD 2022-09-13T07:30:09.781 DEBUG:boto:Path: /yxvvuh-15/ 2022-09-13T07:30:09.781 DEBUG:boto:Data: 2022-09-13T07:30:09.781 DEBUG:boto:Headers: {} 2022-09-13T07:30:09.782 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:30:09.783 DEBUG:boto:Port: 8000 2022-09-13T07:30:09.783 DEBUG:boto:Params: {} 2022-09-13T07:30:09.784 DEBUG:boto:Token: None 2022-09-13T07:30:09.784 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:30:09 GMT /yxvvuh-15/ 2022-09-13T07:30:09.785 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LKF4KqOiPsimXhmXAhuV2qg+3mc= 2022-09-13T07:30:09.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:30:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LKF4KqOiPsimXhmXAhuV2qg+3mc=', 'Content-Length': '0'} 2022-09-13T07:30:09.789 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006adfdfc3f439b086-0063203181-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:30:09.789 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:30:09.790 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:30:09.790 DEBUG:boto:Method: HEAD 2022-09-13T07:30:09.791 DEBUG:boto:Path: /yxvvuh-15/ 2022-09-13T07:30:09.791 DEBUG:boto:Data: 2022-09-13T07:30:09.792 DEBUG:boto:Headers: {} 2022-09-13T07:30:09.792 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:30:09.793 DEBUG:boto:Port: 8001 2022-09-13T07:30:09.793 DEBUG:boto:Params: {} 2022-09-13T07:30:09.794 DEBUG:boto:Token: None 2022-09-13T07:30:09.795 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:30:09 GMT /yxvvuh-15/ 2022-09-13T07:30:09.795 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LKF4KqOiPsimXhmXAhuV2qg+3mc= 2022-09-13T07:30:09.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:30:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LKF4KqOiPsimXhmXAhuV2qg+3mc=', 'Content-Length': '0'} 2022-09-13T07:30:09.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b3cbf8a7594a1b7-0063203181-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:30:09.800 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:30:09.800 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:30:09.801 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:30:09.801 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:30:09.802 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:30:09.802 DEBUG:boto:Method: GET 2022-09-13T07:30:09.803 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:30:09.803 DEBUG:boto:Data: 2022-09-13T07:30:09.804 DEBUG:boto:Headers: {} 2022-09-13T07:30:09.804 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:30:09.805 DEBUG:boto:Port: 8000 2022-09-13T07:30:09.805 DEBUG:boto:Params: {} 2022-09-13T07:30:09.806 DEBUG:boto:Token: None 2022-09-13T07:30:09.807 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:30:09 GMT /yxvvuh-15/?versions 2022-09-13T07:30:09.808 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hzcwMpp5LT/KK9H8h7+MbeS1POQ= 2022-09-13T07:30:09.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:30:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hzcwMpp5LT/KK9H8h7+MbeS1POQ=', 'Content-Length': '0'} 2022-09-13T07:30:09.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5f983d3c89965b9-0063203181-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:30:09.812 DEBUG:boto:b'yxvvuh-15999false' 2022-09-13T07:30:09.812 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:30:09.813 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:30:09.813 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:30:09.814 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:30:09.814 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:30:09.815 DEBUG:boto:Method: GET 2022-09-13T07:30:09.816 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:30:09.816 DEBUG:boto:Data: 2022-09-13T07:30:09.817 DEBUG:boto:Headers: {} 2022-09-13T07:30:09.818 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:30:09.818 DEBUG:boto:Port: 8001 2022-09-13T07:30:09.819 DEBUG:boto:Params: {} 2022-09-13T07:30:09.819 DEBUG:boto:Token: None 2022-09-13T07:30:09.820 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:30:09 GMT /yxvvuh-15/?versions 2022-09-13T07:30:09.820 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hzcwMpp5LT/KK9H8h7+MbeS1POQ= 2022-09-13T07:30:09.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:30:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hzcwMpp5LT/KK9H8h7+MbeS1POQ=', 'Content-Length': '0'} 2022-09-13T07:30:09.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000116252f156dde1ca-0063203181-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:30:09.824 DEBUG:boto:b'yxvvuh-15999false' 2022-09-13T07:30:09.825 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:30:09.825 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:30:09.826 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:30:09.826 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:30:09.827 DEBUG:boto:Method: GET 2022-09-13T07:30:09.827 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:30:09.828 DEBUG:boto:Data: 2022-09-13T07:30:09.828 DEBUG:boto:Headers: {} 2022-09-13T07:30:09.829 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:30:09.829 DEBUG:boto:Port: 8000 2022-09-13T07:30:09.830 DEBUG:boto:Params: {} 2022-09-13T07:30:09.831 DEBUG:boto:Token: None 2022-09-13T07:30:09.831 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:30:09 GMT /yxvvuh-15/?versions 2022-09-13T07:30:09.832 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hzcwMpp5LT/KK9H8h7+MbeS1POQ= 2022-09-13T07:30:09.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:30:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hzcwMpp5LT/KK9H8h7+MbeS1POQ=', 'Content-Length': '0'} 2022-09-13T07:30:09.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009823559f0f09848e-0063203181-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:30:09.837 DEBUG:boto:b'yxvvuh-15999false' 2022-09-13T07:30:09.838 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:30:09.838 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:30:09.839 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:30:09.839 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:30:09.840 DEBUG:boto:Method: GET 2022-09-13T07:30:09.840 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:30:09.841 DEBUG:boto:Data: 2022-09-13T07:30:09.841 DEBUG:boto:Headers: {} 2022-09-13T07:30:09.842 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:30:09.842 DEBUG:boto:Port: 8001 2022-09-13T07:30:09.843 DEBUG:boto:Params: {} 2022-09-13T07:30:09.843 DEBUG:boto:Token: None 2022-09-13T07:30:09.844 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:30:09 GMT /yxvvuh-15/?versions 2022-09-13T07:30:09.844 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hzcwMpp5LT/KK9H8h7+MbeS1POQ= 2022-09-13T07:30:09.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:30:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hzcwMpp5LT/KK9H8h7+MbeS1POQ=', 'Content-Length': '0'} 2022-09-13T07:30:09.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc4b840132640d54-0063203181-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:30:09.848 DEBUG:boto:b'yxvvuh-15999false' 2022-09-13T07:30:09.849 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-15 zones={test-zone1, test-zone3} 2022-09-13T07:30:09.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:30:09.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:30:09.851 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:30:09.915 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:30:09.917 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:30:09.957 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:30:09.953+0000 7f736a616600 1 waiting for incremental sync to catch up: 2022-09-13T07:30:09.958 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:30:09.958 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:30:14.958 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:30:14.955+0000 7f736a616600 1 waiting for incremental sync to catch up: 2022-09-13T07:30:14.959 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:30:14.959 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:30:19.960 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:30:19.957+0000 7f736a616600 1 waiting for incremental sync to catch up: 2022-09-13T07:30:19.961 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:30:19.961 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:30:24.962 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:30:24.959+0000 7f736a616600 1 waiting for incremental sync to catch up: 2022-09-13T07:30:24.962 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:30:24.962 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:30:29.963 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:30:29.959+0000 7f736a616600 1 waiting for incremental sync to catch up: 2022-09-13T07:30:29.963 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:30:29.964 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:30:34.965 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:30:34.961+0000 7f736a616600 1 waiting for incremental sync to catch up: 2022-09-13T07:30:34.965 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:30:34.966 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:30:39.966 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:30:39.962+0000 7f736a616600 1 waiting for incremental sync to catch up: 2022-09-13T07:30:39.966 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:30:39.967 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:30:44.968 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:30:44.963+0000 7f736a616600 1 waiting for incremental sync to catch up: 2022-09-13T07:30:44.969 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:30:44.969 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:30:49.968 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:30:49.965+0000 7f736a616600 1 waiting for incremental sync to catch up: 2022-09-13T07:30:49.969 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:30:49.969 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:30:54.970 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:30:54.966+0000 7f736a616600 1 waiting for incremental sync to catch up: 2022-09-13T07:30:54.971 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:30:54.971 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:30:59.971 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:30:59.968+0000 7f736a616600 1 waiting for incremental sync to catch up: 2022-09-13T07:30:59.972 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:30:59.972 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:31:04.972 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:31:04.970+0000 7f736a616600 1 waiting for incremental sync to catch up: 2022-09-13T07:31:04.973 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000001.36.5, , , , , , , , , ] 2022-09-13T07:31:04.973 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:31:09.974 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:31:09.970+0000 7f736a616600 1 bucket sync caught up with source: 2022-09-13T07:31:09.975 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:31:09.975 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000002.38.5, , , , , , , , , ] 2022-09-13T07:31:09.975 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:31:09.970+0000 7f736a616600 0 bucket checkpoint complete 2022-09-13T07:31:09.990 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-15 zones={test-zone2, test-zone3} 2022-09-13T07:31:09.991 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:31:09.992 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:31:09.993 DEBUG:boto:Method: HEAD 2022-09-13T07:31:09.993 DEBUG:boto:Path: /yxvvuh-15/ 2022-09-13T07:31:09.994 DEBUG:boto:Data: 2022-09-13T07:31:09.994 DEBUG:boto:Headers: {} 2022-09-13T07:31:09.995 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:31:09.995 DEBUG:boto:Port: 8000 2022-09-13T07:31:09.996 DEBUG:boto:Params: {} 2022-09-13T07:31:09.997 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T07:31:09.998 DEBUG:boto:Token: None 2022-09-13T07:31:09.998 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:31:09 GMT /yxvvuh-15/ 2022-09-13T07:31:09.999 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:geK0I+bsyuLqcElyXnPlvF3AVDQ= 2022-09-13T07:31:10.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:31:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:geK0I+bsyuLqcElyXnPlvF3AVDQ=', 'Content-Length': '0'} 2022-09-13T07:31:10.004 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e297d2c8b0c34204-00632031bd-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:10.005 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:31:10.005 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:31:10.006 DEBUG:boto:Method: HEAD 2022-09-13T07:31:10.006 DEBUG:boto:Path: /yxvvuh-15/ 2022-09-13T07:31:10.007 DEBUG:boto:Data: 2022-09-13T07:31:10.007 DEBUG:boto:Headers: {} 2022-09-13T07:31:10.008 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:31:10.008 DEBUG:boto:Port: 8001 2022-09-13T07:31:10.009 DEBUG:boto:Params: {} 2022-09-13T07:31:10.009 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-13T07:31:10.010 DEBUG:boto:Token: None 2022-09-13T07:31:10.011 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:31:10 GMT /yxvvuh-15/ 2022-09-13T07:31:10.011 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:u21LrWd13SZqmk9Jr7TXIBCt6Xo= 2022-09-13T07:31:10.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:31:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:u21LrWd13SZqmk9Jr7TXIBCt6Xo=', 'Content-Length': '0'} 2022-09-13T07:31:10.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5c6b5709a4cc1d8-00632031be-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:10.016 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:31:10.017 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:31:10.017 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:31:10.018 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:31:10.018 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:31:10.019 DEBUG:boto:Method: GET 2022-09-13T07:31:10.019 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:31:10.020 DEBUG:boto:Data: 2022-09-13T07:31:10.020 DEBUG:boto:Headers: {} 2022-09-13T07:31:10.021 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:31:10.021 DEBUG:boto:Port: 8000 2022-09-13T07:31:10.022 DEBUG:boto:Params: {} 2022-09-13T07:31:10.022 DEBUG:boto:Token: None 2022-09-13T07:31:10.023 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:31:10 GMT /yxvvuh-15/?versions 2022-09-13T07:31:10.024 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Z1viES1fpgXCZFar1f5BLpM3llI= 2022-09-13T07:31:10.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:31:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Z1viES1fpgXCZFar1f5BLpM3llI=', 'Content-Length': '0'} 2022-09-13T07:31:10.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b116bb03d24cfef-00632031be-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:10.027 DEBUG:boto:b'yxvvuh-15999false' 2022-09-13T07:31:10.028 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:31:10.029 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:31:10.029 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:31:10.030 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:31:10.030 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:31:10.031 DEBUG:boto:Method: GET 2022-09-13T07:31:10.031 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:31:10.032 DEBUG:boto:Data: 2022-09-13T07:31:10.032 DEBUG:boto:Headers: {} 2022-09-13T07:31:10.033 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:31:10.033 DEBUG:boto:Port: 8001 2022-09-13T07:31:10.034 DEBUG:boto:Params: {} 2022-09-13T07:31:10.034 DEBUG:boto:Token: None 2022-09-13T07:31:10.035 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:31:10 GMT /yxvvuh-15/?versions 2022-09-13T07:31:10.036 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Z1viES1fpgXCZFar1f5BLpM3llI= 2022-09-13T07:31:10.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:31:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Z1viES1fpgXCZFar1f5BLpM3llI=', 'Content-Length': '0'} 2022-09-13T07:31:10.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058ba343bee8f2ef3-00632031be-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:10.040 DEBUG:boto:b'yxvvuh-15999false' 2022-09-13T07:31:10.040 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:31:10.041 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:31:10.041 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:31:10.042 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:31:10.042 DEBUG:boto:Method: GET 2022-09-13T07:31:10.043 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:31:10.043 DEBUG:boto:Data: 2022-09-13T07:31:10.044 DEBUG:boto:Headers: {} 2022-09-13T07:31:10.044 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:31:10.045 DEBUG:boto:Port: 8000 2022-09-13T07:31:10.045 DEBUG:boto:Params: {} 2022-09-13T07:31:10.046 DEBUG:boto:Token: None 2022-09-13T07:31:10.046 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:31:10 GMT /yxvvuh-15/?versions 2022-09-13T07:31:10.047 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Z1viES1fpgXCZFar1f5BLpM3llI= 2022-09-13T07:31:10.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:31:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Z1viES1fpgXCZFar1f5BLpM3llI=', 'Content-Length': '0'} 2022-09-13T07:31:10.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a0544bc77b312cc-00632031be-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:10.051 DEBUG:boto:b'yxvvuh-15999false' 2022-09-13T07:31:10.051 DEBUG:boto:path=/yxvvuh-15/ 2022-09-13T07:31:10.052 DEBUG:boto:auth_path=/yxvvuh-15/ 2022-09-13T07:31:10.052 DEBUG:boto:path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:31:10.053 DEBUG:boto:auth_path=/yxvvuh-15/?versions&max-keys=999 2022-09-13T07:31:10.053 DEBUG:boto:Method: GET 2022-09-13T07:31:10.054 DEBUG:boto:Path: /yxvvuh-15/?versions&max-keys=999 2022-09-13T07:31:10.054 DEBUG:boto:Data: 2022-09-13T07:31:10.055 DEBUG:boto:Headers: {} 2022-09-13T07:31:10.055 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:31:10.056 DEBUG:boto:Port: 8001 2022-09-13T07:31:10.056 DEBUG:boto:Params: {} 2022-09-13T07:31:10.057 DEBUG:boto:Token: None 2022-09-13T07:31:10.057 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:31:10 GMT /yxvvuh-15/?versions 2022-09-13T07:31:10.058 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Z1viES1fpgXCZFar1f5BLpM3llI= 2022-09-13T07:31:10.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:31:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Z1viES1fpgXCZFar1f5BLpM3llI=', 'Content-Length': '0'} 2022-09-13T07:31:10.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8ee7bb0fe3f25a3-00632031be-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:10.062 DEBUG:boto:b'yxvvuh-15999false' 2022-09-13T07:31:10.063 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-15 zones={test-zone2, test-zone3} 2022-09-13T07:31:10.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2022-09-13T07:31:10.065 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-16 2022-09-13T07:31:10.066 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:31:10.066 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:31:10.067 DEBUG:boto:Method: PUT 2022-09-13T07:31:10.067 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:31:10.068 DEBUG:boto:Data: 2022-09-13T07:31:10.068 DEBUG:boto:Headers: {} 2022-09-13T07:31:10.069 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:31:10.069 DEBUG:boto:Port: 8000 2022-09-13T07:31:10.069 DEBUG:boto:Params: {} 2022-09-13T07:31:10.070 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T07:31:10.071 DEBUG:boto:Token: None 2022-09-13T07:31:10.071 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:31:10 GMT /yxvvuh-16/ 2022-09-13T07:31:10.072 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6B5uRxEXbkydBVyS/h52x8Nphao= 2022-09-13T07:31:10.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:31:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6B5uRxEXbkydBVyS/h52x8Nphao=', 'Content-Length': '0'} 2022-09-13T07:31:10.107 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000019e060044bfe6afe-00632031be-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:10.108 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-17 2022-09-13T07:31:10.109 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:31:10.110 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:31:10.110 DEBUG:boto:Method: PUT 2022-09-13T07:31:10.111 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:31:10.112 DEBUG:boto:Data: 2022-09-13T07:31:10.113 DEBUG:boto:Headers: {} 2022-09-13T07:31:10.114 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:31:10.114 DEBUG:boto:Port: 8000 2022-09-13T07:31:10.115 DEBUG:boto:Params: {} 2022-09-13T07:31:10.115 DEBUG:boto:Token: None 2022-09-13T07:31:10.116 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:31:10 GMT /yxvvuh-17/ 2022-09-13T07:31:10.116 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Ec/sLkBEj9MImz2z8PFeqZDdPVU= 2022-09-13T07:31:10.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:31:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Ec/sLkBEj9MImz2z8PFeqZDdPVU=', 'Content-Length': '0'} 2022-09-13T07:31:10.152 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009ba4be3fcafc29b1-00632031be-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:10.152 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-18 2022-09-13T07:31:10.153 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:31:10.153 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:31:10.154 DEBUG:boto:Method: PUT 2022-09-13T07:31:10.154 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:31:10.155 DEBUG:boto:Data: 2022-09-13T07:31:10.155 DEBUG:boto:Headers: {} 2022-09-13T07:31:10.156 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:31:10.156 DEBUG:boto:Port: 8001 2022-09-13T07:31:10.157 DEBUG:boto:Params: {} 2022-09-13T07:31:10.157 DEBUG:boto:Token: None 2022-09-13T07:31:10.158 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:31:10 GMT /yxvvuh-18/ 2022-09-13T07:31:10.159 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fzwd3bB5EVzXGoJls5ya8YObYfw= 2022-09-13T07:31:10.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:31:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fzwd3bB5EVzXGoJls5ya8YObYfw=', 'Content-Length': '0'} 2022-09-13T07:31:10.194 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005b51e0574b79040f-00632031be-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:10.196 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:31:10.196 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:31:10.197 DEBUG:boto:path=/yxvvuh-16/?versioning 2022-09-13T07:31:10.197 DEBUG:boto:auth_path=/yxvvuh-16/?versioning 2022-09-13T07:31:10.197 DEBUG:boto:Method: PUT 2022-09-13T07:31:10.198 DEBUG:boto:Path: /yxvvuh-16/?versioning 2022-09-13T07:31:10.198 DEBUG:boto:Data: Enabled Disabled 2022-09-13T07:31:10.199 DEBUG:boto:Headers: {} 2022-09-13T07:31:10.199 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:31:10.200 DEBUG:boto:Port: 8000 2022-09-13T07:31:10.200 DEBUG:boto:Params: {} 2022-09-13T07:31:10.201 DEBUG:boto:Token: None 2022-09-13T07:31:10.201 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:31:10 GMT /yxvvuh-16/?versioning 2022-09-13T07:31:10.202 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ffF+dWwvA6aShl0ih2NdS7ypJw0= 2022-09-13T07:31:10.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:31:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ffF+dWwvA6aShl0ih2NdS7ypJw0=', 'Content-Length': '228'} 2022-09-13T07:31:10.209 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000805c329ac88dff5d-00632031be-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:10.210 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:31:10.210 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:31:10.210 DEBUG:boto:path=/yxvvuh-17/?versioning 2022-09-13T07:31:10.211 DEBUG:boto:auth_path=/yxvvuh-17/?versioning 2022-09-13T07:31:10.211 DEBUG:boto:Method: PUT 2022-09-13T07:31:10.212 DEBUG:boto:Path: /yxvvuh-17/?versioning 2022-09-13T07:31:10.212 DEBUG:boto:Data: Enabled Disabled 2022-09-13T07:31:10.213 DEBUG:boto:Headers: {} 2022-09-13T07:31:10.213 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:31:10.214 DEBUG:boto:Port: 8000 2022-09-13T07:31:10.214 DEBUG:boto:Params: {} 2022-09-13T07:31:10.215 DEBUG:boto:Token: None 2022-09-13T07:31:10.215 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:31:10 GMT /yxvvuh-17/?versioning 2022-09-13T07:31:10.216 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:owDn/svUgRToiTPBs3dAo7ulzXw= 2022-09-13T07:31:10.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:31:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:owDn/svUgRToiTPBs3dAo7ulzXw=', 'Content-Length': '228'} 2022-09-13T07:31:10.225 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004696ed6b231fb0ee-00632031be-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:10.226 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:31:10.226 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:31:10.227 DEBUG:boto:path=/yxvvuh-18/?versioning 2022-09-13T07:31:10.227 DEBUG:boto:auth_path=/yxvvuh-18/?versioning 2022-09-13T07:31:10.228 DEBUG:boto:Method: PUT 2022-09-13T07:31:10.228 DEBUG:boto:Path: /yxvvuh-18/?versioning 2022-09-13T07:31:10.229 DEBUG:boto:Data: Enabled Disabled 2022-09-13T07:31:10.229 DEBUG:boto:Headers: {} 2022-09-13T07:31:10.230 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:31:10.230 DEBUG:boto:Port: 8001 2022-09-13T07:31:10.231 DEBUG:boto:Params: {} 2022-09-13T07:31:10.231 DEBUG:boto:Token: None 2022-09-13T07:31:10.232 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:31:10 GMT /yxvvuh-18/?versioning 2022-09-13T07:31:10.232 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VmCssGVj8bduup++n5mXHbb0ZS4= 2022-09-13T07:31:10.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:31:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VmCssGVj8bduup++n5mXHbb0ZS4=', 'Content-Length': '228'} 2022-09-13T07:31:10.242 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000064f4f367075dc092-00632031be-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:10.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:31:10.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:31:10.244 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:31:10.307 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:31:10.308 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:31:10.363 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:31:10.380 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:31:10.381 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:31:10.381 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:31:10.381 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:31:10.382 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:31:10.382 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:31:10.382 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:31:10.382 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:31:10.383 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:31:10.383 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:31:10.383 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054270.235863_105.1", 2022-09-13T07:31:10.383 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:31:10.235863Z" 2022-09-13T07:31:10.384 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:31:10.384 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:31:10.384 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:31:10.384 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T07:31:10.385 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:31:10.385 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:31:10.387 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054270.235863_105.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:31:10.388 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:31:10.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:31:10.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:31:10.391 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:31:10.445 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:31:10.445 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:31:10.480 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:31:10.476+0000 7f4ff803b600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:31:10.482 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:31:10.477+0000 7f4ff803b600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:31:10.494 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:31:10.494 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:31:10.495 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:31:10.495 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:31:10.495 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:31:10.495 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:31:10.496 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:31:10.496 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:31:10.496 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:31:10.496 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:31:10.497 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:31:10.497 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:31:10.497 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:31:10.497 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:31:10.498 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:31:10.498 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:31:10.498 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:31:10.498 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:31:10.499 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:31:10.499 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:31:10.499 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:31:10.500 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:31:10.500 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:31:10.501 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:31:10.501 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:31:10.502 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:31:10.502 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:31:10.503 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:31:10.503 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:31:10.504 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:31:10.504 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:31:10.505 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:31:10.505 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:31:10.505 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:31:10.506 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:31:10.506 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:31:10.507 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:31:10.507 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054270.235863_105.1", 2022-09-13T07:31:10.508 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:31:10.508 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:31:10.509 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:31:10.509 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:31:10.235863Z", 2022-09-13T07:31:10.510 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:31:10.510 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:31:10.511 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:31:10.511 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:31:10.512 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:31:10.512 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:31:10.513 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:31:10.513 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:31:10.513 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:31:10.514 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:31:10.514 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:31:10.515 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:31:10.515 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:31:10.516 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:31:10.516 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:31:10.516 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:31:10.517 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:31:10.517 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:31:10.518 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:31:10.518 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:31:10.519 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:31:10.519 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:31:10Z" 2022-09-13T07:31:10.519 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:31:10.521 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054270.235863_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:31:10.235863Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:31:10Z" } 2022-09-13T07:31:10.522 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054270.235863_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:31:10.235863Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:31:10.523 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054270.235863_105.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:31:10.524 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054270.235863_105.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:31:10.525 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:31:10.526 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:31:10.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:31:10.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:31:10.528 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:31:10.580 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:31:10.580 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:31:10.612 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:31:10.609+0000 7fb324b47600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:31:10.612 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:31:10.610+0000 7fb324b47600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:31:10.624 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:31:10.625 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:31:10.625 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:31:10.625 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:31:10.626 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:31:10.626 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:31:10.626 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:31:10.626 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:31:10.627 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:31:10.627 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:31:10.627 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:31:10.628 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:31:10.628 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:31:10.628 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:31:10.629 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:31:10.629 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:31:10.629 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:31:10.629 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:31:10.630 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:31:10.630 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:31:10.630 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:31:10.630 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:31:10.631 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:31:10.631 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:31:10.631 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:31:10.631 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:31:10.632 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:31:10.632 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:31:10.632 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:31:10.632 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:31:10.633 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:31:10.633 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:31:10.633 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:31:10.633 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:31:10.634 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:31:10.634 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:31:10.634 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:31:10.634 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054270.235863_105.1", 2022-09-13T07:31:10.635 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:31:10.635 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:31:10.635 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:31:10.635 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:31:10.235863Z", 2022-09-13T07:31:10.636 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:31:10.636 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:31:10.636 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:31:10.636 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:31:10.636 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:31:10.637 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:31:10.637 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:31:10.637 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:31:10.637 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:31:10.638 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:31:10.638 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:31:10.638 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:31:10.639 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:31:10.639 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:31:10.640 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:31:10.640 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:31:10.640 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:31:10.640 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:31:10.640 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:31:10.641 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:31:10.641 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:31:10.641 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:31:10Z" 2022-09-13T07:31:10.641 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:31:10.643 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054270.235863_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:31:10.235863Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:31:10Z" } 2022-09-13T07:31:10.643 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054270.235863_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:31:10.235863Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:31:10.644 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054270.235863_105.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:31:10.644 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054270.235863_105.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:31:10.645 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:31:10.645 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:31:10.646 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:31:10.646 DEBUG:boto:Method: HEAD 2022-09-13T07:31:10.647 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:31:10.647 DEBUG:boto:Data: 2022-09-13T07:31:10.648 DEBUG:boto:Headers: {} 2022-09-13T07:31:10.648 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:31:10.649 DEBUG:boto:Port: 8000 2022-09-13T07:31:10.649 DEBUG:boto:Params: {} 2022-09-13T07:31:10.650 DEBUG:boto:Token: None 2022-09-13T07:31:10.651 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:31:10 GMT /yxvvuh-16/ 2022-09-13T07:31:10.651 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PQ71n4pfsJhvcViUs8Yqgvpt5ig= 2022-09-13T07:31:10.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:31:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PQ71n4pfsJhvcViUs8Yqgvpt5ig=', 'Content-Length': '0'} 2022-09-13T07:31:10.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dca1653d5d792f13-00632031be-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:10.655 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:31:10.655 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:31:10.656 DEBUG:boto:Method: PUT 2022-09-13T07:31:10.656 DEBUG:boto:Path: /yxvvuh-16/dummy 2022-09-13T07:31:10.657 DEBUG:boto:Data: 2022-09-13T07:31:10.657 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T07:31:10.658 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:31:10.658 DEBUG:boto:Port: 8000 2022-09-13T07:31:10.659 DEBUG:boto:Params: {} 2022-09-13T07:31:10.659 DEBUG:boto:Token: None 2022-09-13T07:31:10.660 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 13 Sep 2022 07:31:10 GMT /yxvvuh-16/dummy 2022-09-13T07:31:10.661 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bnka1qjmCIXebVrTQ/olqIs7bUw= 2022-09-13T07:31:10.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:31:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bnka1qjmCIXebVrTQ/olqIs7bUw='} 2022-09-13T07:31:10.673 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-amz-request-id', 'tx00000dd16c4e90f5bdb44-00632031be-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:10.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:31:10.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:31:10.675 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:31:10.732 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:31:10.733 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:31:10.769 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:31:10.766+0000 7fc62d36f600 1 waiting to reach incremental sync.. 2022-09-13T07:31:15.771 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:31:15.766+0000 7fc62d36f600 1 waiting to reach incremental sync.. 2022-09-13T07:31:20.771 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:31:20.767+0000 7fc62d36f600 1 waiting to reach incremental sync.. 2022-09-13T07:31:25.772 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:31:25.768+0000 7fc62d36f600 1 waiting to reach incremental sync.. 2022-09-13T07:31:30.774 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:31:30.770+0000 7fc62d36f600 1 bucket sync caught up with source: 2022-09-13T07:31:30.774 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , 00000000002.13.10, , , , , ] 2022-09-13T07:31:30.774 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , 00000000002.13.10, , , , , ] 2022-09-13T07:31:30.775 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:31:30.770+0000 7fc62d36f600 0 bucket checkpoint complete 2022-09-13T07:31:30.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:31:30.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:31:30.792 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:31:30.860 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:31:30.860 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:31:30.906 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:31:30.902+0000 7f834c5e8600 1 bucket sync caught up with source: 2022-09-13T07:31:30.907 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000002.13.10, , , , , ] 2022-09-13T07:31:30.907 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000002.13.10, , , , , ] 2022-09-13T07:31:30.907 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:31:30.902+0000 7f834c5e8600 0 bucket checkpoint complete 2022-09-13T07:31:30.924 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:31:30.924 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:31:30.925 DEBUG:boto:Method: HEAD 2022-09-13T07:31:30.926 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:31:30.926 DEBUG:boto:Data: 2022-09-13T07:31:30.926 DEBUG:boto:Headers: {} 2022-09-13T07:31:30.927 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:31:30.927 DEBUG:boto:Port: 8000 2022-09-13T07:31:30.928 DEBUG:boto:Params: {} 2022-09-13T07:31:30.929 DEBUG:boto:Token: None 2022-09-13T07:31:30.929 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:31:30 GMT /yxvvuh-17/ 2022-09-13T07:31:30.930 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TYb3/koGHzToV5lM2BSRs6QOfNg= 2022-09-13T07:31:30.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:31:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TYb3/koGHzToV5lM2BSRs6QOfNg=', 'Content-Length': '0'} 2022-09-13T07:31:30.934 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f8d66cc7a7bc538-00632031d2-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:30.934 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:31:30.935 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:31:30.935 DEBUG:boto:Method: PUT 2022-09-13T07:31:30.936 DEBUG:boto:Path: /yxvvuh-17/dummy 2022-09-13T07:31:30.936 DEBUG:boto:Data: 2022-09-13T07:31:30.937 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T07:31:30.937 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:31:30.938 DEBUG:boto:Port: 8000 2022-09-13T07:31:30.938 DEBUG:boto:Params: {} 2022-09-13T07:31:30.939 DEBUG:boto:Token: None 2022-09-13T07:31:30.939 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 13 Sep 2022 07:31:30 GMT /yxvvuh-17/dummy 2022-09-13T07:31:30.940 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xiDVkq0GjyaR5geYwdmmhjr4qkE= 2022-09-13T07:31:30.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:31:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xiDVkq0GjyaR5geYwdmmhjr4qkE='} 2022-09-13T07:31:30.953 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-amz-request-id', 'tx00000240469a42b561da5-00632031d2-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:30.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:31:30.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:31:30.955 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:31:31.020 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:31:31.021 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:31:31.059 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:31:31.056+0000 7f8f5edb2600 1 waiting to reach incremental sync.. 2022-09-13T07:31:36.060 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:31:36.056+0000 7f8f5edb2600 1 waiting to reach incremental sync.. 2022-09-13T07:31:41.061 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:31:41.057+0000 7f8f5edb2600 1 waiting to reach incremental sync.. 2022-09-13T07:31:46.061 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:31:46.058+0000 7f8f5edb2600 1 waiting to reach incremental sync.. 2022-09-13T07:31:51.064 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:31:51.060+0000 7f8f5edb2600 1 bucket sync caught up with source: 2022-09-13T07:31:51.064 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2022-09-13T07:31:51.064 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000002.21.10, , , , , ] 2022-09-13T07:31:51.065 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:31:51.060+0000 7f8f5edb2600 0 bucket checkpoint complete 2022-09-13T07:31:51.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:31:51.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:31:51.089 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:31:51.151 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:31:51.152 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:31:51.187 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:31:51.183+0000 7fc4abc07600 1 waiting for incremental sync to catch up: 2022-09-13T07:31:51.187 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2022-09-13T07:31:51.188 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.25.9, , , , , ] 2022-09-13T07:31:56.188 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:31:56.185+0000 7fc4abc07600 1 bucket sync caught up with source: 2022-09-13T07:31:56.189 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.25.9, , , , , ] 2022-09-13T07:31:56.189 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.25.9, , , , , ] 2022-09-13T07:31:56.189 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:31:56.185+0000 7fc4abc07600 0 bucket checkpoint complete 2022-09-13T07:31:56.205 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:31:56.205 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:31:56.206 DEBUG:boto:Method: HEAD 2022-09-13T07:31:56.206 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:31:56.207 DEBUG:boto:Data: 2022-09-13T07:31:56.207 DEBUG:boto:Headers: {} 2022-09-13T07:31:56.208 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:31:56.208 DEBUG:boto:Port: 8001 2022-09-13T07:31:56.209 DEBUG:boto:Params: {} 2022-09-13T07:31:56.210 DEBUG:boto:Token: None 2022-09-13T07:31:56.210 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:31:56 GMT /yxvvuh-18/ 2022-09-13T07:31:56.211 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MSQkYc43Pycc6tp49nuPw3rqSJY= 2022-09-13T07:31:56.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:31:56 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MSQkYc43Pycc6tp49nuPw3rqSJY=', 'Content-Length': '0'} 2022-09-13T07:31:56.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ae0df1a572d7791-00632031ec-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:56.217 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:31:56.217 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:31:56.218 DEBUG:boto:Method: PUT 2022-09-13T07:31:56.219 DEBUG:boto:Path: /yxvvuh-18/dummy 2022-09-13T07:31:56.220 DEBUG:boto:Data: 2022-09-13T07:31:56.220 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T07:31:56.221 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:31:56.221 DEBUG:boto:Port: 8001 2022-09-13T07:31:56.222 DEBUG:boto:Params: {} 2022-09-13T07:31:56.222 DEBUG:boto:Token: None 2022-09-13T07:31:56.223 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 13 Sep 2022 07:31:56 GMT /yxvvuh-18/dummy 2022-09-13T07:31:56.223 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nLPh7nWkhWNJo5e1BwoY/TXIHGM= 2022-09-13T07:31:56.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:31:56 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLPh7nWkhWNJo5e1BwoY/TXIHGM='} 2022-09-13T07:31:56.239 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-amz-request-id', 'tx00000036ef44e4516cdb7-00632031ec-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:31:56.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:31:56.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:31:56.241 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:31:56.304 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:31:56.306 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:31:56.340 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:31:56.336+0000 7fe2dc1a2600 1 waiting to reach incremental sync.. 2022-09-13T07:32:01.341 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:32:01.337+0000 7fe2dc1a2600 1 waiting to reach incremental sync.. 2022-09-13T07:32:06.343 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:32:06.339+0000 7fe2dc1a2600 1 bucket sync caught up with source: 2022-09-13T07:32:06.343 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000002.24.10, , , , , ] 2022-09-13T07:32:06.344 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000002.24.10, , , , , ] 2022-09-13T07:32:06.344 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:32:06.339+0000 7fe2dc1a2600 0 bucket checkpoint complete 2022-09-13T07:32:06.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:32:06.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:32:06.361 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:32:06.425 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:32:06.426 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:32:06.477 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:32:06.473+0000 7fe151d17600 1 bucket sync caught up with source: 2022-09-13T07:32:06.477 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , 00000000002.24.10, , , , , ] 2022-09-13T07:32:06.478 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , 00000000002.24.10, , , , , ] 2022-09-13T07:32:06.478 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:32:06.473+0000 7fe151d17600 0 bucket checkpoint complete 2022-09-13T07:32:06.495 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:32:06.495 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:32:06.496 DEBUG:boto:Method: HEAD 2022-09-13T07:32:06.496 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:32:06.497 DEBUG:boto:Data: 2022-09-13T07:32:06.497 DEBUG:boto:Headers: {} 2022-09-13T07:32:06.498 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:06.498 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.499 DEBUG:boto:Params: {} 2022-09-13T07:32:06.499 DEBUG:boto:Token: None 2022-09-13T07:32:06.500 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/ 2022-09-13T07:32:06.501 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:N5yhZ/LahDbCGyjEIv8Yuep9YNk= 2022-09-13T07:32:06.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:N5yhZ/LahDbCGyjEIv8Yuep9YNk=', 'Content-Length': '0'} 2022-09-13T07:32:06.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b86b53dc3af2dd98-00632031f6-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.506 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.506 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.507 DEBUG:boto:Method: PUT 2022-09-13T07:32:06.507 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.508 DEBUG:boto:Data: 2022-09-13T07:32:06.508 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:06.509 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:06.509 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.510 DEBUG:boto:Params: {} 2022-09-13T07:32:06.510 DEBUG:boto:Token: None 2022-09-13T07:32:06.511 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.512 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AsJ7gcCTylC3xHk7En3dBGdjSws= 2022-09-13T07:32:06.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AsJ7gcCTylC3xHk7En3dBGdjSws='} 2022-09-13T07:32:06.528 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-amz-request-id', 'tx00000c40dddbcc47ce793-00632031f6-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.529 DEBUG:rgw_multi.tests:version1 id=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:32:06.530 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.530 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.531 DEBUG:boto:Method: PUT 2022-09-13T07:32:06.531 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.532 DEBUG:boto:Data: 2022-09-13T07:32:06.532 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:06.533 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:06.534 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.534 DEBUG:boto:Params: {} 2022-09-13T07:32:06.535 DEBUG:boto:Token: None 2022-09-13T07:32:06.535 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.536 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zEBMBi2CvX8E9BaAUkoq4ueom9k= 2022-09-13T07:32:06.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zEBMBi2CvX8E9BaAUkoq4ueom9k='} 2022-09-13T07:32:06.549 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'rUfn2YiB-yKVwULUrUhTam65WON33RP'), ('x-amz-request-id', 'tx000004c0f4e63518ae7e0-00632031f6-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.550 DEBUG:rgw_multi.tests:version2 id=rUfn2YiB-yKVwULUrUhTam65WON33RP 2022-09-13T07:32:06.550 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.551 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.551 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=rUfn2YiB-yKVwULUrUhTam65WON33RP 2022-09-13T07:32:06.552 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=rUfn2YiB-yKVwULUrUhTam65WON33RP 2022-09-13T07:32:06.552 DEBUG:boto:Method: DELETE 2022-09-13T07:32:06.553 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=rUfn2YiB-yKVwULUrUhTam65WON33RP 2022-09-13T07:32:06.553 DEBUG:boto:Data: 2022-09-13T07:32:06.554 DEBUG:boto:Headers: {} 2022-09-13T07:32:06.554 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:06.555 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.556 DEBUG:boto:Params: {} 2022-09-13T07:32:06.557 DEBUG:boto:Token: None 2022-09-13T07:32:06.558 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/obj-test-zone1?versionId=rUfn2YiB-yKVwULUrUhTam65WON33RP 2022-09-13T07:32:06.558 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+i+0MBeh3IExASJkP28YtAMSlxY= 2022-09-13T07:32:06.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+i+0MBeh3IExASJkP28YtAMSlxY=', 'Content-Length': '0'} 2022-09-13T07:32:06.571 DEBUG:boto:Response headers: [('x-amz-version-id', 'rUfn2YiB-yKVwULUrUhTam65WON33RP'), ('x-amz-request-id', 'tx00000fce5337263984c23-00632031f6-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.572 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.572 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.573 DEBUG:boto:Method: PUT 2022-09-13T07:32:06.573 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.574 DEBUG:boto:Data: 2022-09-13T07:32:06.574 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:06.575 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:06.576 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.576 DEBUG:boto:Params: {} 2022-09-13T07:32:06.577 DEBUG:boto:Token: None 2022-09-13T07:32:06.578 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.578 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FLjanw+576KgJhKcPuVbm18EiRg= 2022-09-13T07:32:06.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FLjanw+576KgJhKcPuVbm18EiRg='} 2022-09-13T07:32:06.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WdrCEAZCV.NaL9ANMvbF8mT5OX5U9mf'), ('x-amz-request-id', 'tx000000aa4c113e26d5859-00632031f6-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.623 DEBUG:rgw_multi.tests:version3 id=WdrCEAZCV.NaL9ANMvbF8mT5OX5U9mf 2022-09-13T07:32:06.624 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.624 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:32:06.625 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=WdrCEAZCV.NaL9ANMvbF8mT5OX5U9mf 2022-09-13T07:32:06.625 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=WdrCEAZCV.NaL9ANMvbF8mT5OX5U9mf 2022-09-13T07:32:06.626 DEBUG:boto:Method: DELETE 2022-09-13T07:32:06.626 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=WdrCEAZCV.NaL9ANMvbF8mT5OX5U9mf 2022-09-13T07:32:06.627 DEBUG:boto:Data: 2022-09-13T07:32:06.627 DEBUG:boto:Headers: {} 2022-09-13T07:32:06.628 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:06.628 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.629 DEBUG:boto:Params: {} 2022-09-13T07:32:06.629 DEBUG:boto:Token: None 2022-09-13T07:32:06.630 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/obj-test-zone1?versionId=WdrCEAZCV.NaL9ANMvbF8mT5OX5U9mf 2022-09-13T07:32:06.631 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fQDsgWZ0vAcS6Uh3upiUeNd/BYA= 2022-09-13T07:32:06.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fQDsgWZ0vAcS6Uh3upiUeNd/BYA=', 'Content-Length': '0'} 2022-09-13T07:32:06.642 DEBUG:boto:Response headers: [('x-amz-version-id', 'WdrCEAZCV.NaL9ANMvbF8mT5OX5U9mf'), ('x-amz-request-id', 'tx000002e77718d8da2ee5c-00632031f6-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.643 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:32:06.644 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:32:06.644 DEBUG:boto:Method: HEAD 2022-09-13T07:32:06.645 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:32:06.645 DEBUG:boto:Data: 2022-09-13T07:32:06.646 DEBUG:boto:Headers: {} 2022-09-13T07:32:06.647 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:06.648 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.649 DEBUG:boto:Params: {} 2022-09-13T07:32:06.650 DEBUG:boto:Token: None 2022-09-13T07:32:06.650 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/ 2022-09-13T07:32:06.651 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:N5yhZ/LahDbCGyjEIv8Yuep9YNk= 2022-09-13T07:32:06.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:N5yhZ/LahDbCGyjEIv8Yuep9YNk=', 'Content-Length': '0'} 2022-09-13T07:32:06.655 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000805e18d141e86ea0-00632031f6-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.656 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.657 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.658 DEBUG:boto:Method: PUT 2022-09-13T07:32:06.659 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.659 DEBUG:boto:Data: 2022-09-13T07:32:06.660 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:06.661 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:06.662 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.663 DEBUG:boto:Params: {} 2022-09-13T07:32:06.663 DEBUG:boto:Token: None 2022-09-13T07:32:06.664 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.665 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SqQ3OxIPX4toq0QVWSqN9cpOAnM= 2022-09-13T07:32:06.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SqQ3OxIPX4toq0QVWSqN9cpOAnM='} 2022-09-13T07:32:06.677 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-amz-request-id', 'tx000006c3043437860b33a-00632031f6-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.678 DEBUG:rgw_multi.tests:version1 id=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:32:06.679 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.680 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.680 DEBUG:boto:Method: PUT 2022-09-13T07:32:06.681 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.682 DEBUG:boto:Data: 2022-09-13T07:32:06.683 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:06.684 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:06.684 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.685 DEBUG:boto:Params: {} 2022-09-13T07:32:06.686 DEBUG:boto:Token: None 2022-09-13T07:32:06.687 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.688 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1dG2oyUH9lRviNGJu7J48KDFS+Q= 2022-09-13T07:32:06.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1dG2oyUH9lRviNGJu7J48KDFS+Q='} 2022-09-13T07:32:06.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '1rGL.oSh.GZaTuKR9yzDHhLYyl9G29L'), ('x-amz-request-id', 'tx00000672712984fd0d9fc-00632031f6-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.731 DEBUG:rgw_multi.tests:version2 id=1rGL.oSh.GZaTuKR9yzDHhLYyl9G29L 2022-09-13T07:32:06.732 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.733 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.734 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=1rGL.oSh.GZaTuKR9yzDHhLYyl9G29L 2022-09-13T07:32:06.734 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=1rGL.oSh.GZaTuKR9yzDHhLYyl9G29L 2022-09-13T07:32:06.735 DEBUG:boto:Method: DELETE 2022-09-13T07:32:06.736 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=1rGL.oSh.GZaTuKR9yzDHhLYyl9G29L 2022-09-13T07:32:06.736 DEBUG:boto:Data: 2022-09-13T07:32:06.737 DEBUG:boto:Headers: {} 2022-09-13T07:32:06.738 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:06.739 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.739 DEBUG:boto:Params: {} 2022-09-13T07:32:06.740 DEBUG:boto:Token: None 2022-09-13T07:32:06.740 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/obj-test-zone2?versionId=1rGL.oSh.GZaTuKR9yzDHhLYyl9G29L 2022-09-13T07:32:06.741 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7pBQysROODkW/C0HR42m3gFPf58= 2022-09-13T07:32:06.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7pBQysROODkW/C0HR42m3gFPf58=', 'Content-Length': '0'} 2022-09-13T07:32:06.750 DEBUG:boto:Response headers: [('x-amz-version-id', '1rGL.oSh.GZaTuKR9yzDHhLYyl9G29L'), ('x-amz-request-id', 'tx00000b9f5e72b16629f5d-00632031f6-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.751 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.752 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.752 DEBUG:boto:Method: PUT 2022-09-13T07:32:06.753 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.753 DEBUG:boto:Data: 2022-09-13T07:32:06.754 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:06.754 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:06.755 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.756 DEBUG:boto:Params: {} 2022-09-13T07:32:06.756 DEBUG:boto:Token: None 2022-09-13T07:32:06.757 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.758 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Z+lGnuBuwVkBY9HJZPZcKIyQM8U= 2022-09-13T07:32:06.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Z+lGnuBuwVkBY9HJZPZcKIyQM8U='} 2022-09-13T07:32:06.769 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'bYtThGOasKGsgKlk654jPehB1L4RJI.'), ('x-amz-request-id', 'tx00000fe31e7c9840cf39e-00632031f6-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.769 DEBUG:rgw_multi.tests:version3 id=bYtThGOasKGsgKlk654jPehB1L4RJI. 2022-09-13T07:32:06.770 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.770 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:32:06.771 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=bYtThGOasKGsgKlk654jPehB1L4RJI. 2022-09-13T07:32:06.771 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=bYtThGOasKGsgKlk654jPehB1L4RJI. 2022-09-13T07:32:06.772 DEBUG:boto:Method: DELETE 2022-09-13T07:32:06.772 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=bYtThGOasKGsgKlk654jPehB1L4RJI. 2022-09-13T07:32:06.773 DEBUG:boto:Data: 2022-09-13T07:32:06.773 DEBUG:boto:Headers: {} 2022-09-13T07:32:06.774 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:06.774 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.775 DEBUG:boto:Params: {} 2022-09-13T07:32:06.775 DEBUG:boto:Token: None 2022-09-13T07:32:06.776 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/obj-test-zone2?versionId=bYtThGOasKGsgKlk654jPehB1L4RJI. 2022-09-13T07:32:06.777 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:M8S0ou+3m4Yah5Vyh45ChBYcKbw= 2022-09-13T07:32:06.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:M8S0ou+3m4Yah5Vyh45ChBYcKbw=', 'Content-Length': '0'} 2022-09-13T07:32:06.786 DEBUG:boto:Response headers: [('x-amz-version-id', 'bYtThGOasKGsgKlk654jPehB1L4RJI.'), ('x-amz-request-id', 'tx00000fb354bd16337f2de-00632031f6-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.787 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:32:06.788 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:32:06.788 DEBUG:boto:Method: HEAD 2022-09-13T07:32:06.789 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:32:06.789 DEBUG:boto:Data: 2022-09-13T07:32:06.790 DEBUG:boto:Headers: {} 2022-09-13T07:32:06.790 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:06.791 DEBUG:boto:Port: 8001 2022-09-13T07:32:06.791 DEBUG:boto:Params: {} 2022-09-13T07:32:06.792 DEBUG:boto:Token: None 2022-09-13T07:32:06.792 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/ 2022-09-13T07:32:06.793 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:N5yhZ/LahDbCGyjEIv8Yuep9YNk= 2022-09-13T07:32:06.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:N5yhZ/LahDbCGyjEIv8Yuep9YNk=', 'Content-Length': '0'} 2022-09-13T07:32:06.797 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d792202616c7249-00632031f6-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.797 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.798 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.799 DEBUG:boto:Method: PUT 2022-09-13T07:32:06.799 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.800 DEBUG:boto:Data: 2022-09-13T07:32:06.800 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:06.801 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:06.801 DEBUG:boto:Port: 8001 2022-09-13T07:32:06.801 DEBUG:boto:Params: {} 2022-09-13T07:32:06.802 DEBUG:boto:Token: None 2022-09-13T07:32:06.803 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.803 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:g80RM2UgGtuvO+1solbuzH9Ef08= 2022-09-13T07:32:06.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:g80RM2UgGtuvO+1solbuzH9Ef08='} 2022-09-13T07:32:06.816 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-amz-request-id', 'tx0000009a5d31224f184bf-00632031f6-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.816 DEBUG:rgw_multi.tests:version1 id=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:32:06.817 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.817 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.818 DEBUG:boto:Method: PUT 2022-09-13T07:32:06.818 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.819 DEBUG:boto:Data: 2022-09-13T07:32:06.819 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:06.820 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:06.820 DEBUG:boto:Port: 8001 2022-09-13T07:32:06.821 DEBUG:boto:Params: {} 2022-09-13T07:32:06.821 DEBUG:boto:Token: None 2022-09-13T07:32:06.822 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.822 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7wHjH4lFU34Gyf2ReE79+2pDwOc= 2022-09-13T07:32:06.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7wHjH4lFU34Gyf2ReE79+2pDwOc='} 2022-09-13T07:32:06.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'FClYptxMZCMz0AuVOs0O6CaRfRFXzRT'), ('x-amz-request-id', 'tx00000914dcbeb4fd8d529-00632031f6-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.835 DEBUG:rgw_multi.tests:version2 id=FClYptxMZCMz0AuVOs0O6CaRfRFXzRT 2022-09-13T07:32:06.835 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.835 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.836 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=FClYptxMZCMz0AuVOs0O6CaRfRFXzRT 2022-09-13T07:32:06.836 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=FClYptxMZCMz0AuVOs0O6CaRfRFXzRT 2022-09-13T07:32:06.837 DEBUG:boto:Method: DELETE 2022-09-13T07:32:06.837 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=FClYptxMZCMz0AuVOs0O6CaRfRFXzRT 2022-09-13T07:32:06.838 DEBUG:boto:Data: 2022-09-13T07:32:06.838 DEBUG:boto:Headers: {} 2022-09-13T07:32:06.839 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:06.839 DEBUG:boto:Port: 8001 2022-09-13T07:32:06.840 DEBUG:boto:Params: {} 2022-09-13T07:32:06.840 DEBUG:boto:Token: None 2022-09-13T07:32:06.841 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/obj-test-zone3?versionId=FClYptxMZCMz0AuVOs0O6CaRfRFXzRT 2022-09-13T07:32:06.841 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:D1bVTP60ktF9plmmlcBCP3wyz3g= 2022-09-13T07:32:06.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:D1bVTP60ktF9plmmlcBCP3wyz3g=', 'Content-Length': '0'} 2022-09-13T07:32:06.858 DEBUG:boto:Response headers: [('x-amz-version-id', 'FClYptxMZCMz0AuVOs0O6CaRfRFXzRT'), ('x-amz-request-id', 'tx000007d2ca9f089e0f9a8-00632031f6-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.859 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.859 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.860 DEBUG:boto:Method: PUT 2022-09-13T07:32:06.860 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.861 DEBUG:boto:Data: 2022-09-13T07:32:06.861 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:06.861 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:06.862 DEBUG:boto:Port: 8001 2022-09-13T07:32:06.862 DEBUG:boto:Params: {} 2022-09-13T07:32:06.863 DEBUG:boto:Token: None 2022-09-13T07:32:06.863 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.864 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PFVA0LA0CxBoCFak+rc9KbGB4pQ= 2022-09-13T07:32:06.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PFVA0LA0CxBoCFak+rc9KbGB4pQ='} 2022-09-13T07:32:06.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'QG1GnIUHLVdDKFaFdiAormiQ6EstjRT'), ('x-amz-request-id', 'tx00000baf2d45dcce04a1f-00632031f6-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.915 DEBUG:rgw_multi.tests:version3 id=QG1GnIUHLVdDKFaFdiAormiQ6EstjRT 2022-09-13T07:32:06.915 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.916 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:32:06.916 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=QG1GnIUHLVdDKFaFdiAormiQ6EstjRT 2022-09-13T07:32:06.917 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=QG1GnIUHLVdDKFaFdiAormiQ6EstjRT 2022-09-13T07:32:06.917 DEBUG:boto:Method: DELETE 2022-09-13T07:32:06.918 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=QG1GnIUHLVdDKFaFdiAormiQ6EstjRT 2022-09-13T07:32:06.918 DEBUG:boto:Data: 2022-09-13T07:32:06.919 DEBUG:boto:Headers: {} 2022-09-13T07:32:06.919 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:06.920 DEBUG:boto:Port: 8001 2022-09-13T07:32:06.920 DEBUG:boto:Params: {} 2022-09-13T07:32:06.921 DEBUG:boto:Token: None 2022-09-13T07:32:06.921 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-16/obj-test-zone3?versionId=QG1GnIUHLVdDKFaFdiAormiQ6EstjRT 2022-09-13T07:32:06.922 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S7TXKGLHB+3ZhZqJdszBEZLLSJQ= 2022-09-13T07:32:06.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S7TXKGLHB+3ZhZqJdszBEZLLSJQ=', 'Content-Length': '0'} 2022-09-13T07:32:06.939 DEBUG:boto:Response headers: [('x-amz-version-id', 'QG1GnIUHLVdDKFaFdiAormiQ6EstjRT'), ('x-amz-request-id', 'tx000004ba87ef417ace2bb-00632031f6-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.940 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:32:06.940 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:32:06.941 DEBUG:boto:Method: HEAD 2022-09-13T07:32:06.941 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:32:06.942 DEBUG:boto:Data: 2022-09-13T07:32:06.942 DEBUG:boto:Headers: {} 2022-09-13T07:32:06.943 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:06.943 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.944 DEBUG:boto:Params: {} 2022-09-13T07:32:06.944 DEBUG:boto:Token: None 2022-09-13T07:32:06.945 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-17/ 2022-09-13T07:32:06.945 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Lfz/sUHWPcpp7tL6roKh8N7JqZg= 2022-09-13T07:32:06.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Lfz/sUHWPcpp7tL6roKh8N7JqZg=', 'Content-Length': '0'} 2022-09-13T07:32:06.948 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da9b6d82b9fb2197-00632031f6-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.949 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:32:06.950 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:32:06.950 DEBUG:boto:Method: PUT 2022-09-13T07:32:06.951 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1 2022-09-13T07:32:06.951 DEBUG:boto:Data: 2022-09-13T07:32:06.952 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:06.952 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:06.952 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.953 DEBUG:boto:Params: {} 2022-09-13T07:32:06.953 DEBUG:boto:Token: None 2022-09-13T07:32:06.954 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-17/obj-test-zone1 2022-09-13T07:32:06.955 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zdntlFhGRAb6n2+SdYnlrPUZp+k= 2022-09-13T07:32:06.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zdntlFhGRAb6n2+SdYnlrPUZp+k='} 2022-09-13T07:32:06.969 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-amz-request-id', 'tx00000bbf720f7cb513dae-00632031f6-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.969 DEBUG:rgw_multi.tests:version1 id=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:32:06.970 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:32:06.970 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:32:06.971 DEBUG:boto:Method: PUT 2022-09-13T07:32:06.972 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1 2022-09-13T07:32:06.972 DEBUG:boto:Data: 2022-09-13T07:32:06.972 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:06.973 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:06.973 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.974 DEBUG:boto:Params: {} 2022-09-13T07:32:06.974 DEBUG:boto:Token: None 2022-09-13T07:32:06.975 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-17/obj-test-zone1 2022-09-13T07:32:06.976 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hQVrz/rhOcxcybVMpDEbYRRweCA= 2022-09-13T07:32:06.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hQVrz/rhOcxcybVMpDEbYRRweCA='} 2022-09-13T07:32:06.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7f9.kDyjLV05jlIgVLP2695-H..NHIC'), ('x-amz-request-id', 'tx000002d303cdcbf171ee2-00632031f6-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:06.991 DEBUG:rgw_multi.tests:version2 id=7f9.kDyjLV05jlIgVLP2695-H..NHIC 2022-09-13T07:32:06.991 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:32:06.992 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:32:06.992 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=7f9.kDyjLV05jlIgVLP2695-H..NHIC 2022-09-13T07:32:06.993 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=7f9.kDyjLV05jlIgVLP2695-H..NHIC 2022-09-13T07:32:06.993 DEBUG:boto:Method: DELETE 2022-09-13T07:32:06.994 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=7f9.kDyjLV05jlIgVLP2695-H..NHIC 2022-09-13T07:32:06.994 DEBUG:boto:Data: 2022-09-13T07:32:06.995 DEBUG:boto:Headers: {} 2022-09-13T07:32:06.995 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:06.996 DEBUG:boto:Port: 8000 2022-09-13T07:32:06.996 DEBUG:boto:Params: {} 2022-09-13T07:32:06.997 DEBUG:boto:Token: None 2022-09-13T07:32:06.997 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:06 GMT /yxvvuh-17/obj-test-zone1?versionId=7f9.kDyjLV05jlIgVLP2695-H..NHIC 2022-09-13T07:32:06.998 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gQJPL6qr3fOmie9PCz2d/aFRcOA= 2022-09-13T07:32:06.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:06 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gQJPL6qr3fOmie9PCz2d/aFRcOA=', 'Content-Length': '0'} 2022-09-13T07:32:07.009 DEBUG:boto:Response headers: [('x-amz-version-id', '7f9.kDyjLV05jlIgVLP2695-H..NHIC'), ('x-amz-request-id', 'tx00000d0b1abad3e38cf35-00632031f6-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.010 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:32:07.010 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:32:07.011 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.012 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1 2022-09-13T07:32:07.013 DEBUG:boto:Data: 2022-09-13T07:32:07.014 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.014 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:07.015 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.015 DEBUG:boto:Params: {} 2022-09-13T07:32:07.016 DEBUG:boto:Token: None 2022-09-13T07:32:07.016 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-17/obj-test-zone1 2022-09-13T07:32:07.017 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PIgaaeVlrii/GxU5jUhZR4kpjg4= 2022-09-13T07:32:07.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PIgaaeVlrii/GxU5jUhZR4kpjg4='} 2022-09-13T07:32:07.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'AD0o5mA0m03jT1jtT06FSGZKwpKodL-'), ('x-amz-request-id', 'tx0000009af097c5e73bf12-00632031f7-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.063 DEBUG:rgw_multi.tests:version3 id=AD0o5mA0m03jT1jtT06FSGZKwpKodL- 2022-09-13T07:32:07.063 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:32:07.064 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:32:07.064 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=AD0o5mA0m03jT1jtT06FSGZKwpKodL- 2022-09-13T07:32:07.065 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=AD0o5mA0m03jT1jtT06FSGZKwpKodL- 2022-09-13T07:32:07.065 DEBUG:boto:Method: DELETE 2022-09-13T07:32:07.066 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=AD0o5mA0m03jT1jtT06FSGZKwpKodL- 2022-09-13T07:32:07.066 DEBUG:boto:Data: 2022-09-13T07:32:07.067 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.068 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:07.068 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.069 DEBUG:boto:Params: {} 2022-09-13T07:32:07.069 DEBUG:boto:Token: None 2022-09-13T07:32:07.070 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-17/obj-test-zone1?versionId=AD0o5mA0m03jT1jtT06FSGZKwpKodL- 2022-09-13T07:32:07.070 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rHJdTn7oUg1NH8zFHvWY1htLbY8= 2022-09-13T07:32:07.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rHJdTn7oUg1NH8zFHvWY1htLbY8=', 'Content-Length': '0'} 2022-09-13T07:32:07.082 DEBUG:boto:Response headers: [('x-amz-version-id', 'AD0o5mA0m03jT1jtT06FSGZKwpKodL-'), ('x-amz-request-id', 'tx00000fe452b7ee54466a9-00632031f7-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.082 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:32:07.083 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:32:07.083 DEBUG:boto:Method: HEAD 2022-09-13T07:32:07.084 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:32:07.084 DEBUG:boto:Data: 2022-09-13T07:32:07.085 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.085 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:07.086 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.086 DEBUG:boto:Params: {} 2022-09-13T07:32:07.087 DEBUG:boto:Token: None 2022-09-13T07:32:07.087 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-17/ 2022-09-13T07:32:07.088 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CiJe+9NS4APASgcLZrhOMAiadt8= 2022-09-13T07:32:07.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CiJe+9NS4APASgcLZrhOMAiadt8=', 'Content-Length': '0'} 2022-09-13T07:32:07.091 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcb10dd1a5444a09-00632031f7-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.091 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.092 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.092 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.093 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.093 DEBUG:boto:Data: 2022-09-13T07:32:07.094 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.094 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:07.095 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.095 DEBUG:boto:Params: {} 2022-09-13T07:32:07.096 DEBUG:boto:Token: None 2022-09-13T07:32:07.097 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.097 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t1OM7G9r/dbGKpiIT81JVVNX2eg= 2022-09-13T07:32:07.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t1OM7G9r/dbGKpiIT81JVVNX2eg='} 2022-09-13T07:32:07.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-amz-request-id', 'tx00000d1548265c12b4ab1-00632031f7-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.131 DEBUG:rgw_multi.tests:version1 id=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:32:07.131 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.132 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.132 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.133 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.134 DEBUG:boto:Data: 2022-09-13T07:32:07.134 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.135 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:07.136 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.136 DEBUG:boto:Params: {} 2022-09-13T07:32:07.137 DEBUG:boto:Token: None 2022-09-13T07:32:07.137 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.138 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SNIH6N5Y/5Xq0UHrDhZaDGj4IoA= 2022-09-13T07:32:07.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SNIH6N5Y/5Xq0UHrDhZaDGj4IoA='} 2022-09-13T07:32:07.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Z.x.5SjxN418k5T5PqKXoyGbK7T0pEI'), ('x-amz-request-id', 'tx00000ae28a90ba6447fa7-00632031f7-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.148 DEBUG:rgw_multi.tests:version2 id=Z.x.5SjxN418k5T5PqKXoyGbK7T0pEI 2022-09-13T07:32:07.149 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.149 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.150 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=Z.x.5SjxN418k5T5PqKXoyGbK7T0pEI 2022-09-13T07:32:07.150 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=Z.x.5SjxN418k5T5PqKXoyGbK7T0pEI 2022-09-13T07:32:07.151 DEBUG:boto:Method: DELETE 2022-09-13T07:32:07.151 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=Z.x.5SjxN418k5T5PqKXoyGbK7T0pEI 2022-09-13T07:32:07.152 DEBUG:boto:Data: 2022-09-13T07:32:07.152 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.153 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:07.154 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.154 DEBUG:boto:Params: {} 2022-09-13T07:32:07.155 DEBUG:boto:Token: None 2022-09-13T07:32:07.155 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-17/obj-test-zone2?versionId=Z.x.5SjxN418k5T5PqKXoyGbK7T0pEI 2022-09-13T07:32:07.156 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:unhn587iIu6FLUYk1XRx5pijvbU= 2022-09-13T07:32:07.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:unhn587iIu6FLUYk1XRx5pijvbU=', 'Content-Length': '0'} 2022-09-13T07:32:07.167 DEBUG:boto:Response headers: [('x-amz-version-id', 'Z.x.5SjxN418k5T5PqKXoyGbK7T0pEI'), ('x-amz-request-id', 'tx00000490d2caadbab7a4f-00632031f7-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.168 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.168 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.169 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.170 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.170 DEBUG:boto:Data: 2022-09-13T07:32:07.171 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.171 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:07.172 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.172 DEBUG:boto:Params: {} 2022-09-13T07:32:07.173 DEBUG:boto:Token: None 2022-09-13T07:32:07.173 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.174 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CAA2qj8eFtbR2PKOB/H9YjujKPM= 2022-09-13T07:32:07.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CAA2qj8eFtbR2PKOB/H9YjujKPM='} 2022-09-13T07:32:07.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'n-.VIlcipqOVF9gf1Rt8ARrDXyx.XyA'), ('x-amz-request-id', 'tx00000caa355d0a823e01c-00632031f7-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.219 DEBUG:rgw_multi.tests:version3 id=n-.VIlcipqOVF9gf1Rt8ARrDXyx.XyA 2022-09-13T07:32:07.220 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.220 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:32:07.220 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=n-.VIlcipqOVF9gf1Rt8ARrDXyx.XyA 2022-09-13T07:32:07.221 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=n-.VIlcipqOVF9gf1Rt8ARrDXyx.XyA 2022-09-13T07:32:07.221 DEBUG:boto:Method: DELETE 2022-09-13T07:32:07.222 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=n-.VIlcipqOVF9gf1Rt8ARrDXyx.XyA 2022-09-13T07:32:07.222 DEBUG:boto:Data: 2022-09-13T07:32:07.223 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.223 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:07.224 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.224 DEBUG:boto:Params: {} 2022-09-13T07:32:07.225 DEBUG:boto:Token: None 2022-09-13T07:32:07.225 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-17/obj-test-zone2?versionId=n-.VIlcipqOVF9gf1Rt8ARrDXyx.XyA 2022-09-13T07:32:07.226 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lZXaUqxeYqWEFqKe1X+7RySP+is= 2022-09-13T07:32:07.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lZXaUqxeYqWEFqKe1X+7RySP+is=', 'Content-Length': '0'} 2022-09-13T07:32:07.236 DEBUG:boto:Response headers: [('x-amz-version-id', 'n-.VIlcipqOVF9gf1Rt8ARrDXyx.XyA'), ('x-amz-request-id', 'tx000008e95450d197dfa8d-00632031f7-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.237 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:32:07.238 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:32:07.238 DEBUG:boto:Method: HEAD 2022-09-13T07:32:07.239 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:32:07.239 DEBUG:boto:Data: 2022-09-13T07:32:07.240 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.241 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:07.241 DEBUG:boto:Port: 8001 2022-09-13T07:32:07.242 DEBUG:boto:Params: {} 2022-09-13T07:32:07.242 DEBUG:boto:Token: None 2022-09-13T07:32:07.243 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-17/ 2022-09-13T07:32:07.243 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CiJe+9NS4APASgcLZrhOMAiadt8= 2022-09-13T07:32:07.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CiJe+9NS4APASgcLZrhOMAiadt8=', 'Content-Length': '0'} 2022-09-13T07:32:07.246 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f5ef69b29a74a77-00632031f7-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.247 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.248 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.248 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.249 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.250 DEBUG:boto:Data: 2022-09-13T07:32:07.250 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.251 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:07.251 DEBUG:boto:Port: 8001 2022-09-13T07:32:07.252 DEBUG:boto:Params: {} 2022-09-13T07:32:07.252 DEBUG:boto:Token: None 2022-09-13T07:32:07.253 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.254 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:A0mbPFY1Nwpfa4cfFM44EvGa0I4= 2022-09-13T07:32:07.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:A0mbPFY1Nwpfa4cfFM44EvGa0I4='} 2022-09-13T07:32:07.265 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-amz-request-id', 'tx00000fef3471aafb93674-00632031f7-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.266 DEBUG:rgw_multi.tests:version1 id=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:32:07.266 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.267 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.267 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.268 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.268 DEBUG:boto:Data: 2022-09-13T07:32:07.269 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.269 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:07.270 DEBUG:boto:Port: 8001 2022-09-13T07:32:07.270 DEBUG:boto:Params: {} 2022-09-13T07:32:07.271 DEBUG:boto:Token: None 2022-09-13T07:32:07.271 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.272 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BFT0sA8U8K76Rijq8GBjTpaKS0A= 2022-09-13T07:32:07.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BFT0sA8U8K76Rijq8GBjTpaKS0A='} 2022-09-13T07:32:07.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'uZAfa5kVPN8N-ITpVISIYO-W5.l4mzY'), ('x-amz-request-id', 'tx0000045be5d9612ec1d7e-00632031f7-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.319 DEBUG:rgw_multi.tests:version2 id=uZAfa5kVPN8N-ITpVISIYO-W5.l4mzY 2022-09-13T07:32:07.319 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.320 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.320 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=uZAfa5kVPN8N-ITpVISIYO-W5.l4mzY 2022-09-13T07:32:07.321 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=uZAfa5kVPN8N-ITpVISIYO-W5.l4mzY 2022-09-13T07:32:07.321 DEBUG:boto:Method: DELETE 2022-09-13T07:32:07.322 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=uZAfa5kVPN8N-ITpVISIYO-W5.l4mzY 2022-09-13T07:32:07.322 DEBUG:boto:Data: 2022-09-13T07:32:07.323 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.323 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:07.335 DEBUG:boto:Port: 8001 2022-09-13T07:32:07.335 DEBUG:boto:Params: {} 2022-09-13T07:32:07.336 DEBUG:boto:Token: None 2022-09-13T07:32:07.336 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-17/obj-test-zone3?versionId=uZAfa5kVPN8N-ITpVISIYO-W5.l4mzY 2022-09-13T07:32:07.337 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:alExE+aeXt19slQH6s0FbBNVdiU= 2022-09-13T07:32:07.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:alExE+aeXt19slQH6s0FbBNVdiU=', 'Content-Length': '0'} 2022-09-13T07:32:07.350 DEBUG:boto:Response headers: [('x-amz-version-id', 'uZAfa5kVPN8N-ITpVISIYO-W5.l4mzY'), ('x-amz-request-id', 'tx0000064cb3adf21e0d117-00632031f7-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.351 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.352 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.352 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.353 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.353 DEBUG:boto:Data: 2022-09-13T07:32:07.354 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.354 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:07.355 DEBUG:boto:Port: 8001 2022-09-13T07:32:07.355 DEBUG:boto:Params: {} 2022-09-13T07:32:07.356 DEBUG:boto:Token: None 2022-09-13T07:32:07.356 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.357 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uBIqVa1yfbNPXWBohZXSg9zj8iA= 2022-09-13T07:32:07.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uBIqVa1yfbNPXWBohZXSg9zj8iA='} 2022-09-13T07:32:07.369 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'MTgn7S8En5Kaee6NWa3nNjgfwiOweDV'), ('x-amz-request-id', 'tx00000c23593038dbaa135-00632031f7-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.370 DEBUG:rgw_multi.tests:version3 id=MTgn7S8En5Kaee6NWa3nNjgfwiOweDV 2022-09-13T07:32:07.371 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.371 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:32:07.372 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=MTgn7S8En5Kaee6NWa3nNjgfwiOweDV 2022-09-13T07:32:07.372 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=MTgn7S8En5Kaee6NWa3nNjgfwiOweDV 2022-09-13T07:32:07.373 DEBUG:boto:Method: DELETE 2022-09-13T07:32:07.373 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=MTgn7S8En5Kaee6NWa3nNjgfwiOweDV 2022-09-13T07:32:07.374 DEBUG:boto:Data: 2022-09-13T07:32:07.374 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.375 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:07.375 DEBUG:boto:Port: 8001 2022-09-13T07:32:07.376 DEBUG:boto:Params: {} 2022-09-13T07:32:07.376 DEBUG:boto:Token: None 2022-09-13T07:32:07.377 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-17/obj-test-zone3?versionId=MTgn7S8En5Kaee6NWa3nNjgfwiOweDV 2022-09-13T07:32:07.377 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:P5e72uldO3B9KNGmrQUFVFYahpU= 2022-09-13T07:32:07.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:P5e72uldO3B9KNGmrQUFVFYahpU=', 'Content-Length': '0'} 2022-09-13T07:32:07.391 DEBUG:boto:Response headers: [('x-amz-version-id', 'MTgn7S8En5Kaee6NWa3nNjgfwiOweDV'), ('x-amz-request-id', 'tx0000039cf8814e8a0d3aa-00632031f7-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.392 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:32:07.392 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:32:07.393 DEBUG:boto:Method: HEAD 2022-09-13T07:32:07.393 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:32:07.394 DEBUG:boto:Data: 2022-09-13T07:32:07.394 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.395 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:07.395 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.396 DEBUG:boto:Params: {} 2022-09-13T07:32:07.396 DEBUG:boto:Token: None 2022-09-13T07:32:07.397 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/ 2022-09-13T07:32:07.397 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AmFSDkAzC7DPn8CruFLtRPDLFcc= 2022-09-13T07:32:07.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AmFSDkAzC7DPn8CruFLtRPDLFcc=', 'Content-Length': '0'} 2022-09-13T07:32:07.401 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b4f2f40ca12fdfc-00632031f7-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.401 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.402 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.402 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.403 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.403 DEBUG:boto:Data: 2022-09-13T07:32:07.404 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.404 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:07.405 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.405 DEBUG:boto:Params: {} 2022-09-13T07:32:07.406 DEBUG:boto:Token: None 2022-09-13T07:32:07.407 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.407 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:sbNMsaJO9pU9WzFqu76qgiYeHYE= 2022-09-13T07:32:07.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:sbNMsaJO9pU9WzFqu76qgiYeHYE='} 2022-09-13T07:32:07.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-amz-request-id', 'tx0000011d2c73202dc7e62-00632031f7-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.428 DEBUG:rgw_multi.tests:version1 id=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:32:07.428 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.429 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.429 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.430 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.430 DEBUG:boto:Data: 2022-09-13T07:32:07.431 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.431 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:07.432 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.432 DEBUG:boto:Params: {} 2022-09-13T07:32:07.433 DEBUG:boto:Token: None 2022-09-13T07:32:07.433 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.434 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tU46an7pSR0vRiMDudb7l+yZxdw= 2022-09-13T07:32:07.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tU46an7pSR0vRiMDudb7l+yZxdw='} 2022-09-13T07:32:07.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'XsS27Z3dfioZRGA4B9WQVcgT8cs076i'), ('x-amz-request-id', 'tx00000b39f0c9fcf724baf-00632031f7-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.455 DEBUG:rgw_multi.tests:version2 id=XsS27Z3dfioZRGA4B9WQVcgT8cs076i 2022-09-13T07:32:07.455 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.456 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.456 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=XsS27Z3dfioZRGA4B9WQVcgT8cs076i 2022-09-13T07:32:07.457 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=XsS27Z3dfioZRGA4B9WQVcgT8cs076i 2022-09-13T07:32:07.457 DEBUG:boto:Method: DELETE 2022-09-13T07:32:07.458 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=XsS27Z3dfioZRGA4B9WQVcgT8cs076i 2022-09-13T07:32:07.458 DEBUG:boto:Data: 2022-09-13T07:32:07.459 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.459 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:07.459 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.460 DEBUG:boto:Params: {} 2022-09-13T07:32:07.460 DEBUG:boto:Token: None 2022-09-13T07:32:07.461 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/obj-test-zone1?versionId=XsS27Z3dfioZRGA4B9WQVcgT8cs076i 2022-09-13T07:32:07.461 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IRzmq6yt8hmUw9OJOw5UmsQM+WQ= 2022-09-13T07:32:07.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IRzmq6yt8hmUw9OJOw5UmsQM+WQ=', 'Content-Length': '0'} 2022-09-13T07:32:07.480 DEBUG:boto:Response headers: [('x-amz-version-id', 'XsS27Z3dfioZRGA4B9WQVcgT8cs076i'), ('x-amz-request-id', 'tx0000094d037e010ca2aef-00632031f7-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.481 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.482 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.483 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.484 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.485 DEBUG:boto:Data: 2022-09-13T07:32:07.486 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.487 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:07.488 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.489 DEBUG:boto:Params: {} 2022-09-13T07:32:07.490 DEBUG:boto:Token: None 2022-09-13T07:32:07.490 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.491 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BS/C18EkFdnC0YiW+m4ADrkQN6Y= 2022-09-13T07:32:07.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BS/C18EkFdnC0YiW+m4ADrkQN6Y='} 2022-09-13T07:32:07.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ZFax4VQ4TJeUVy.DBnNlFOxDEKT5S7L'), ('x-amz-request-id', 'tx00000ca077af6ba190e78-00632031f7-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.535 DEBUG:rgw_multi.tests:version3 id=ZFax4VQ4TJeUVy.DBnNlFOxDEKT5S7L 2022-09-13T07:32:07.536 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.536 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:32:07.537 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=ZFax4VQ4TJeUVy.DBnNlFOxDEKT5S7L 2022-09-13T07:32:07.537 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=ZFax4VQ4TJeUVy.DBnNlFOxDEKT5S7L 2022-09-13T07:32:07.538 DEBUG:boto:Method: DELETE 2022-09-13T07:32:07.538 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=ZFax4VQ4TJeUVy.DBnNlFOxDEKT5S7L 2022-09-13T07:32:07.539 DEBUG:boto:Data: 2022-09-13T07:32:07.539 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.540 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:32:07.540 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.541 DEBUG:boto:Params: {} 2022-09-13T07:32:07.541 DEBUG:boto:Token: None 2022-09-13T07:32:07.542 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/obj-test-zone1?versionId=ZFax4VQ4TJeUVy.DBnNlFOxDEKT5S7L 2022-09-13T07:32:07.543 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rEsHen+ie18m/himf0m9R8BWbLE= 2022-09-13T07:32:07.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rEsHen+ie18m/himf0m9R8BWbLE=', 'Content-Length': '0'} 2022-09-13T07:32:07.556 DEBUG:boto:Response headers: [('x-amz-version-id', 'ZFax4VQ4TJeUVy.DBnNlFOxDEKT5S7L'), ('x-amz-request-id', 'tx00000af16320af808d8f6-00632031f7-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.557 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:32:07.558 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:32:07.558 DEBUG:boto:Method: HEAD 2022-09-13T07:32:07.559 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:32:07.560 DEBUG:boto:Data: 2022-09-13T07:32:07.560 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.561 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:07.561 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.562 DEBUG:boto:Params: {} 2022-09-13T07:32:07.562 DEBUG:boto:Token: None 2022-09-13T07:32:07.563 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/ 2022-09-13T07:32:07.563 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AmFSDkAzC7DPn8CruFLtRPDLFcc= 2022-09-13T07:32:07.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AmFSDkAzC7DPn8CruFLtRPDLFcc=', 'Content-Length': '0'} 2022-09-13T07:32:07.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5eb4c99fe3ac0e6-00632031f7-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.567 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.568 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.568 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.569 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.570 DEBUG:boto:Data: 2022-09-13T07:32:07.570 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.571 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:07.571 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.571 DEBUG:boto:Params: {} 2022-09-13T07:32:07.572 DEBUG:boto:Token: None 2022-09-13T07:32:07.573 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.573 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gc8g3EoRzQArfA+jAKLK/fd6NE4= 2022-09-13T07:32:07.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gc8g3EoRzQArfA+jAKLK/fd6NE4='} 2022-09-13T07:32:07.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-amz-request-id', 'tx00000534766da1c46a508-00632031f7-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.587 DEBUG:rgw_multi.tests:version1 id=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:32:07.588 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.589 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.590 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.591 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.591 DEBUG:boto:Data: 2022-09-13T07:32:07.592 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.593 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:07.594 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.595 DEBUG:boto:Params: {} 2022-09-13T07:32:07.596 DEBUG:boto:Token: None 2022-09-13T07:32:07.597 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.598 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QF2lYDsLCaMUn5Cr06/nzsKakh0= 2022-09-13T07:32:07.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QF2lYDsLCaMUn5Cr06/nzsKakh0='} 2022-09-13T07:32:07.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2TpeU8a2LVr2DfAPyKK-iWGZesdk-1d'), ('x-amz-request-id', 'tx00000d2a7fdffdb8cd491-00632031f7-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.644 DEBUG:rgw_multi.tests:version2 id=2TpeU8a2LVr2DfAPyKK-iWGZesdk-1d 2022-09-13T07:32:07.644 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.645 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.646 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=2TpeU8a2LVr2DfAPyKK-iWGZesdk-1d 2022-09-13T07:32:07.647 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=2TpeU8a2LVr2DfAPyKK-iWGZesdk-1d 2022-09-13T07:32:07.648 DEBUG:boto:Method: DELETE 2022-09-13T07:32:07.649 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=2TpeU8a2LVr2DfAPyKK-iWGZesdk-1d 2022-09-13T07:32:07.650 DEBUG:boto:Data: 2022-09-13T07:32:07.651 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.651 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:07.652 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.653 DEBUG:boto:Params: {} 2022-09-13T07:32:07.654 DEBUG:boto:Token: None 2022-09-13T07:32:07.655 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/obj-test-zone2?versionId=2TpeU8a2LVr2DfAPyKK-iWGZesdk-1d 2022-09-13T07:32:07.656 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3Oogv20m1Lt/H99nUdGdcz/qnAo= 2022-09-13T07:32:07.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3Oogv20m1Lt/H99nUdGdcz/qnAo=', 'Content-Length': '0'} 2022-09-13T07:32:07.667 DEBUG:boto:Response headers: [('x-amz-version-id', '2TpeU8a2LVr2DfAPyKK-iWGZesdk-1d'), ('x-amz-request-id', 'tx000007895786fd7d7e44e-00632031f7-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.668 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.668 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.669 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.669 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.670 DEBUG:boto:Data: 2022-09-13T07:32:07.670 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.671 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:07.671 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.672 DEBUG:boto:Params: {} 2022-09-13T07:32:07.672 DEBUG:boto:Token: None 2022-09-13T07:32:07.673 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.674 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/5NOZjPqPlpF6D33XBhFFSj4aqo= 2022-09-13T07:32:07.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/5NOZjPqPlpF6D33XBhFFSj4aqo='} 2022-09-13T07:32:07.683 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'T17YD-AuQ3i66lBihuLeliKAC0Emmub'), ('x-amz-request-id', 'tx00000447ba48610ef6d93-00632031f7-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.684 DEBUG:rgw_multi.tests:version3 id=T17YD-AuQ3i66lBihuLeliKAC0Emmub 2022-09-13T07:32:07.685 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.685 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:32:07.686 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=T17YD-AuQ3i66lBihuLeliKAC0Emmub 2022-09-13T07:32:07.686 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=T17YD-AuQ3i66lBihuLeliKAC0Emmub 2022-09-13T07:32:07.686 DEBUG:boto:Method: DELETE 2022-09-13T07:32:07.687 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=T17YD-AuQ3i66lBihuLeliKAC0Emmub 2022-09-13T07:32:07.688 DEBUG:boto:Data: 2022-09-13T07:32:07.688 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.689 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:32:07.689 DEBUG:boto:Port: 8000 2022-09-13T07:32:07.689 DEBUG:boto:Params: {} 2022-09-13T07:32:07.690 DEBUG:boto:Token: None 2022-09-13T07:32:07.691 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/obj-test-zone2?versionId=T17YD-AuQ3i66lBihuLeliKAC0Emmub 2022-09-13T07:32:07.691 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+AybmPgMFgh1tMC8BNUKdvm/KqE= 2022-09-13T07:32:07.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+AybmPgMFgh1tMC8BNUKdvm/KqE=', 'Content-Length': '0'} 2022-09-13T07:32:07.701 DEBUG:boto:Response headers: [('x-amz-version-id', 'T17YD-AuQ3i66lBihuLeliKAC0Emmub'), ('x-amz-request-id', 'tx00000cd7a3fd9f4066645-00632031f7-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.703 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:32:07.703 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:32:07.704 DEBUG:boto:Method: HEAD 2022-09-13T07:32:07.704 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:32:07.705 DEBUG:boto:Data: 2022-09-13T07:32:07.705 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.706 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:07.706 DEBUG:boto:Port: 8001 2022-09-13T07:32:07.707 DEBUG:boto:Params: {} 2022-09-13T07:32:07.708 DEBUG:boto:Token: None 2022-09-13T07:32:07.708 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/ 2022-09-13T07:32:07.709 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AmFSDkAzC7DPn8CruFLtRPDLFcc= 2022-09-13T07:32:07.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AmFSDkAzC7DPn8CruFLtRPDLFcc=', 'Content-Length': '0'} 2022-09-13T07:32:07.712 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f2075351b5eb0bb-00632031f7-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.712 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.713 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.713 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.714 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.714 DEBUG:boto:Data: 2022-09-13T07:32:07.715 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.715 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:07.716 DEBUG:boto:Port: 8001 2022-09-13T07:32:07.716 DEBUG:boto:Params: {} 2022-09-13T07:32:07.717 DEBUG:boto:Token: None 2022-09-13T07:32:07.717 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.718 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QvSLW9jnJ/KkIjEMLlALOBCveGE= 2022-09-13T07:32:07.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QvSLW9jnJ/KkIjEMLlALOBCveGE='} 2022-09-13T07:32:07.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-amz-request-id', 'tx000005d9aeb125513d175-00632031f7-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.734 DEBUG:rgw_multi.tests:version1 id=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:32:07.735 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.735 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.736 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.736 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.737 DEBUG:boto:Data: 2022-09-13T07:32:07.737 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.738 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:07.739 DEBUG:boto:Port: 8001 2022-09-13T07:32:07.739 DEBUG:boto:Params: {} 2022-09-13T07:32:07.740 DEBUG:boto:Token: None 2022-09-13T07:32:07.740 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.741 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gIZ2eNeS87tgFfJNekN6Y6CSg30= 2022-09-13T07:32:07.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gIZ2eNeS87tgFfJNekN6Y6CSg30='} 2022-09-13T07:32:07.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'nn95jjYsS.aidVmLEEM621drMynpn9o'), ('x-amz-request-id', 'tx00000460f86abb267fe88-00632031f7-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.756 DEBUG:rgw_multi.tests:version2 id=nn95jjYsS.aidVmLEEM621drMynpn9o 2022-09-13T07:32:07.757 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.757 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.758 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=nn95jjYsS.aidVmLEEM621drMynpn9o 2022-09-13T07:32:07.758 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=nn95jjYsS.aidVmLEEM621drMynpn9o 2022-09-13T07:32:07.759 DEBUG:boto:Method: DELETE 2022-09-13T07:32:07.759 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=nn95jjYsS.aidVmLEEM621drMynpn9o 2022-09-13T07:32:07.760 DEBUG:boto:Data: 2022-09-13T07:32:07.760 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.760 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:07.761 DEBUG:boto:Port: 8001 2022-09-13T07:32:07.761 DEBUG:boto:Params: {} 2022-09-13T07:32:07.762 DEBUG:boto:Token: None 2022-09-13T07:32:07.763 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/obj-test-zone3?versionId=nn95jjYsS.aidVmLEEM621drMynpn9o 2022-09-13T07:32:07.763 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pqVC5wplVgQGC4ORhhAC52F0T9E= 2022-09-13T07:32:07.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pqVC5wplVgQGC4ORhhAC52F0T9E=', 'Content-Length': '0'} 2022-09-13T07:32:07.774 DEBUG:boto:Response headers: [('x-amz-version-id', 'nn95jjYsS.aidVmLEEM621drMynpn9o'), ('x-amz-request-id', 'tx0000072321d70a78ed165-00632031f7-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.774 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.775 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.775 DEBUG:boto:Method: PUT 2022-09-13T07:32:07.776 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.776 DEBUG:boto:Data: 2022-09-13T07:32:07.777 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:32:07.777 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:07.778 DEBUG:boto:Port: 8001 2022-09-13T07:32:07.778 DEBUG:boto:Params: {} 2022-09-13T07:32:07.779 DEBUG:boto:Token: None 2022-09-13T07:32:07.779 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.780 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qrhwAHeKP5Rfuwuwt6StNSWIO5c= 2022-09-13T07:32:07.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qrhwAHeKP5Rfuwuwt6StNSWIO5c='} 2022-09-13T07:32:07.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ebFOd1PIrY4Vr-n9FATOo7mNjgtddpX'), ('x-amz-request-id', 'tx000004d33f533e77fd2d1-00632031f7-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.827 DEBUG:rgw_multi.tests:version3 id=ebFOd1PIrY4Vr-n9FATOo7mNjgtddpX 2022-09-13T07:32:07.827 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.828 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:32:07.828 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=ebFOd1PIrY4Vr-n9FATOo7mNjgtddpX 2022-09-13T07:32:07.829 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=ebFOd1PIrY4Vr-n9FATOo7mNjgtddpX 2022-09-13T07:32:07.829 DEBUG:boto:Method: DELETE 2022-09-13T07:32:07.830 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=ebFOd1PIrY4Vr-n9FATOo7mNjgtddpX 2022-09-13T07:32:07.830 DEBUG:boto:Data: 2022-09-13T07:32:07.831 DEBUG:boto:Headers: {} 2022-09-13T07:32:07.831 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:32:07.832 DEBUG:boto:Port: 8001 2022-09-13T07:32:07.832 DEBUG:boto:Params: {} 2022-09-13T07:32:07.833 DEBUG:boto:Token: None 2022-09-13T07:32:07.833 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:32:07 GMT /yxvvuh-18/obj-test-zone3?versionId=ebFOd1PIrY4Vr-n9FATOo7mNjgtddpX 2022-09-13T07:32:07.834 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WPvLkS8VJm93fMoxwpWxWKFiodk= 2022-09-13T07:32:07.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:32:07 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WPvLkS8VJm93fMoxwpWxWKFiodk=', 'Content-Length': '0'} 2022-09-13T07:32:07.844 DEBUG:boto:Response headers: [('x-amz-version-id', 'ebFOd1PIrY4Vr-n9FATOo7mNjgtddpX'), ('x-amz-request-id', 'tx00000200c587614d69847-00632031f7-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:32:07.845 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-16 2022-09-13T07:32:07.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:32:07.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:32:07.847 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:32:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:32:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:32:07.943 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:32:07.938+0000 7f6f05a1f600 1 waiting for incremental sync to catch up: 2022-09-13T07:32:07.943 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000010.31.5, , , , 00000000003.15.9, , , , , ] 2022-09-13T07:32:07.944 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000010.31.5, , , , 00000000004.21.9, , , , , ] 2022-09-13T07:32:12.946 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:32:12.940+0000 7f6f05a1f600 1 waiting for incremental sync to catch up: 2022-09-13T07:32:12.946 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000010.31.5, , , , 00000000003.15.9, , , , , ] 2022-09-13T07:32:12.947 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000010.31.5, , , , 00000000004.21.9, , , , , ] 2022-09-13T07:32:17.945 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:32:17.942+0000 7f6f05a1f600 1 waiting for incremental sync to catch up: 2022-09-13T07:32:17.946 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000010.31.5, , , , 00000000003.15.9, , , , , ] 2022-09-13T07:32:17.946 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000010.31.5, , , , 00000000004.21.9, , , , , ] 2022-09-13T07:32:22.947 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:32:22.944+0000 7f6f05a1f600 1 waiting for incremental sync to catch up: 2022-09-13T07:32:22.948 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000010.31.5, , , , 00000000003.15.9, , , , , ] 2022-09-13T07:32:22.948 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000010.31.5, , , , 00000000004.21.9, , , , , ] 2022-09-13T07:32:27.948 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:32:27.944+0000 7f6f05a1f600 1 bucket sync caught up with source: 2022-09-13T07:32:27.948 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000010.31.5, , 00000000002.10.10, , 00000000004.21.9, , , , , ] 2022-09-13T07:32:27.949 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000010.31.5, , , , 00000000004.21.9, , , , , ] 2022-09-13T07:32:27.949 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:32:27.944+0000 7f6f05a1f600 0 bucket checkpoint complete 2022-09-13T07:32:27.971 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-16 2022-09-13T07:32:27.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:32:27.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:32:27.973 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:32:28.042 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:32:28.042 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:32:28.092 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:32:28.088+0000 7f969775f600 1 waiting for incremental sync to catch up: 2022-09-13T07:32:28.093 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000001.14.5, , , , 00000000003.15.9, , , , , ] 2022-09-13T07:32:28.093 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000010.31.5, , 00000000002.10.10, , 00000000004.21.9, , , , , ] 2022-09-13T07:32:33.094 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:32:33.090+0000 7f969775f600 1 bucket sync caught up with source: 2022-09-13T07:32:33.094 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000010.31.5, , 00000000002.10.10, , 00000000004.21.9, , , , , ] 2022-09-13T07:32:33.095 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000010.31.5, , 00000000002.10.10, , 00000000004.21.9, , , , , ] 2022-09-13T07:32:33.095 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:32:33.090+0000 7f969775f600 0 bucket checkpoint complete 2022-09-13T07:32:33.110 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-16 2022-09-13T07:32:33.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:32:33.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:32:33.112 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:32:33.177 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:32:33.177 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:32:33.215 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:32:33.212+0000 7f0f476b3600 1 waiting for incremental sync to catch up: 2022-09-13T07:32:33.216 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000002.49.10, , 00000000001.18.5, , 00000000003.28.9, , , , , ] 2022-09-13T07:32:33.216 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.49.10, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:32:38.217 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:32:38.214+0000 7f0f476b3600 1 waiting for incremental sync to catch up: 2022-09-13T07:32:38.218 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000002.49.10, , 00000000001.18.5, , 00000000003.28.9, , , , , ] 2022-09-13T07:32:38.218 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.49.10, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:32:43.218 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:32:43.216+0000 7f0f476b3600 1 waiting for incremental sync to catch up: 2022-09-13T07:32:43.219 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000002.49.10, , 00000000001.18.5, , 00000000003.28.9, , , , , ] 2022-09-13T07:32:43.219 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.49.10, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:32:48.220 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:32:48.217+0000 7f0f476b3600 1 bucket sync caught up with source: 2022-09-13T07:32:48.221 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000002.49.10, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:32:48.221 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000002.49.10, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:32:48.221 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:32:48.217+0000 7f0f476b3600 0 bucket checkpoint complete 2022-09-13T07:32:48.237 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-16 2022-09-13T07:32:48.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:32:48.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:32:48.239 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:32:48.303 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:32:48.303 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:32:48.350 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:32:48.348+0000 7f3a14faf600 1 waiting for incremental sync to catch up: 2022-09-13T07:32:48.351 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000002.49.10, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:32:48.351 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.51.9, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:32:53.352 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:32:53.349+0000 7f3a14faf600 1 waiting for incremental sync to catch up: 2022-09-13T07:32:53.353 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000002.49.10, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:32:53.353 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.51.9, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:32:58.354 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:32:58.350+0000 7f3a14faf600 1 waiting for incremental sync to catch up: 2022-09-13T07:32:58.355 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000002.49.10, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:32:58.355 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.51.9, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:33:03.355 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:33:03.351+0000 7f3a14faf600 1 waiting for incremental sync to catch up: 2022-09-13T07:33:03.356 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000002.49.10, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:33:03.356 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.51.9, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:33:08.357 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:33:08.353+0000 7f3a14faf600 1 waiting for incremental sync to catch up: 2022-09-13T07:33:08.357 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000002.49.10, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:33:08.357 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.51.9, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:33:13.358 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:33:13.354+0000 7f3a14faf600 1 bucket sync caught up with source: 2022-09-13T07:33:13.359 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.51.9, , 00000000010.35.5, , 00000000007.50.9, , , , , ] 2022-09-13T07:33:13.359 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.51.9, , 00000000010.35.5, , 00000000004.36.9, , , , , ] 2022-09-13T07:33:13.359 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:33:13.354+0000 7f3a14faf600 0 bucket checkpoint complete 2022-09-13T07:33:13.381 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-16 2022-09-13T07:33:13.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:33:13.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:33:13.383 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:33:13.449 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:33:13.449 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:33:13.501 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:33:13.497+0000 7fd66a427600 1 waiting for incremental sync to catch up: 2022-09-13T07:33:13.502 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.55.9, , 00000000002.17.10, , 00000000014.44.5, , , , , ] 2022-09-13T07:33:13.502 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.55.9, , 00000000003.19.9, , 00000000014.44.5, , , , , ] 2022-09-13T07:33:18.503 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:33:18.499+0000 7fd66a427600 1 waiting for incremental sync to catch up: 2022-09-13T07:33:18.504 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.55.9, , 00000000002.17.10, , 00000000014.44.5, , , , , ] 2022-09-13T07:33:18.504 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.55.9, , 00000000003.19.9, , 00000000014.44.5, , , , , ] 2022-09-13T07:33:23.504 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:33:23.500+0000 7fd66a427600 1 waiting for incremental sync to catch up: 2022-09-13T07:33:23.505 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.55.9, , 00000000002.17.10, , 00000000014.44.5, , , , , ] 2022-09-13T07:33:23.505 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.55.9, , 00000000003.19.9, , 00000000014.44.5, , , , , ] 2022-09-13T07:33:28.506 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:33:28.502+0000 7fd66a427600 1 bucket sync caught up with source: 2022-09-13T07:33:28.507 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.55.9, , 00000000003.19.9, , 00000000014.44.5, , , , , ] 2022-09-13T07:33:28.507 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.55.9, , 00000000003.19.9, , 00000000014.44.5, , , , , ] 2022-09-13T07:33:28.507 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:33:28.502+0000 7fd66a427600 0 bucket checkpoint complete 2022-09-13T07:33:28.528 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-16 2022-09-13T07:33:28.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:33:28.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:33:28.530 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:33:28.593 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:33:28.594 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:33:28.647 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:33:28.643+0000 7fb5c6f35600 1 waiting for incremental sync to catch up: 2022-09-13T07:33:28.648 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000003.55.9, , 00000000003.19.9, , 00000000008.32.11, , , , , ] 2022-09-13T07:33:28.648 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000003.55.9, , 00000000003.19.9, , 00000000014.44.5, , , , , ] 2022-09-13T07:33:33.649 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:33:33.646+0000 7fb5c6f35600 1 waiting for incremental sync to catch up: 2022-09-13T07:33:33.649 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000003.55.9, , 00000000003.19.9, , 00000000008.32.11, , , , , ] 2022-09-13T07:33:33.649 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000003.55.9, , 00000000003.19.9, , 00000000014.44.5, , , , , ] 2022-09-13T07:33:38.650 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:33:38.647+0000 7fb5c6f35600 1 waiting for incremental sync to catch up: 2022-09-13T07:33:38.651 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000003.55.9, , 00000000003.19.9, , 00000000008.32.11, , , , , ] 2022-09-13T07:33:38.652 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000003.55.9, , 00000000003.19.9, , 00000000014.44.5, , , , , ] 2022-09-13T07:33:43.652 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:33:43.648+0000 7fb5c6f35600 1 waiting for incremental sync to catch up: 2022-09-13T07:33:43.652 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000003.55.9, , 00000000003.19.9, , 00000000008.32.11, , , , , ] 2022-09-13T07:33:43.653 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000003.55.9, , 00000000003.19.9, , 00000000014.44.5, , , , , ] 2022-09-13T07:33:48.653 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:33:48.649+0000 7fb5c6f35600 1 waiting for incremental sync to catch up: 2022-09-13T07:33:48.653 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000003.55.9, , 00000000003.19.9, , 00000000008.32.11, , , , , ] 2022-09-13T07:33:48.654 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000003.55.9, , 00000000003.19.9, , 00000000014.44.5, , , , , ] 2022-09-13T07:33:53.655 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:33:53.650+0000 7fb5c6f35600 1 waiting for incremental sync to catch up: 2022-09-13T07:33:53.655 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000003.55.9, , 00000000003.19.9, , 00000000008.32.11, , , , , ] 2022-09-13T07:33:53.655 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000003.55.9, , 00000000003.19.9, , 00000000014.44.5, , , , , ] 2022-09-13T07:33:58.656 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:33:58.652+0000 7fb5c6f35600 1 waiting for incremental sync to catch up: 2022-09-13T07:33:58.656 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000003.55.9, , 00000000003.19.9, , 00000000008.32.11, , , , , ] 2022-09-13T07:33:58.657 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000003.55.9, , 00000000003.19.9, , 00000000014.44.5, , , , , ] 2022-09-13T07:34:03.657 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:34:03.653+0000 7fb5c6f35600 1 waiting for incremental sync to catch up: 2022-09-13T07:34:03.658 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000003.55.9, , 00000000003.19.9, , 00000000008.32.11, , , , , ] 2022-09-13T07:34:03.659 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000003.55.9, , 00000000003.19.9, , 00000000014.44.5, , , , , ] 2022-09-13T07:34:08.659 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:34:08.654+0000 7fb5c6f35600 1 bucket sync caught up with source: 2022-09-13T07:34:08.659 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000003.55.9, , 00000000003.19.9, , 00000000014.44.5, , , , , ] 2022-09-13T07:34:08.660 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000003.55.9, , 00000000003.19.9, , 00000000014.44.5, , , , , ] 2022-09-13T07:34:08.660 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:34:08.654+0000 7fb5c6f35600 0 bucket checkpoint complete 2022-09-13T07:34:08.680 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-16 zones={test-zone2, test-zone1} 2022-09-13T07:34:08.681 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:08.681 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:08.682 DEBUG:boto:Method: HEAD 2022-09-13T07:34:08.682 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:34:08.683 DEBUG:boto:Data: 2022-09-13T07:34:08.683 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.684 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:08.684 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.685 DEBUG:boto:Params: {} 2022-09-13T07:34:08.686 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T07:34:08.687 DEBUG:boto:Token: None 2022-09-13T07:34:08.687 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/ 2022-09-13T07:34:08.688 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:esFP+lsYSDkf1E3wpAxwmqQttB8= 2022-09-13T07:34:08.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:esFP+lsYSDkf1E3wpAxwmqQttB8=', 'Content-Length': '0'} 2022-09-13T07:34:08.693 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067ca7f071f8affba-0063203270-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.694 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:08.694 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:08.695 DEBUG:boto:Method: HEAD 2022-09-13T07:34:08.695 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:34:08.696 DEBUG:boto:Data: 2022-09-13T07:34:08.696 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.697 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:08.697 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.698 DEBUG:boto:Params: {} 2022-09-13T07:34:08.698 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T07:34:08.699 DEBUG:boto:Token: None 2022-09-13T07:34:08.700 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/ 2022-09-13T07:34:08.700 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:esFP+lsYSDkf1E3wpAxwmqQttB8= 2022-09-13T07:34:08.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:esFP+lsYSDkf1E3wpAxwmqQttB8=', 'Content-Length': '0'} 2022-09-13T07:34:08.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005bda2bd2b17f08ce-0063203270-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.706 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:34:08.707 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:08.708 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:08.709 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:08.709 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:08.710 DEBUG:boto:Method: GET 2022-09-13T07:34:08.711 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:08.712 DEBUG:boto:Data: 2022-09-13T07:34:08.713 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.714 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:08.715 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.716 DEBUG:boto:Params: {} 2022-09-13T07:34:08.717 DEBUG:boto:Token: None 2022-09-13T07:34:08.718 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/?versions 2022-09-13T07:34:08.719 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iXj5ZoZLULIgsH9voEWKe0+iRNs= 2022-09-13T07:34:08.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iXj5ZoZLULIgsH9voEWKe0+iRNs=', 'Content-Length': '0'} 2022-09-13T07:34:08.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024cf97515e70d977-0063203270-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.724 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:08.727 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:34:08.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:34:08.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:34:08.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:34:08.729 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:34:08.730 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:08.730 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:08.731 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:08.731 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:08.732 DEBUG:boto:Method: GET 2022-09-13T07:34:08.732 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:08.733 DEBUG:boto:Data: 2022-09-13T07:34:08.733 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.734 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:08.734 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.735 DEBUG:boto:Params: {} 2022-09-13T07:34:08.736 DEBUG:boto:Token: None 2022-09-13T07:34:08.736 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/?versions 2022-09-13T07:34:08.737 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iXj5ZoZLULIgsH9voEWKe0+iRNs= 2022-09-13T07:34:08.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iXj5ZoZLULIgsH9voEWKe0+iRNs=', 'Content-Length': '0'} 2022-09-13T07:34:08.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afd8211eed6f3185-0063203270-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.741 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:08.742 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:34:08.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:34:08.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:34:08.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:34:08.744 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:08.745 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:08.745 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:08.746 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:08.746 DEBUG:boto:Method: GET 2022-09-13T07:34:08.747 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:08.748 DEBUG:boto:Data: 2022-09-13T07:34:08.748 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.749 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:08.749 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.749 DEBUG:boto:Params: {} 2022-09-13T07:34:08.750 DEBUG:boto:Token: None 2022-09-13T07:34:08.751 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/?versions 2022-09-13T07:34:08.751 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iXj5ZoZLULIgsH9voEWKe0+iRNs= 2022-09-13T07:34:08.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iXj5ZoZLULIgsH9voEWKe0+iRNs=', 'Content-Length': '0'} 2022-09-13T07:34:08.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000317295f56ddcaf7b-0063203270-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.755 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:08.756 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:08.757 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:08.757 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:08.758 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:08.758 DEBUG:boto:Method: GET 2022-09-13T07:34:08.759 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:08.759 DEBUG:boto:Data: 2022-09-13T07:34:08.760 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.760 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:08.761 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.761 DEBUG:boto:Params: {} 2022-09-13T07:34:08.762 DEBUG:boto:Token: None 2022-09-13T07:34:08.763 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/?versions 2022-09-13T07:34:08.763 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iXj5ZoZLULIgsH9voEWKe0+iRNs= 2022-09-13T07:34:08.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iXj5ZoZLULIgsH9voEWKe0+iRNs=', 'Content-Length': '0'} 2022-09-13T07:34:08.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000124f9de508d027a0-0063203270-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.767 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:08.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:08.770 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:08.771 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:08.772 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.772 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.773 DEBUG:boto:Method: GET 2022-09-13T07:34:08.774 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.775 DEBUG:boto:Data: 2022-09-13T07:34:08.776 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.777 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:08.778 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.779 DEBUG:boto:Params: {} 2022-09-13T07:34:08.780 DEBUG:boto:Token: None 2022-09-13T07:34:08.781 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.782 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HxSIUnu4EybAQOT0g2NzROPEwKg= 2022-09-13T07:34:08.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HxSIUnu4EybAQOT0g2NzROPEwKg=', 'Content-Length': '0'} 2022-09-13T07:34:08.785 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a54df1062ae0747b-0063203270-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.787 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:08.788 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:08.789 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.789 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.790 DEBUG:boto:Method: GET 2022-09-13T07:34:08.791 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.792 DEBUG:boto:Data: 2022-09-13T07:34:08.793 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.794 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:08.795 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.796 DEBUG:boto:Params: {} 2022-09-13T07:34:08.797 DEBUG:boto:Token: None 2022-09-13T07:34:08.797 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.798 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HxSIUnu4EybAQOT0g2NzROPEwKg= 2022-09-13T07:34:08.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HxSIUnu4EybAQOT0g2NzROPEwKg=', 'Content-Length': '0'} 2022-09-13T07:34:08.801 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000319f75aaf14f3aa0-0063203270-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.802 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:08.803 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:08.804 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.804 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.805 DEBUG:boto:Method: HEAD 2022-09-13T07:34:08.805 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.806 DEBUG:boto:Data: 2022-09-13T07:34:08.806 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.807 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:08.807 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.808 DEBUG:boto:Params: {} 2022-09-13T07:34:08.809 DEBUG:boto:Token: None 2022-09-13T07:34:08.809 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.810 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LlPqqlYqfyK3NEh4Jrqb8BrjFQY= 2022-09-13T07:34:08.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LlPqqlYqfyK3NEh4Jrqb8BrjFQY=', 'Content-Length': '0'} 2022-09-13T07:34:08.813 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f2f3ca82fb32a70-0063203270-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.814 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:08.815 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:08.815 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.816 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.817 DEBUG:boto:Method: HEAD 2022-09-13T07:34:08.818 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.818 DEBUG:boto:Data: 2022-09-13T07:34:08.819 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.819 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:08.820 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.820 DEBUG:boto:Params: {} 2022-09-13T07:34:08.821 DEBUG:boto:Token: None 2022-09-13T07:34:08.822 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.822 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LlPqqlYqfyK3NEh4Jrqb8BrjFQY= 2022-09-13T07:34:08.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LlPqqlYqfyK3NEh4Jrqb8BrjFQY=', 'Content-Length': '0'} 2022-09-13T07:34:08.825 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000efe76cf645ab6af8-0063203270-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:08.826 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:08.827 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:08.827 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.828 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.828 DEBUG:boto:Method: GET 2022-09-13T07:34:08.829 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.829 DEBUG:boto:Data: 2022-09-13T07:34:08.830 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.830 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:08.831 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.831 DEBUG:boto:Params: {} 2022-09-13T07:34:08.832 DEBUG:boto:Token: None 2022-09-13T07:34:08.833 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.833 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HxSIUnu4EybAQOT0g2NzROPEwKg= 2022-09-13T07:34:08.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HxSIUnu4EybAQOT0g2NzROPEwKg=', 'Content-Length': '0'} 2022-09-13T07:34:08.836 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000071d541fa8869feea-0063203270-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.837 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:08.837 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:08.838 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.838 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.839 DEBUG:boto:Method: GET 2022-09-13T07:34:08.839 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.840 DEBUG:boto:Data: 2022-09-13T07:34:08.840 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.841 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:08.841 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.842 DEBUG:boto:Params: {} 2022-09-13T07:34:08.842 DEBUG:boto:Token: None 2022-09-13T07:34:08.843 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.844 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HxSIUnu4EybAQOT0g2NzROPEwKg= 2022-09-13T07:34:08.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HxSIUnu4EybAQOT0g2NzROPEwKg=', 'Content-Length': '0'} 2022-09-13T07:34:08.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000087b93de186389e71-0063203270-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.847 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:08.848 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:08.848 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:08.849 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:08.850 DEBUG:boto:Method: GET 2022-09-13T07:34:08.850 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:34:08.851 DEBUG:boto:Data: 2022-09-13T07:34:08.851 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.852 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:08.852 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.853 DEBUG:boto:Params: {} 2022-09-13T07:34:08.853 DEBUG:boto:Token: None 2022-09-13T07:34:08.854 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/ 2022-09-13T07:34:08.855 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zzFEqyWkviQidNwUCh6BOUMRYWs= 2022-09-13T07:34:08.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zzFEqyWkviQidNwUCh6BOUMRYWs=', 'Content-Length': '0'} 2022-09-13T07:34:08.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080d9d22c3585946d-0063203270-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.859 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:08.860 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:08.861 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:08.861 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:08.862 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:08.863 DEBUG:boto:Method: GET 2022-09-13T07:34:08.863 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:34:08.864 DEBUG:boto:Data: 2022-09-13T07:34:08.864 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.865 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:08.865 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.866 DEBUG:boto:Params: {} 2022-09-13T07:34:08.866 DEBUG:boto:Token: None 2022-09-13T07:34:08.867 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/ 2022-09-13T07:34:08.867 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zzFEqyWkviQidNwUCh6BOUMRYWs= 2022-09-13T07:34:08.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zzFEqyWkviQidNwUCh6BOUMRYWs=', 'Content-Length': '0'} 2022-09-13T07:34:08.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5b81d651e45e55e-0063203270-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.871 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:08.872 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:08.873 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:08.873 DEBUG:boto:Method: HEAD 2022-09-13T07:34:08.874 DEBUG:boto:Path: /yxvvuh-16/dummy 2022-09-13T07:34:08.874 DEBUG:boto:Data: 2022-09-13T07:34:08.875 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.875 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:08.876 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.876 DEBUG:boto:Params: {} 2022-09-13T07:34:08.877 DEBUG:boto:Token: None 2022-09-13T07:34:08.878 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/dummy 2022-09-13T07:34:08.878 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:w1a131qsfsFXq62Xss0HbTy4B1Q= 2022-09-13T07:34:08.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:w1a131qsfsFXq62Xss0HbTy4B1Q=', 'Content-Length': '0'} 2022-09-13T07:34:08.881 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000228044291bf55589-0063203270-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.882 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:08.882 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:08.883 DEBUG:boto:Method: HEAD 2022-09-13T07:34:08.883 DEBUG:boto:Path: /yxvvuh-16/dummy 2022-09-13T07:34:08.884 DEBUG:boto:Data: 2022-09-13T07:34:08.884 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.885 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:08.885 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.886 DEBUG:boto:Params: {} 2022-09-13T07:34:08.886 DEBUG:boto:Token: None 2022-09-13T07:34:08.887 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/dummy 2022-09-13T07:34:08.887 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:w1a131qsfsFXq62Xss0HbTy4B1Q= 2022-09-13T07:34:08.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:w1a131qsfsFXq62Xss0HbTy4B1Q=', 'Content-Length': '0'} 2022-09-13T07:34:08.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000201bc5dbfe47282d-0063203270-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:08.892 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:08.893 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:08.893 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.894 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.894 DEBUG:boto:Method: GET 2022-09-13T07:34:08.895 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.895 DEBUG:boto:Data: 2022-09-13T07:34:08.896 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.896 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:08.897 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.898 DEBUG:boto:Params: {} 2022-09-13T07:34:08.898 DEBUG:boto:Token: None 2022-09-13T07:34:08.899 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.899 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HxSIUnu4EybAQOT0g2NzROPEwKg= 2022-09-13T07:34:08.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HxSIUnu4EybAQOT0g2NzROPEwKg=', 'Content-Length': '0'} 2022-09-13T07:34:08.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b5bf8d518eb255ac-0063203270-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.903 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:08.904 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:08.904 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.905 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.905 DEBUG:boto:Method: GET 2022-09-13T07:34:08.906 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.906 DEBUG:boto:Data: 2022-09-13T07:34:08.907 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.907 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:08.908 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.908 DEBUG:boto:Params: {} 2022-09-13T07:34:08.909 DEBUG:boto:Token: None 2022-09-13T07:34:08.909 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:08.910 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HxSIUnu4EybAQOT0g2NzROPEwKg= 2022-09-13T07:34:08.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HxSIUnu4EybAQOT0g2NzROPEwKg=', 'Content-Length': '0'} 2022-09-13T07:34:08.913 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c9838b3810474560-0063203270-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.914 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:08.915 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:08.915 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:08.916 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:08.916 DEBUG:boto:Method: GET 2022-09-13T07:34:08.917 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:34:08.917 DEBUG:boto:Data: 2022-09-13T07:34:08.918 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.919 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:08.919 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.920 DEBUG:boto:Params: {} 2022-09-13T07:34:08.920 DEBUG:boto:Token: None 2022-09-13T07:34:08.921 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/ 2022-09-13T07:34:08.921 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zzFEqyWkviQidNwUCh6BOUMRYWs= 2022-09-13T07:34:08.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zzFEqyWkviQidNwUCh6BOUMRYWs=', 'Content-Length': '0'} 2022-09-13T07:34:08.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8277825ede6f711-0063203270-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.966 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:08.967 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:08.968 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:08.968 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:08.969 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:08.969 DEBUG:boto:Method: GET 2022-09-13T07:34:08.970 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:34:08.971 DEBUG:boto:Data: 2022-09-13T07:34:08.971 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.972 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:08.972 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.973 DEBUG:boto:Params: {} 2022-09-13T07:34:08.973 DEBUG:boto:Token: None 2022-09-13T07:34:08.974 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/ 2022-09-13T07:34:08.975 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zzFEqyWkviQidNwUCh6BOUMRYWs= 2022-09-13T07:34:08.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zzFEqyWkviQidNwUCh6BOUMRYWs=', 'Content-Length': '0'} 2022-09-13T07:34:08.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b79cf3f1e436237b-0063203270-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.979 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:08.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:08.980 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:08.981 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:08.981 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:08.982 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:08.982 DEBUG:boto:Method: GET 2022-09-13T07:34:08.983 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:08.983 DEBUG:boto:Data: 2022-09-13T07:34:08.984 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.985 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:08.985 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.986 DEBUG:boto:Params: {} 2022-09-13T07:34:08.986 DEBUG:boto:Token: None 2022-09-13T07:34:08.987 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:08.988 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3zkyDVCmS6VcCj0C2Cx8BoSuiOo= 2022-09-13T07:34:08.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3zkyDVCmS6VcCj0C2Cx8BoSuiOo=', 'Content-Length': '0'} 2022-09-13T07:34:08.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abd1fb6c9e5df04b-0063203270-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:08.992 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:08.992 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:08.993 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:08.993 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:08.994 DEBUG:boto:Method: GET 2022-09-13T07:34:08.994 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:08.995 DEBUG:boto:Data: 2022-09-13T07:34:08.995 DEBUG:boto:Headers: {} 2022-09-13T07:34:08.996 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:08.996 DEBUG:boto:Port: 8000 2022-09-13T07:34:08.997 DEBUG:boto:Params: {} 2022-09-13T07:34:08.998 DEBUG:boto:Token: None 2022-09-13T07:34:08.999 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:08 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.000 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3zkyDVCmS6VcCj0C2Cx8BoSuiOo= 2022-09-13T07:34:09.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3zkyDVCmS6VcCj0C2Cx8BoSuiOo=', 'Content-Length': '0'} 2022-09-13T07:34:09.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023bf032893d54b2b-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.004 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.005 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.005 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.006 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.007 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.007 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.008 DEBUG:boto:Data: 2022-09-13T07:34:09.008 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.009 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.009 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.010 DEBUG:boto:Params: {} 2022-09-13T07:34:09.010 DEBUG:boto:Token: None 2022-09-13T07:34:09.011 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.011 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qZC2qV5nCvaL8z8MgEs1iF8XiLI= 2022-09-13T07:34:09.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qZC2qV5nCvaL8z8MgEs1iF8XiLI=', 'Content-Length': '0'} 2022-09-13T07:34:09.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de3c859235dc084b-0063203271-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.015 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.016 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.016 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.017 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.017 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.018 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.018 DEBUG:boto:Data: 2022-09-13T07:34:09.019 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.019 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.020 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.020 DEBUG:boto:Params: {} 2022-09-13T07:34:09.021 DEBUG:boto:Token: None 2022-09-13T07:34:09.022 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.022 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qZC2qV5nCvaL8z8MgEs1iF8XiLI= 2022-09-13T07:34:09.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qZC2qV5nCvaL8z8MgEs1iF8XiLI=', 'Content-Length': '0'} 2022-09-13T07:34:09.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e87748023c6d634-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:09.026 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.027 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.027 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.028 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.028 DEBUG:boto:Method: GET 2022-09-13T07:34:09.029 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.029 DEBUG:boto:Data: 2022-09-13T07:34:09.030 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.030 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.031 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.031 DEBUG:boto:Params: {} 2022-09-13T07:34:09.032 DEBUG:boto:Token: None 2022-09-13T07:34:09.033 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.033 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6TyuM36fr6i3oBiCBsJg3npyEkY= 2022-09-13T07:34:09.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6TyuM36fr6i3oBiCBsJg3npyEkY=', 'Content-Length': '0'} 2022-09-13T07:34:09.036 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d2a249a5ee9f350-0063203271-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.078 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.079 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.080 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.080 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.081 DEBUG:boto:Method: GET 2022-09-13T07:34:09.081 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.082 DEBUG:boto:Data: 2022-09-13T07:34:09.083 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.083 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.084 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.085 DEBUG:boto:Params: {} 2022-09-13T07:34:09.086 DEBUG:boto:Token: None 2022-09-13T07:34:09.087 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.088 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6TyuM36fr6i3oBiCBsJg3npyEkY= 2022-09-13T07:34:09.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6TyuM36fr6i3oBiCBsJg3npyEkY=', 'Content-Length': '0'} 2022-09-13T07:34:09.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002af8582000e2889f-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.092 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.093 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.093 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:09.094 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:09.094 DEBUG:boto:Method: GET 2022-09-13T07:34:09.095 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:09.095 DEBUG:boto:Data: 2022-09-13T07:34:09.096 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.096 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.097 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.097 DEBUG:boto:Params: {} 2022-09-13T07:34:09.098 DEBUG:boto:Token: None 2022-09-13T07:34:09.098 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.099 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo= 2022-09-13T07:34:09.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo=', 'Content-Length': '0'} 2022-09-13T07:34:09.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfa1e3dcdc695a59-0063203271-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.103 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.103 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.104 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.104 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:09.105 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:09.106 DEBUG:boto:Method: GET 2022-09-13T07:34:09.106 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:09.107 DEBUG:boto:Data: 2022-09-13T07:34:09.107 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.108 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.108 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.109 DEBUG:boto:Params: {} 2022-09-13T07:34:09.109 DEBUG:boto:Token: None 2022-09-13T07:34:09.110 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.111 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo= 2022-09-13T07:34:09.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo=', 'Content-Length': '0'} 2022-09-13T07:34:09.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2680a614f66af77-0063203271-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.154 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.155 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.156 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.156 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.157 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.157 DEBUG:boto:Data: 2022-09-13T07:34:09.158 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.158 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.159 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.159 DEBUG:boto:Params: {} 2022-09-13T07:34:09.160 DEBUG:boto:Token: None 2022-09-13T07:34:09.161 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.162 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VcgLDD6wIfKzOsQj3gVogTJegZ8= 2022-09-13T07:34:09.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VcgLDD6wIfKzOsQj3gVogTJegZ8=', 'Content-Length': '0'} 2022-09-13T07:34:09.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084bd3a0633662b6e-0063203271-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.166 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.166 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.167 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.167 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.168 DEBUG:boto:Data: 2022-09-13T07:34:09.168 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.169 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.169 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.170 DEBUG:boto:Params: {} 2022-09-13T07:34:09.171 DEBUG:boto:Token: None 2022-09-13T07:34:09.171 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.172 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VcgLDD6wIfKzOsQj3gVogTJegZ8= 2022-09-13T07:34:09.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VcgLDD6wIfKzOsQj3gVogTJegZ8=', 'Content-Length': '0'} 2022-09-13T07:34:09.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d55f16cad629e308-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:09.176 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.177 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.177 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.178 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.178 DEBUG:boto:Method: GET 2022-09-13T07:34:09.179 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.180 DEBUG:boto:Data: 2022-09-13T07:34:09.180 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.181 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.181 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.182 DEBUG:boto:Params: {} 2022-09-13T07:34:09.182 DEBUG:boto:Token: None 2022-09-13T07:34:09.183 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.183 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6TyuM36fr6i3oBiCBsJg3npyEkY= 2022-09-13T07:34:09.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6TyuM36fr6i3oBiCBsJg3npyEkY=', 'Content-Length': '0'} 2022-09-13T07:34:09.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000493fc534ce5a3067-0063203271-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.187 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.188 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:09.189 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.189 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.190 DEBUG:boto:Method: GET 2022-09-13T07:34:09.190 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.191 DEBUG:boto:Data: 2022-09-13T07:34:09.191 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.192 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.192 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.193 DEBUG:boto:Params: {} 2022-09-13T07:34:09.193 DEBUG:boto:Token: None 2022-09-13T07:34:09.194 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:09.195 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6TyuM36fr6i3oBiCBsJg3npyEkY= 2022-09-13T07:34:09.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6TyuM36fr6i3oBiCBsJg3npyEkY=', 'Content-Length': '0'} 2022-09-13T07:34:09.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3b9ee4ab8b25a49-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.200 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.200 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.201 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:09.201 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:09.202 DEBUG:boto:Method: GET 2022-09-13T07:34:09.203 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:09.203 DEBUG:boto:Data: 2022-09-13T07:34:09.204 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.204 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.205 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.205 DEBUG:boto:Params: {} 2022-09-13T07:34:09.206 DEBUG:boto:Token: None 2022-09-13T07:34:09.207 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.208 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo= 2022-09-13T07:34:09.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo=', 'Content-Length': '0'} 2022-09-13T07:34:09.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e91ad1421ae48e7-0063203271-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.212 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.213 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.214 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.215 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:09.216 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:09.217 DEBUG:boto:Method: GET 2022-09-13T07:34:09.218 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:09.219 DEBUG:boto:Data: 2022-09-13T07:34:09.220 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.221 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.222 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.222 DEBUG:boto:Params: {} 2022-09-13T07:34:09.224 DEBUG:boto:Token: None 2022-09-13T07:34:09.225 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.226 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo= 2022-09-13T07:34:09.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo=', 'Content-Length': '0'} 2022-09-13T07:34:09.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c7c151b9aac817f-0063203271-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.230 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:09.232 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.232 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.233 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.233 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.234 DEBUG:boto:Method: GET 2022-09-13T07:34:09.235 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.235 DEBUG:boto:Data: 2022-09-13T07:34:09.236 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.236 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.237 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.237 DEBUG:boto:Params: {} 2022-09-13T07:34:09.238 DEBUG:boto:Token: None 2022-09-13T07:34:09.238 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.239 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aDIFvWQ6qzz2QjBYg4+NDebAGwQ= 2022-09-13T07:34:09.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aDIFvWQ6qzz2QjBYg4+NDebAGwQ=', 'Content-Length': '0'} 2022-09-13T07:34:09.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090dfcbd7e602d495-0063203271-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.286 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.287 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.288 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.289 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.289 DEBUG:boto:Method: GET 2022-09-13T07:34:09.290 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.290 DEBUG:boto:Data: 2022-09-13T07:34:09.291 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.291 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.292 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.293 DEBUG:boto:Params: {} 2022-09-13T07:34:09.293 DEBUG:boto:Token: None 2022-09-13T07:34:09.294 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.295 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aDIFvWQ6qzz2QjBYg4+NDebAGwQ= 2022-09-13T07:34:09.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aDIFvWQ6qzz2QjBYg4+NDebAGwQ=', 'Content-Length': '0'} 2022-09-13T07:34:09.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f388cf9f67ce6e8b-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.299 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.299 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.300 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.300 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.301 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.301 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.302 DEBUG:boto:Data: 2022-09-13T07:34:09.302 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.303 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.304 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.304 DEBUG:boto:Params: {} 2022-09-13T07:34:09.305 DEBUG:boto:Token: None 2022-09-13T07:34:09.305 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.306 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hbDIuVJnazqCoHeFIuhu2LhR6Lc= 2022-09-13T07:34:09.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hbDIuVJnazqCoHeFIuhu2LhR6Lc=', 'Content-Length': '0'} 2022-09-13T07:34:09.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009778bc11112576d3-0063203271-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.309 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.310 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.311 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.312 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.313 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.314 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.315 DEBUG:boto:Data: 2022-09-13T07:34:09.316 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.316 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.317 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.317 DEBUG:boto:Params: {} 2022-09-13T07:34:09.318 DEBUG:boto:Token: None 2022-09-13T07:34:09.318 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.319 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hbDIuVJnazqCoHeFIuhu2LhR6Lc= 2022-09-13T07:34:09.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hbDIuVJnazqCoHeFIuhu2LhR6Lc=', 'Content-Length': '0'} 2022-09-13T07:34:09.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e54be96020836e5-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:09.324 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.324 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.325 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.326 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.326 DEBUG:boto:Method: GET 2022-09-13T07:34:09.327 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.327 DEBUG:boto:Data: 2022-09-13T07:34:09.328 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.328 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.329 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.330 DEBUG:boto:Params: {} 2022-09-13T07:34:09.330 DEBUG:boto:Token: None 2022-09-13T07:34:09.331 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.332 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aDIFvWQ6qzz2QjBYg4+NDebAGwQ= 2022-09-13T07:34:09.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aDIFvWQ6qzz2QjBYg4+NDebAGwQ=', 'Content-Length': '0'} 2022-09-13T07:34:09.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7cabc72bf0bc9e5-0063203271-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.337 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.338 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.339 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.340 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.341 DEBUG:boto:Method: GET 2022-09-13T07:34:09.342 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.343 DEBUG:boto:Data: 2022-09-13T07:34:09.344 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.345 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.346 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.347 DEBUG:boto:Params: {} 2022-09-13T07:34:09.348 DEBUG:boto:Token: None 2022-09-13T07:34:09.349 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.350 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aDIFvWQ6qzz2QjBYg4+NDebAGwQ= 2022-09-13T07:34:09.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aDIFvWQ6qzz2QjBYg4+NDebAGwQ=', 'Content-Length': '0'} 2022-09-13T07:34:09.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e5502b0c4f4c185-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.398 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.399 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.400 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:09.401 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:09.403 DEBUG:boto:Method: GET 2022-09-13T07:34:09.404 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:09.405 DEBUG:boto:Data: 2022-09-13T07:34:09.406 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.407 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.408 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.409 DEBUG:boto:Params: {} 2022-09-13T07:34:09.410 DEBUG:boto:Token: None 2022-09-13T07:34:09.411 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.413 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo= 2022-09-13T07:34:09.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo=', 'Content-Length': '0'} 2022-09-13T07:34:09.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084eff5e238320d3a-0063203271-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.418 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.420 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.421 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.422 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:09.423 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:09.424 DEBUG:boto:Method: GET 2022-09-13T07:34:09.425 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:09.426 DEBUG:boto:Data: 2022-09-13T07:34:09.427 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.428 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.429 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.430 DEBUG:boto:Params: {} 2022-09-13T07:34:09.431 DEBUG:boto:Token: None 2022-09-13T07:34:09.432 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.433 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo= 2022-09-13T07:34:09.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo=', 'Content-Length': '0'} 2022-09-13T07:34:09.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006738bcab5ef3bc8c-0063203271-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.439 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.440 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.441 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.442 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.443 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.444 DEBUG:boto:Data: 2022-09-13T07:34:09.445 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.446 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.447 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.448 DEBUG:boto:Params: {} 2022-09-13T07:34:09.449 DEBUG:boto:Token: None 2022-09-13T07:34:09.450 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.451 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ByicD/46P4EHnoYQku+rw145lc0= 2022-09-13T07:34:09.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ByicD/46P4EHnoYQku+rw145lc0=', 'Content-Length': '0'} 2022-09-13T07:34:09.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099cb173ee19c5e81-0063203271-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.457 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.457 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.458 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.459 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.460 DEBUG:boto:Data: 2022-09-13T07:34:09.461 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.462 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.463 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.464 DEBUG:boto:Params: {} 2022-09-13T07:34:09.464 DEBUG:boto:Token: None 2022-09-13T07:34:09.465 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.465 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ByicD/46P4EHnoYQku+rw145lc0= 2022-09-13T07:34:09.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ByicD/46P4EHnoYQku+rw145lc0=', 'Content-Length': '0'} 2022-09-13T07:34:09.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f838e2f0dcae136-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:09.470 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.471 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.471 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.472 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.472 DEBUG:boto:Method: GET 2022-09-13T07:34:09.473 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.473 DEBUG:boto:Data: 2022-09-13T07:34:09.474 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.475 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.477 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.478 DEBUG:boto:Params: {} 2022-09-13T07:34:09.479 DEBUG:boto:Token: None 2022-09-13T07:34:09.480 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.481 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aDIFvWQ6qzz2QjBYg4+NDebAGwQ= 2022-09-13T07:34:09.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aDIFvWQ6qzz2QjBYg4+NDebAGwQ=', 'Content-Length': '0'} 2022-09-13T07:34:09.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000585176ea3767c079-0063203271-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.485 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.485 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:09.486 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.487 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.488 DEBUG:boto:Method: GET 2022-09-13T07:34:09.488 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.489 DEBUG:boto:Data: 2022-09-13T07:34:09.489 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.490 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.490 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.491 DEBUG:boto:Params: {} 2022-09-13T07:34:09.491 DEBUG:boto:Token: None 2022-09-13T07:34:09.492 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:09.492 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aDIFvWQ6qzz2QjBYg4+NDebAGwQ= 2022-09-13T07:34:09.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aDIFvWQ6qzz2QjBYg4+NDebAGwQ=', 'Content-Length': '0'} 2022-09-13T07:34:09.495 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000793fa3571cfcc268-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.496 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.497 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.498 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:09.498 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:09.499 DEBUG:boto:Method: GET 2022-09-13T07:34:09.499 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:09.500 DEBUG:boto:Data: 2022-09-13T07:34:09.501 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.501 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.502 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.503 DEBUG:boto:Params: {} 2022-09-13T07:34:09.504 DEBUG:boto:Token: None 2022-09-13T07:34:09.505 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.506 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo= 2022-09-13T07:34:09.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo=', 'Content-Length': '0'} 2022-09-13T07:34:09.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003910cb3516d79a26-0063203271-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.511 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.512 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.513 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.514 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:09.515 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:09.515 DEBUG:boto:Method: GET 2022-09-13T07:34:09.516 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:09.516 DEBUG:boto:Data: 2022-09-13T07:34:09.517 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.517 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.518 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.518 DEBUG:boto:Params: {} 2022-09-13T07:34:09.519 DEBUG:boto:Token: None 2022-09-13T07:34:09.520 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.520 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo= 2022-09-13T07:34:09.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo=', 'Content-Length': '0'} 2022-09-13T07:34:09.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b85e0213e63fa015-0063203271-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.524 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:09.526 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.526 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.527 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.527 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.528 DEBUG:boto:Method: GET 2022-09-13T07:34:09.528 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.529 DEBUG:boto:Data: 2022-09-13T07:34:09.529 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.530 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.531 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.531 DEBUG:boto:Params: {} 2022-09-13T07:34:09.532 DEBUG:boto:Token: None 2022-09-13T07:34:09.533 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.533 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9JeUHHVXJ4JYORgimvH6DnGou3o= 2022-09-13T07:34:09.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9JeUHHVXJ4JYORgimvH6DnGou3o=', 'Content-Length': '0'} 2022-09-13T07:34:09.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ed438bc62623371-0063203271-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.578 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.579 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.579 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.580 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.580 DEBUG:boto:Method: GET 2022-09-13T07:34:09.581 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.581 DEBUG:boto:Data: 2022-09-13T07:34:09.582 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.582 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.583 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.584 DEBUG:boto:Params: {} 2022-09-13T07:34:09.584 DEBUG:boto:Token: None 2022-09-13T07:34:09.585 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.585 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9JeUHHVXJ4JYORgimvH6DnGou3o= 2022-09-13T07:34:09.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9JeUHHVXJ4JYORgimvH6DnGou3o=', 'Content-Length': '0'} 2022-09-13T07:34:09.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000caa876c9c97ed31b-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.589 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.590 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.590 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.591 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.591 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.592 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.592 DEBUG:boto:Data: 2022-09-13T07:34:09.592 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.593 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.594 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.594 DEBUG:boto:Params: {} 2022-09-13T07:34:09.595 DEBUG:boto:Token: None 2022-09-13T07:34:09.596 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.596 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pZPEQTgKS1dbVD/rnvQxMkfsbd0= 2022-09-13T07:34:09.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pZPEQTgKS1dbVD/rnvQxMkfsbd0=', 'Content-Length': '0'} 2022-09-13T07:34:09.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004c133293d070b60-0063203271-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.599 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.600 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.600 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.601 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.601 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.602 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.602 DEBUG:boto:Data: 2022-09-13T07:34:09.603 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.604 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.604 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.605 DEBUG:boto:Params: {} 2022-09-13T07:34:09.605 DEBUG:boto:Token: None 2022-09-13T07:34:09.606 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.606 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pZPEQTgKS1dbVD/rnvQxMkfsbd0= 2022-09-13T07:34:09.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pZPEQTgKS1dbVD/rnvQxMkfsbd0=', 'Content-Length': '0'} 2022-09-13T07:34:09.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f001e0f456c697e-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:09.610 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.611 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.611 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.612 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.612 DEBUG:boto:Method: GET 2022-09-13T07:34:09.613 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.614 DEBUG:boto:Data: 2022-09-13T07:34:09.614 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.615 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.615 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.616 DEBUG:boto:Params: {} 2022-09-13T07:34:09.616 DEBUG:boto:Token: None 2022-09-13T07:34:09.617 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.618 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9JeUHHVXJ4JYORgimvH6DnGou3o= 2022-09-13T07:34:09.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9JeUHHVXJ4JYORgimvH6DnGou3o=', 'Content-Length': '0'} 2022-09-13T07:34:09.620 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1c146c6e919601b-0063203271-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.621 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.622 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.622 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.623 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.623 DEBUG:boto:Method: GET 2022-09-13T07:34:09.624 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.624 DEBUG:boto:Data: 2022-09-13T07:34:09.625 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.625 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.626 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.626 DEBUG:boto:Params: {} 2022-09-13T07:34:09.627 DEBUG:boto:Token: None 2022-09-13T07:34:09.628 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.628 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9JeUHHVXJ4JYORgimvH6DnGou3o= 2022-09-13T07:34:09.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9JeUHHVXJ4JYORgimvH6DnGou3o=', 'Content-Length': '0'} 2022-09-13T07:34:09.631 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052476c4c5511ce9b-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.632 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.633 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.633 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:09.634 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:09.634 DEBUG:boto:Method: GET 2022-09-13T07:34:09.635 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:09.635 DEBUG:boto:Data: 2022-09-13T07:34:09.636 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.636 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.637 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.637 DEBUG:boto:Params: {} 2022-09-13T07:34:09.638 DEBUG:boto:Token: None 2022-09-13T07:34:09.639 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.639 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo= 2022-09-13T07:34:09.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo=', 'Content-Length': '0'} 2022-09-13T07:34:09.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a283acb0db70f067-0063203271-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.644 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.644 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.645 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.645 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:09.646 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:09.646 DEBUG:boto:Method: GET 2022-09-13T07:34:09.647 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:09.647 DEBUG:boto:Data: 2022-09-13T07:34:09.648 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.648 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.649 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.650 DEBUG:boto:Params: {} 2022-09-13T07:34:09.650 DEBUG:boto:Token: None 2022-09-13T07:34:09.651 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.652 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo= 2022-09-13T07:34:09.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo=', 'Content-Length': '0'} 2022-09-13T07:34:09.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a90f87df51071e0f-0063203271-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.656 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.656 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.657 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.657 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.658 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.659 DEBUG:boto:Data: 2022-09-13T07:34:09.659 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.660 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.660 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.661 DEBUG:boto:Params: {} 2022-09-13T07:34:09.661 DEBUG:boto:Token: None 2022-09-13T07:34:09.662 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.663 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:C5Ea+NP4HFNG6mciu6VVKRMyq3s= 2022-09-13T07:34:09.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:C5Ea+NP4HFNG6mciu6VVKRMyq3s=', 'Content-Length': '0'} 2022-09-13T07:34:09.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003964dadab3261279-0063203271-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.666 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.667 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.668 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.668 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.669 DEBUG:boto:Data: 2022-09-13T07:34:09.669 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.670 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.670 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.671 DEBUG:boto:Params: {} 2022-09-13T07:34:09.671 DEBUG:boto:Token: None 2022-09-13T07:34:09.672 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.673 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:C5Ea+NP4HFNG6mciu6VVKRMyq3s= 2022-09-13T07:34:09.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:C5Ea+NP4HFNG6mciu6VVKRMyq3s=', 'Content-Length': '0'} 2022-09-13T07:34:09.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e7dcf7af18bb48b-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:09.678 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.678 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.679 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.679 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.680 DEBUG:boto:Method: GET 2022-09-13T07:34:09.680 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.681 DEBUG:boto:Data: 2022-09-13T07:34:09.681 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.682 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.682 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.683 DEBUG:boto:Params: {} 2022-09-13T07:34:09.683 DEBUG:boto:Token: None 2022-09-13T07:34:09.684 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.685 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9JeUHHVXJ4JYORgimvH6DnGou3o= 2022-09-13T07:34:09.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9JeUHHVXJ4JYORgimvH6DnGou3o=', 'Content-Length': '0'} 2022-09-13T07:34:09.688 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030ec497b8de825ad-0063203271-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.730 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.731 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:09.732 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.732 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.733 DEBUG:boto:Method: GET 2022-09-13T07:34:09.733 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.734 DEBUG:boto:Data: 2022-09-13T07:34:09.734 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.735 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.735 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.736 DEBUG:boto:Params: {} 2022-09-13T07:34:09.736 DEBUG:boto:Token: None 2022-09-13T07:34:09.737 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:09.738 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9JeUHHVXJ4JYORgimvH6DnGou3o= 2022-09-13T07:34:09.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9JeUHHVXJ4JYORgimvH6DnGou3o=', 'Content-Length': '0'} 2022-09-13T07:34:09.740 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb402a265621a612-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.741 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.742 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.742 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:09.743 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:09.743 DEBUG:boto:Method: GET 2022-09-13T07:34:09.744 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:09.744 DEBUG:boto:Data: 2022-09-13T07:34:09.745 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.745 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:09.746 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.746 DEBUG:boto:Params: {} 2022-09-13T07:34:09.747 DEBUG:boto:Token: None 2022-09-13T07:34:09.748 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.748 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo= 2022-09-13T07:34:09.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo=', 'Content-Length': '0'} 2022-09-13T07:34:09.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a9fec19c80d538a-0063203271-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.753 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.754 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.755 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.755 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:09.756 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:09.756 DEBUG:boto:Method: GET 2022-09-13T07:34:09.757 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:09.757 DEBUG:boto:Data: 2022-09-13T07:34:09.758 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.759 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.759 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.760 DEBUG:boto:Params: {} 2022-09-13T07:34:09.760 DEBUG:boto:Token: None 2022-09-13T07:34:09.761 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.762 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo= 2022-09-13T07:34:09.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo=', 'Content-Length': '0'} 2022-09-13T07:34:09.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000391df44e56ad8813-0063203271-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.766 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.767 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-16 zones={test-zone2, test-zone1} 2022-09-13T07:34:09.768 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-16 zones={test-zone3, test-zone1} 2022-09-13T07:34:09.768 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.769 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.770 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.770 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:34:09.771 DEBUG:boto:Data: 2022-09-13T07:34:09.771 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.772 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:09.772 DEBUG:boto:Port: 8001 2022-09-13T07:34:09.773 DEBUG:boto:Params: {} 2022-09-13T07:34:09.773 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-13T07:34:09.774 DEBUG:boto:Token: None 2022-09-13T07:34:09.775 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.775 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5/A17qOFxRaj6+Gm8yLrLLwDrCk= 2022-09-13T07:34:09.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5/A17qOFxRaj6+Gm8yLrLLwDrCk=', 'Content-Length': '0'} 2022-09-13T07:34:09.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078b3bf095eb8d94c-0063203271-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.780 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.781 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.781 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.782 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:34:09.782 DEBUG:boto:Data: 2022-09-13T07:34:09.783 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.783 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.784 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.784 DEBUG:boto:Params: {} 2022-09-13T07:34:09.785 DEBUG:boto:Token: None 2022-09-13T07:34:09.786 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.786 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5/A17qOFxRaj6+Gm8yLrLLwDrCk= 2022-09-13T07:34:09.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5/A17qOFxRaj6+Gm8yLrLLwDrCk=', 'Content-Length': '0'} 2022-09-13T07:34:09.789 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee67dabf8ef90088-0063203271-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.790 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:34:09.791 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.791 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.792 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:09.792 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:09.793 DEBUG:boto:Method: GET 2022-09-13T07:34:09.793 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:09.794 DEBUG:boto:Data: 2022-09-13T07:34:09.794 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.795 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:09.795 DEBUG:boto:Port: 8001 2022-09-13T07:34:09.796 DEBUG:boto:Params: {} 2022-09-13T07:34:09.796 DEBUG:boto:Token: None 2022-09-13T07:34:09.797 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/?versions 2022-09-13T07:34:09.798 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Np0qD4wsW9sCmYYN6jLNDZ22mVU= 2022-09-13T07:34:09.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Np0qD4wsW9sCmYYN6jLNDZ22mVU=', 'Content-Length': '0'} 2022-09-13T07:34:09.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d34d82483e162bb0-0063203271-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.802 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.803 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:34:09.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:34:09.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:34:09.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:34:09.807 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:34:09.808 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.809 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.809 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:09.810 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:09.811 DEBUG:boto:Method: GET 2022-09-13T07:34:09.813 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:09.813 DEBUG:boto:Data: 2022-09-13T07:34:09.814 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.815 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.816 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.817 DEBUG:boto:Params: {} 2022-09-13T07:34:09.818 DEBUG:boto:Token: None 2022-09-13T07:34:09.819 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/?versions 2022-09-13T07:34:09.821 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Np0qD4wsW9sCmYYN6jLNDZ22mVU= 2022-09-13T07:34:09.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Np0qD4wsW9sCmYYN6jLNDZ22mVU=', 'Content-Length': '0'} 2022-09-13T07:34:09.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004d70a18f053fcd7-0063203271-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.826 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.828 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:34:09.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:34:09.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:34:09.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:34:09.832 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.833 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.834 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:09.835 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:09.836 DEBUG:boto:Method: GET 2022-09-13T07:34:09.837 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:09.838 DEBUG:boto:Data: 2022-09-13T07:34:09.839 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.840 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:09.841 DEBUG:boto:Port: 8001 2022-09-13T07:34:09.842 DEBUG:boto:Params: {} 2022-09-13T07:34:09.843 DEBUG:boto:Token: None 2022-09-13T07:34:09.844 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/?versions 2022-09-13T07:34:09.845 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Np0qD4wsW9sCmYYN6jLNDZ22mVU= 2022-09-13T07:34:09.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Np0qD4wsW9sCmYYN6jLNDZ22mVU=', 'Content-Length': '0'} 2022-09-13T07:34:09.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099c2c3a29c6b5f61-0063203271-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.851 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.852 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.853 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.854 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:09.855 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:09.856 DEBUG:boto:Method: GET 2022-09-13T07:34:09.857 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:09.858 DEBUG:boto:Data: 2022-09-13T07:34:09.858 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.859 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.859 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.860 DEBUG:boto:Params: {} 2022-09-13T07:34:09.860 DEBUG:boto:Token: None 2022-09-13T07:34:09.861 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/?versions 2022-09-13T07:34:09.862 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Np0qD4wsW9sCmYYN6jLNDZ22mVU= 2022-09-13T07:34:09.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Np0qD4wsW9sCmYYN6jLNDZ22mVU=', 'Content-Length': '0'} 2022-09-13T07:34:09.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099cfee075ceb0dfb-0063203271-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.866 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:09.867 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:09.868 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:09.869 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.869 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.870 DEBUG:boto:Method: GET 2022-09-13T07:34:09.870 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.871 DEBUG:boto:Data: 2022-09-13T07:34:09.871 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.872 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:09.872 DEBUG:boto:Port: 8001 2022-09-13T07:34:09.873 DEBUG:boto:Params: {} 2022-09-13T07:34:09.873 DEBUG:boto:Token: None 2022-09-13T07:34:09.874 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.875 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MDGtpHGxuRbkWMksVzJidVUf9hs= 2022-09-13T07:34:09.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MDGtpHGxuRbkWMksVzJidVUf9hs=', 'Content-Length': '0'} 2022-09-13T07:34:09.877 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001ddb7f5c739f8683-0063203271-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.878 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:09.879 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:09.879 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.880 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.881 DEBUG:boto:Method: GET 2022-09-13T07:34:09.881 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.882 DEBUG:boto:Data: 2022-09-13T07:34:09.882 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.883 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.884 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.884 DEBUG:boto:Params: {} 2022-09-13T07:34:09.885 DEBUG:boto:Token: None 2022-09-13T07:34:09.885 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.886 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MDGtpHGxuRbkWMksVzJidVUf9hs= 2022-09-13T07:34:09.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MDGtpHGxuRbkWMksVzJidVUf9hs=', 'Content-Length': '0'} 2022-09-13T07:34:09.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000025e8c2fe1111019a-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.889 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:09.890 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:09.891 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.891 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.892 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.892 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.893 DEBUG:boto:Data: 2022-09-13T07:34:09.893 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.894 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:09.895 DEBUG:boto:Port: 8001 2022-09-13T07:34:09.895 DEBUG:boto:Params: {} 2022-09-13T07:34:09.896 DEBUG:boto:Token: None 2022-09-13T07:34:09.896 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.897 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6E5M1Gr4cIq3N+8fjCiX8KS/3zU= 2022-09-13T07:34:09.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6E5M1Gr4cIq3N+8fjCiX8KS/3zU=', 'Content-Length': '0'} 2022-09-13T07:34:09.899 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fea92c152e799611-0063203271-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.900 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:09.901 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:09.902 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.902 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.903 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.903 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.904 DEBUG:boto:Data: 2022-09-13T07:34:09.904 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.905 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.905 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.906 DEBUG:boto:Params: {} 2022-09-13T07:34:09.906 DEBUG:boto:Token: None 2022-09-13T07:34:09.907 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.907 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6E5M1Gr4cIq3N+8fjCiX8KS/3zU= 2022-09-13T07:34:09.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6E5M1Gr4cIq3N+8fjCiX8KS/3zU=', 'Content-Length': '0'} 2022-09-13T07:34:09.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001cc99d757ddb03a4-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:09.912 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:09.912 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:09.913 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.914 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.914 DEBUG:boto:Method: GET 2022-09-13T07:34:09.915 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.915 DEBUG:boto:Data: 2022-09-13T07:34:09.916 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.916 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:09.917 DEBUG:boto:Port: 8001 2022-09-13T07:34:09.917 DEBUG:boto:Params: {} 2022-09-13T07:34:09.918 DEBUG:boto:Token: None 2022-09-13T07:34:09.918 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.919 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MDGtpHGxuRbkWMksVzJidVUf9hs= 2022-09-13T07:34:09.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MDGtpHGxuRbkWMksVzJidVUf9hs=', 'Content-Length': '0'} 2022-09-13T07:34:09.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000076188ce6b334b127-0063203271-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.923 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:09.924 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:09.924 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.925 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.925 DEBUG:boto:Method: GET 2022-09-13T07:34:09.926 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.926 DEBUG:boto:Data: 2022-09-13T07:34:09.927 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.927 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.928 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.928 DEBUG:boto:Params: {} 2022-09-13T07:34:09.929 DEBUG:boto:Token: None 2022-09-13T07:34:09.929 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.930 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MDGtpHGxuRbkWMksVzJidVUf9hs= 2022-09-13T07:34:09.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MDGtpHGxuRbkWMksVzJidVUf9hs=', 'Content-Length': '0'} 2022-09-13T07:34:09.932 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad1f3d80078305c4-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.933 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.934 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.934 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:09.935 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:09.935 DEBUG:boto:Method: GET 2022-09-13T07:34:09.936 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:34:09.936 DEBUG:boto:Data: 2022-09-13T07:34:09.937 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.937 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:09.938 DEBUG:boto:Port: 8001 2022-09-13T07:34:09.938 DEBUG:boto:Params: {} 2022-09-13T07:34:09.939 DEBUG:boto:Token: None 2022-09-13T07:34:09.939 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.940 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo= 2022-09-13T07:34:09.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo=', 'Content-Length': '0'} 2022-09-13T07:34:09.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000120e807fa3f2b786-0063203271-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.944 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.945 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.946 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.946 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:09.947 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:09.947 DEBUG:boto:Method: GET 2022-09-13T07:34:09.948 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:34:09.948 DEBUG:boto:Data: 2022-09-13T07:34:09.949 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.949 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.950 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.950 DEBUG:boto:Params: {} 2022-09-13T07:34:09.951 DEBUG:boto:Token: None 2022-09-13T07:34:09.951 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/ 2022-09-13T07:34:09.952 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo= 2022-09-13T07:34:09.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/13JyL+jdJLNJDHs/rY4RD2KcYo=', 'Content-Length': '0'} 2022-09-13T07:34:09.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f0ccca5866d8214-0063203271-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.956 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:09.957 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:09.957 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:09.958 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.959 DEBUG:boto:Path: /yxvvuh-16/dummy 2022-09-13T07:34:09.959 DEBUG:boto:Data: 2022-09-13T07:34:09.960 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.960 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:09.961 DEBUG:boto:Port: 8001 2022-09-13T07:34:09.961 DEBUG:boto:Params: {} 2022-09-13T07:34:09.962 DEBUG:boto:Token: None 2022-09-13T07:34:09.962 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/dummy 2022-09-13T07:34:09.963 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MnfSL0MCTG7dgc/Fy8VJ3lhcKNU= 2022-09-13T07:34:09.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MnfSL0MCTG7dgc/Fy8VJ3lhcKNU=', 'Content-Length': '0'} 2022-09-13T07:34:09.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000031e06c1d42330f7c-0063203271-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.966 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:09.967 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:09.967 DEBUG:boto:Method: HEAD 2022-09-13T07:34:09.968 DEBUG:boto:Path: /yxvvuh-16/dummy 2022-09-13T07:34:09.969 DEBUG:boto:Data: 2022-09-13T07:34:09.969 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.970 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.970 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.971 DEBUG:boto:Params: {} 2022-09-13T07:34:09.971 DEBUG:boto:Token: None 2022-09-13T07:34:09.972 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/dummy 2022-09-13T07:34:09.972 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MnfSL0MCTG7dgc/Fy8VJ3lhcKNU= 2022-09-13T07:34:09.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MnfSL0MCTG7dgc/Fy8VJ3lhcKNU=', 'Content-Length': '0'} 2022-09-13T07:34:09.975 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002888cb7537a85bb1-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:09.977 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:09.977 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:09.978 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.978 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.979 DEBUG:boto:Method: GET 2022-09-13T07:34:09.979 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.980 DEBUG:boto:Data: 2022-09-13T07:34:09.980 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.981 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:09.981 DEBUG:boto:Port: 8001 2022-09-13T07:34:09.982 DEBUG:boto:Params: {} 2022-09-13T07:34:09.982 DEBUG:boto:Token: None 2022-09-13T07:34:09.983 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.983 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MDGtpHGxuRbkWMksVzJidVUf9hs= 2022-09-13T07:34:09.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MDGtpHGxuRbkWMksVzJidVUf9hs=', 'Content-Length': '0'} 2022-09-13T07:34:09.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000060385838a572a54-0063203271-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.987 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:09.987 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:09.988 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.988 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.989 DEBUG:boto:Method: GET 2022-09-13T07:34:09.989 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.990 DEBUG:boto:Data: 2022-09-13T07:34:09.990 DEBUG:boto:Headers: {} 2022-09-13T07:34:09.991 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:09.991 DEBUG:boto:Port: 8000 2022-09-13T07:34:09.992 DEBUG:boto:Params: {} 2022-09-13T07:34:09.993 DEBUG:boto:Token: None 2022-09-13T07:34:09.993 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:09 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:09.994 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MDGtpHGxuRbkWMksVzJidVUf9hs= 2022-09-13T07:34:09.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MDGtpHGxuRbkWMksVzJidVUf9hs=', 'Content-Length': '0'} 2022-09-13T07:34:09.996 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000270d631fa39870f5-0063203271-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:09.997 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:09.998 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:09.998 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:09.999 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:09.999 DEBUG:boto:Method: GET 2022-09-13T07:34:10.000 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:34:10.000 DEBUG:boto:Data: 2022-09-13T07:34:10.001 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.001 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.002 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.002 DEBUG:boto:Params: {} 2022-09-13T07:34:10.003 DEBUG:boto:Token: None 2022-09-13T07:34:10.003 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.004 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020785b0a7961137d-0063203272-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.008 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.009 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.009 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.010 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:10.010 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:10.011 DEBUG:boto:Method: GET 2022-09-13T07:34:10.011 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:34:10.012 DEBUG:boto:Data: 2022-09-13T07:34:10.012 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.013 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.014 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.014 DEBUG:boto:Params: {} 2022-09-13T07:34:10.015 DEBUG:boto:Token: None 2022-09-13T07:34:10.015 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.016 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000546c8fdbf50086c9-0063203272-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.062 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:10.064 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.064 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.065 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.065 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.066 DEBUG:boto:Method: GET 2022-09-13T07:34:10.066 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.067 DEBUG:boto:Data: 2022-09-13T07:34:10.067 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.068 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.068 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.069 DEBUG:boto:Params: {} 2022-09-13T07:34:10.069 DEBUG:boto:Token: None 2022-09-13T07:34:10.070 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.071 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xISlIWL87wKoi3LcyY3PmzhbuJg= 2022-09-13T07:34:10.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xISlIWL87wKoi3LcyY3PmzhbuJg=', 'Content-Length': '0'} 2022-09-13T07:34:10.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b578f9115c806ca-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.074 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.075 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.075 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.076 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.076 DEBUG:boto:Method: GET 2022-09-13T07:34:10.077 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.077 DEBUG:boto:Data: 2022-09-13T07:34:10.078 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.078 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.079 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.079 DEBUG:boto:Params: {} 2022-09-13T07:34:10.080 DEBUG:boto:Token: None 2022-09-13T07:34:10.081 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.081 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xISlIWL87wKoi3LcyY3PmzhbuJg= 2022-09-13T07:34:10.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xISlIWL87wKoi3LcyY3PmzhbuJg=', 'Content-Length': '0'} 2022-09-13T07:34:10.084 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a791b459ff817a2-0063203272-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.085 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.085 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.086 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.086 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.086 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.087 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.088 DEBUG:boto:Data: 2022-09-13T07:34:10.089 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.089 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.089 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.090 DEBUG:boto:Params: {} 2022-09-13T07:34:10.091 DEBUG:boto:Token: None 2022-09-13T07:34:10.091 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.092 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:C0LJ9H+XQ5hAsMPC1FU3mCF+I2M= 2022-09-13T07:34:10.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:C0LJ9H+XQ5hAsMPC1FU3mCF+I2M=', 'Content-Length': '0'} 2022-09-13T07:34:10.094 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c32fba4e58251d3e-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.095 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.096 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.096 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.097 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.097 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.098 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.099 DEBUG:boto:Data: 2022-09-13T07:34:10.099 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.100 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.100 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.100 DEBUG:boto:Params: {} 2022-09-13T07:34:10.101 DEBUG:boto:Token: None 2022-09-13T07:34:10.102 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.102 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:C0LJ9H+XQ5hAsMPC1FU3mCF+I2M= 2022-09-13T07:34:10.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:C0LJ9H+XQ5hAsMPC1FU3mCF+I2M=', 'Content-Length': '0'} 2022-09-13T07:34:10.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e08ffbc9977a9d39-0063203272-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:10.106 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.107 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.107 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.108 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.108 DEBUG:boto:Method: GET 2022-09-13T07:34:10.109 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.109 DEBUG:boto:Data: 2022-09-13T07:34:10.110 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.110 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.111 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.111 DEBUG:boto:Params: {} 2022-09-13T07:34:10.112 DEBUG:boto:Token: None 2022-09-13T07:34:10.112 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.113 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xISlIWL87wKoi3LcyY3PmzhbuJg= 2022-09-13T07:34:10.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xISlIWL87wKoi3LcyY3PmzhbuJg=', 'Content-Length': '0'} 2022-09-13T07:34:10.116 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d210a1b53b9206ed-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.116 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.117 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.117 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.118 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.119 DEBUG:boto:Method: GET 2022-09-13T07:34:10.119 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.120 DEBUG:boto:Data: 2022-09-13T07:34:10.120 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.121 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.121 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.122 DEBUG:boto:Params: {} 2022-09-13T07:34:10.122 DEBUG:boto:Token: None 2022-09-13T07:34:10.123 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.124 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xISlIWL87wKoi3LcyY3PmzhbuJg= 2022-09-13T07:34:10.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xISlIWL87wKoi3LcyY3PmzhbuJg=', 'Content-Length': '0'} 2022-09-13T07:34:10.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b12127092a9436f-0063203272-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.127 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.128 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.128 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:10.129 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:10.130 DEBUG:boto:Method: GET 2022-09-13T07:34:10.130 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:10.131 DEBUG:boto:Data: 2022-09-13T07:34:10.131 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.132 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.132 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.133 DEBUG:boto:Params: {} 2022-09-13T07:34:10.134 DEBUG:boto:Token: None 2022-09-13T07:34:10.134 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.135 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5311fc723795f47-0063203272-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.138 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.139 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.140 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.140 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:10.141 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:10.141 DEBUG:boto:Method: GET 2022-09-13T07:34:10.142 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:10.142 DEBUG:boto:Data: 2022-09-13T07:34:10.143 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.144 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.144 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.145 DEBUG:boto:Params: {} 2022-09-13T07:34:10.145 DEBUG:boto:Token: None 2022-09-13T07:34:10.146 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.146 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4829d6bc4532fb0-0063203272-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.150 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.151 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.152 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.152 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.153 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.153 DEBUG:boto:Data: 2022-09-13T07:34:10.154 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.155 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.155 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.156 DEBUG:boto:Params: {} 2022-09-13T07:34:10.156 DEBUG:boto:Token: None 2022-09-13T07:34:10.157 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.157 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LHddHiVZcj9Ip0rCYEZuDviSEo8= 2022-09-13T07:34:10.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LHddHiVZcj9Ip0rCYEZuDviSEo8=', 'Content-Length': '0'} 2022-09-13T07:34:10.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b35c2c4f03540100-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.161 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.162 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.162 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.163 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.163 DEBUG:boto:Data: 2022-09-13T07:34:10.164 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.164 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.165 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.165 DEBUG:boto:Params: {} 2022-09-13T07:34:10.166 DEBUG:boto:Token: None 2022-09-13T07:34:10.166 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.167 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LHddHiVZcj9Ip0rCYEZuDviSEo8= 2022-09-13T07:34:10.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LHddHiVZcj9Ip0rCYEZuDviSEo8=', 'Content-Length': '0'} 2022-09-13T07:34:10.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d45b5f86a44edab0-0063203272-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:10.171 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.172 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.172 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.173 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.173 DEBUG:boto:Method: GET 2022-09-13T07:34:10.174 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.174 DEBUG:boto:Data: 2022-09-13T07:34:10.175 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.175 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.176 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.176 DEBUG:boto:Params: {} 2022-09-13T07:34:10.177 DEBUG:boto:Token: None 2022-09-13T07:34:10.177 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.178 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xISlIWL87wKoi3LcyY3PmzhbuJg= 2022-09-13T07:34:10.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xISlIWL87wKoi3LcyY3PmzhbuJg=', 'Content-Length': '0'} 2022-09-13T07:34:10.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7be628a69791b61-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.222 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.223 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:10.223 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.224 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.224 DEBUG:boto:Method: GET 2022-09-13T07:34:10.225 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.225 DEBUG:boto:Data: 2022-09-13T07:34:10.226 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.226 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.227 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.227 DEBUG:boto:Params: {} 2022-09-13T07:34:10.228 DEBUG:boto:Token: None 2022-09-13T07:34:10.228 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:10.229 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xISlIWL87wKoi3LcyY3PmzhbuJg= 2022-09-13T07:34:10.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xISlIWL87wKoi3LcyY3PmzhbuJg=', 'Content-Length': '0'} 2022-09-13T07:34:10.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000811d3d82c8efdf33-0063203272-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.232 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.233 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.233 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:10.234 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:10.234 DEBUG:boto:Method: GET 2022-09-13T07:34:10.235 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:10.235 DEBUG:boto:Data: 2022-09-13T07:34:10.236 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.236 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.237 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.237 DEBUG:boto:Params: {} 2022-09-13T07:34:10.238 DEBUG:boto:Token: None 2022-09-13T07:34:10.239 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.239 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054c287120b5a9ebd-0063203272-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.243 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.244 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.244 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.245 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:10.245 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:10.246 DEBUG:boto:Method: GET 2022-09-13T07:34:10.246 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:10.247 DEBUG:boto:Data: 2022-09-13T07:34:10.247 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.248 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.248 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.249 DEBUG:boto:Params: {} 2022-09-13T07:34:10.249 DEBUG:boto:Token: None 2022-09-13T07:34:10.250 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.251 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000072b018e143a660c-0063203272-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.254 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:10.256 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.256 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.257 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.257 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.258 DEBUG:boto:Method: GET 2022-09-13T07:34:10.258 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.259 DEBUG:boto:Data: 2022-09-13T07:34:10.259 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.260 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.260 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.261 DEBUG:boto:Params: {} 2022-09-13T07:34:10.262 DEBUG:boto:Token: None 2022-09-13T07:34:10.262 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.263 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CxmtW6Ky6qw2KMA47DXBdAQSl00= 2022-09-13T07:34:10.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CxmtW6Ky6qw2KMA47DXBdAQSl00=', 'Content-Length': '0'} 2022-09-13T07:34:10.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd9692661a7438e9-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.266 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.267 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.267 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.268 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.268 DEBUG:boto:Method: GET 2022-09-13T07:34:10.269 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.269 DEBUG:boto:Data: 2022-09-13T07:34:10.270 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.270 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.271 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.271 DEBUG:boto:Params: {} 2022-09-13T07:34:10.272 DEBUG:boto:Token: None 2022-09-13T07:34:10.272 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.273 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CxmtW6Ky6qw2KMA47DXBdAQSl00= 2022-09-13T07:34:10.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CxmtW6Ky6qw2KMA47DXBdAQSl00=', 'Content-Length': '0'} 2022-09-13T07:34:10.276 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee6970d9df3df3b3-0063203272-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.318 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.319 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.320 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.321 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.322 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.323 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.324 DEBUG:boto:Data: 2022-09-13T07:34:10.325 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.325 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.326 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.327 DEBUG:boto:Params: {} 2022-09-13T07:34:10.328 DEBUG:boto:Token: None 2022-09-13T07:34:10.329 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.329 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xI86uXHeiWu9kqvWpGQ5iZuqs94= 2022-09-13T07:34:10.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xI86uXHeiWu9kqvWpGQ5iZuqs94=', 'Content-Length': '0'} 2022-09-13T07:34:10.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9462c36f6ebb821-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.333 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.333 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.334 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.334 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.335 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.335 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.336 DEBUG:boto:Data: 2022-09-13T07:34:10.336 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.336 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.337 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.337 DEBUG:boto:Params: {} 2022-09-13T07:34:10.338 DEBUG:boto:Token: None 2022-09-13T07:34:10.339 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.339 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xI86uXHeiWu9kqvWpGQ5iZuqs94= 2022-09-13T07:34:10.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xI86uXHeiWu9kqvWpGQ5iZuqs94=', 'Content-Length': '0'} 2022-09-13T07:34:10.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000126e62cad076c43c-0063203272-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:10.343 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.344 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.344 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.345 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.345 DEBUG:boto:Method: GET 2022-09-13T07:34:10.346 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.346 DEBUG:boto:Data: 2022-09-13T07:34:10.347 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.348 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.349 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.350 DEBUG:boto:Params: {} 2022-09-13T07:34:10.351 DEBUG:boto:Token: None 2022-09-13T07:34:10.352 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.353 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CxmtW6Ky6qw2KMA47DXBdAQSl00= 2022-09-13T07:34:10.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CxmtW6Ky6qw2KMA47DXBdAQSl00=', 'Content-Length': '0'} 2022-09-13T07:34:10.357 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002e15315a2a64951-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.358 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.359 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.360 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.361 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.362 DEBUG:boto:Method: GET 2022-09-13T07:34:10.363 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.364 DEBUG:boto:Data: 2022-09-13T07:34:10.365 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.366 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.367 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.367 DEBUG:boto:Params: {} 2022-09-13T07:34:10.369 DEBUG:boto:Token: None 2022-09-13T07:34:10.370 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.371 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CxmtW6Ky6qw2KMA47DXBdAQSl00= 2022-09-13T07:34:10.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CxmtW6Ky6qw2KMA47DXBdAQSl00=', 'Content-Length': '0'} 2022-09-13T07:34:10.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc0d1b4810c2ca8a-0063203272-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.376 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.377 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.377 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:10.378 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:10.379 DEBUG:boto:Method: GET 2022-09-13T07:34:10.380 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:10.381 DEBUG:boto:Data: 2022-09-13T07:34:10.382 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.383 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.384 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.385 DEBUG:boto:Params: {} 2022-09-13T07:34:10.386 DEBUG:boto:Token: None 2022-09-13T07:34:10.386 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.387 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f18680d1a4d03aef-0063203272-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.391 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.392 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.392 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.393 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:10.393 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:10.394 DEBUG:boto:Method: GET 2022-09-13T07:34:10.395 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:10.395 DEBUG:boto:Data: 2022-09-13T07:34:10.396 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.397 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.397 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.398 DEBUG:boto:Params: {} 2022-09-13T07:34:10.398 DEBUG:boto:Token: None 2022-09-13T07:34:10.399 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.400 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a3eea72482ac65b-0063203272-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.403 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.405 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.406 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.407 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.408 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.409 DEBUG:boto:Data: 2022-09-13T07:34:10.410 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.411 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.412 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.413 DEBUG:boto:Params: {} 2022-09-13T07:34:10.414 DEBUG:boto:Token: None 2022-09-13T07:34:10.415 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.416 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:raxHAOcs7T1RgA+WZBpa89tHd+Y= 2022-09-13T07:34:10.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:raxHAOcs7T1RgA+WZBpa89tHd+Y=', 'Content-Length': '0'} 2022-09-13T07:34:10.420 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b52816cb61438055-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.421 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.422 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.423 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.424 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.425 DEBUG:boto:Data: 2022-09-13T07:34:10.426 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.427 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.428 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.428 DEBUG:boto:Params: {} 2022-09-13T07:34:10.430 DEBUG:boto:Token: None 2022-09-13T07:34:10.431 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.432 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:raxHAOcs7T1RgA+WZBpa89tHd+Y= 2022-09-13T07:34:10.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:raxHAOcs7T1RgA+WZBpa89tHd+Y=', 'Content-Length': '0'} 2022-09-13T07:34:10.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba85abe3519be6c6-0063203272-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:10.438 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.439 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.441 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.442 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.443 DEBUG:boto:Method: GET 2022-09-13T07:34:10.443 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.444 DEBUG:boto:Data: 2022-09-13T07:34:10.445 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.446 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.447 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.448 DEBUG:boto:Params: {} 2022-09-13T07:34:10.449 DEBUG:boto:Token: None 2022-09-13T07:34:10.450 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.451 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CxmtW6Ky6qw2KMA47DXBdAQSl00= 2022-09-13T07:34:10.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CxmtW6Ky6qw2KMA47DXBdAQSl00=', 'Content-Length': '0'} 2022-09-13T07:34:10.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044dbddd6ea1430ba-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.508 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.509 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:10.510 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.511 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.512 DEBUG:boto:Method: GET 2022-09-13T07:34:10.513 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.514 DEBUG:boto:Data: 2022-09-13T07:34:10.515 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.516 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.517 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.518 DEBUG:boto:Params: {} 2022-09-13T07:34:10.519 DEBUG:boto:Token: None 2022-09-13T07:34:10.520 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:10.521 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CxmtW6Ky6qw2KMA47DXBdAQSl00= 2022-09-13T07:34:10.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CxmtW6Ky6qw2KMA47DXBdAQSl00=', 'Content-Length': '0'} 2022-09-13T07:34:10.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9e290e1cba4a540-0063203272-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.527 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.527 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.528 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:10.529 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:10.530 DEBUG:boto:Method: GET 2022-09-13T07:34:10.531 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:10.532 DEBUG:boto:Data: 2022-09-13T07:34:10.533 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.534 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.535 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.536 DEBUG:boto:Params: {} 2022-09-13T07:34:10.537 DEBUG:boto:Token: None 2022-09-13T07:34:10.538 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.539 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c01936246e00e4a-0063203272-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.544 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.545 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.546 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.547 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:10.548 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:10.549 DEBUG:boto:Method: GET 2022-09-13T07:34:10.551 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:10.552 DEBUG:boto:Data: 2022-09-13T07:34:10.552 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.553 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.554 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.555 DEBUG:boto:Params: {} 2022-09-13T07:34:10.556 DEBUG:boto:Token: None 2022-09-13T07:34:10.556 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.557 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080aaa79405177c4d-0063203272-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.561 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:10.562 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.563 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.563 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.564 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.564 DEBUG:boto:Method: GET 2022-09-13T07:34:10.565 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.565 DEBUG:boto:Data: 2022-09-13T07:34:10.566 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.566 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.567 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.568 DEBUG:boto:Params: {} 2022-09-13T07:34:10.568 DEBUG:boto:Token: None 2022-09-13T07:34:10.569 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.570 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9Rl+ye+c6Uo9fV3aNs2a1CxYKd8= 2022-09-13T07:34:10.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9Rl+ye+c6Uo9fV3aNs2a1CxYKd8=', 'Content-Length': '0'} 2022-09-13T07:34:10.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6ab2b3b249d1cbc-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.573 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.574 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.574 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.575 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.575 DEBUG:boto:Method: GET 2022-09-13T07:34:10.576 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.576 DEBUG:boto:Data: 2022-09-13T07:34:10.577 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.578 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.578 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.579 DEBUG:boto:Params: {} 2022-09-13T07:34:10.579 DEBUG:boto:Token: None 2022-09-13T07:34:10.580 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.581 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9Rl+ye+c6Uo9fV3aNs2a1CxYKd8= 2022-09-13T07:34:10.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9Rl+ye+c6Uo9fV3aNs2a1CxYKd8=', 'Content-Length': '0'} 2022-09-13T07:34:10.583 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000afb65a811b08edb6-0063203272-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.584 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.585 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.586 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.587 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.588 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.589 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.589 DEBUG:boto:Data: 2022-09-13T07:34:10.590 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.590 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.591 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.591 DEBUG:boto:Params: {} 2022-09-13T07:34:10.592 DEBUG:boto:Token: None 2022-09-13T07:34:10.593 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.593 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LzMXbBz/TiOxa30Cbszyl/tzmOA= 2022-09-13T07:34:10.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LzMXbBz/TiOxa30Cbszyl/tzmOA=', 'Content-Length': '0'} 2022-09-13T07:34:10.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b1734dcb4c4c982-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.596 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.597 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.597 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.598 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.598 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.599 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.599 DEBUG:boto:Data: 2022-09-13T07:34:10.600 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.600 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.601 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.601 DEBUG:boto:Params: {} 2022-09-13T07:34:10.602 DEBUG:boto:Token: None 2022-09-13T07:34:10.603 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.603 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LzMXbBz/TiOxa30Cbszyl/tzmOA= 2022-09-13T07:34:10.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LzMXbBz/TiOxa30Cbszyl/tzmOA=', 'Content-Length': '0'} 2022-09-13T07:34:10.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005251f73d3d374b50-0063203272-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:10.607 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.608 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.608 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.609 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.609 DEBUG:boto:Method: GET 2022-09-13T07:34:10.610 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.610 DEBUG:boto:Data: 2022-09-13T07:34:10.611 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.612 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.613 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.614 DEBUG:boto:Params: {} 2022-09-13T07:34:10.615 DEBUG:boto:Token: None 2022-09-13T07:34:10.616 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.617 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9Rl+ye+c6Uo9fV3aNs2a1CxYKd8= 2022-09-13T07:34:10.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9Rl+ye+c6Uo9fV3aNs2a1CxYKd8=', 'Content-Length': '0'} 2022-09-13T07:34:10.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021173d0274f04dd0-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.623 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.623 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.624 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.625 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.626 DEBUG:boto:Method: GET 2022-09-13T07:34:10.627 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.628 DEBUG:boto:Data: 2022-09-13T07:34:10.629 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.629 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.630 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.631 DEBUG:boto:Params: {} 2022-09-13T07:34:10.633 DEBUG:boto:Token: None 2022-09-13T07:34:10.633 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.634 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9Rl+ye+c6Uo9fV3aNs2a1CxYKd8= 2022-09-13T07:34:10.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9Rl+ye+c6Uo9fV3aNs2a1CxYKd8=', 'Content-Length': '0'} 2022-09-13T07:34:10.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060e72a1dcd60f0aa-0063203272-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.638 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.638 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.639 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:10.639 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:10.640 DEBUG:boto:Method: GET 2022-09-13T07:34:10.640 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:10.641 DEBUG:boto:Data: 2022-09-13T07:34:10.641 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.642 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.642 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.643 DEBUG:boto:Params: {} 2022-09-13T07:34:10.644 DEBUG:boto:Token: None 2022-09-13T07:34:10.644 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.645 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d546123281948559-0063203272-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.649 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.650 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.651 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.651 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:10.652 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:10.652 DEBUG:boto:Method: GET 2022-09-13T07:34:10.653 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:10.653 DEBUG:boto:Data: 2022-09-13T07:34:10.654 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.655 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.655 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.656 DEBUG:boto:Params: {} 2022-09-13T07:34:10.656 DEBUG:boto:Token: None 2022-09-13T07:34:10.657 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.658 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097b5addfd28049e8-0063203272-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.662 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.663 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.663 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.664 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.665 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.665 DEBUG:boto:Data: 2022-09-13T07:34:10.666 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.666 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.667 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.668 DEBUG:boto:Params: {} 2022-09-13T07:34:10.668 DEBUG:boto:Token: None 2022-09-13T07:34:10.669 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.669 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ogEHXRkp2QGuHf+u4215/Iuupw8= 2022-09-13T07:34:10.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ogEHXRkp2QGuHf+u4215/Iuupw8=', 'Content-Length': '0'} 2022-09-13T07:34:10.673 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ddaca1dd6ad9002-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.673 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.674 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.674 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.675 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.676 DEBUG:boto:Data: 2022-09-13T07:34:10.676 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.677 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.677 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.678 DEBUG:boto:Params: {} 2022-09-13T07:34:10.679 DEBUG:boto:Token: None 2022-09-13T07:34:10.679 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.680 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ogEHXRkp2QGuHf+u4215/Iuupw8= 2022-09-13T07:34:10.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ogEHXRkp2QGuHf+u4215/Iuupw8=', 'Content-Length': '0'} 2022-09-13T07:34:10.684 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e2a1c38849dd4e4-0063203272-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:10.685 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.685 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.686 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.687 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.688 DEBUG:boto:Method: GET 2022-09-13T07:34:10.689 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.690 DEBUG:boto:Data: 2022-09-13T07:34:10.691 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.692 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.693 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.694 DEBUG:boto:Params: {} 2022-09-13T07:34:10.695 DEBUG:boto:Token: None 2022-09-13T07:34:10.696 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.697 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9Rl+ye+c6Uo9fV3aNs2a1CxYKd8= 2022-09-13T07:34:10.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9Rl+ye+c6Uo9fV3aNs2a1CxYKd8=', 'Content-Length': '0'} 2022-09-13T07:34:10.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057e81b3faf0b0144-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.701 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.702 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:10.702 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.703 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.703 DEBUG:boto:Method: GET 2022-09-13T07:34:10.704 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.704 DEBUG:boto:Data: 2022-09-13T07:34:10.705 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.706 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.706 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.707 DEBUG:boto:Params: {} 2022-09-13T07:34:10.707 DEBUG:boto:Token: None 2022-09-13T07:34:10.708 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:10.708 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9Rl+ye+c6Uo9fV3aNs2a1CxYKd8= 2022-09-13T07:34:10.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9Rl+ye+c6Uo9fV3aNs2a1CxYKd8=', 'Content-Length': '0'} 2022-09-13T07:34:10.711 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086c8d9263d2f8cc7-0063203272-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.712 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.712 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.713 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:10.713 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:10.714 DEBUG:boto:Method: GET 2022-09-13T07:34:10.714 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:10.715 DEBUG:boto:Data: 2022-09-13T07:34:10.715 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.716 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.716 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.717 DEBUG:boto:Params: {} 2022-09-13T07:34:10.718 DEBUG:boto:Token: None 2022-09-13T07:34:10.719 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.720 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dc8596b41ad52c7-0063203272-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.726 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.727 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.728 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.729 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:10.729 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:10.729 DEBUG:boto:Method: GET 2022-09-13T07:34:10.730 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:10.731 DEBUG:boto:Data: 2022-09-13T07:34:10.731 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.731 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:10.732 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.733 DEBUG:boto:Params: {} 2022-09-13T07:34:10.733 DEBUG:boto:Token: None 2022-09-13T07:34:10.734 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.734 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0c4845a688bf84a-0063203272-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.738 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.739 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-16 zones={test-zone3, test-zone1} 2022-09-13T07:34:10.740 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-16 zones={test-zone3, test-zone2} 2022-09-13T07:34:10.740 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.741 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.741 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.742 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:34:10.742 DEBUG:boto:Data: 2022-09-13T07:34:10.743 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.743 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.744 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.744 DEBUG:boto:Params: {} 2022-09-13T07:34:10.745 DEBUG:boto:Token: None 2022-09-13T07:34:10.746 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.746 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ro3gZLpDzpmSBCvHpyZ7pRXLo+0= 2022-09-13T07:34:10.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ro3gZLpDzpmSBCvHpyZ7pRXLo+0=', 'Content-Length': '0'} 2022-09-13T07:34:10.750 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c204a9e0784b9c2-0063203272-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.750 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.751 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.751 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.752 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:34:10.753 DEBUG:boto:Data: 2022-09-13T07:34:10.753 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.753 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:10.754 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.755 DEBUG:boto:Params: {} 2022-09-13T07:34:10.755 DEBUG:boto:Token: None 2022-09-13T07:34:10.756 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.756 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ro3gZLpDzpmSBCvHpyZ7pRXLo+0= 2022-09-13T07:34:10.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ro3gZLpDzpmSBCvHpyZ7pRXLo+0=', 'Content-Length': '0'} 2022-09-13T07:34:10.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052d056f50cb9dd5c-0063203272-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.760 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:34:10.761 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.761 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.762 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:10.762 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:10.763 DEBUG:boto:Method: GET 2022-09-13T07:34:10.764 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:10.764 DEBUG:boto:Data: 2022-09-13T07:34:10.765 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.765 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.766 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.766 DEBUG:boto:Params: {} 2022-09-13T07:34:10.767 DEBUG:boto:Token: None 2022-09-13T07:34:10.767 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/?versions 2022-09-13T07:34:10.768 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:E+OF7akFwVQChca6XwGg805mmh4= 2022-09-13T07:34:10.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:E+OF7akFwVQChca6XwGg805mmh4=', 'Content-Length': '0'} 2022-09-13T07:34:10.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006426b26d407f097a-0063203272-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.772 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.773 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:34:10.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:34:10.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:34:10.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:34:10.775 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:34:10.775 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.776 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.776 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:10.777 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:10.777 DEBUG:boto:Method: GET 2022-09-13T07:34:10.778 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:10.778 DEBUG:boto:Data: 2022-09-13T07:34:10.779 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.779 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:10.780 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.780 DEBUG:boto:Params: {} 2022-09-13T07:34:10.781 DEBUG:boto:Token: None 2022-09-13T07:34:10.781 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/?versions 2022-09-13T07:34:10.782 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:E+OF7akFwVQChca6XwGg805mmh4= 2022-09-13T07:34:10.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:E+OF7akFwVQChca6XwGg805mmh4=', 'Content-Length': '0'} 2022-09-13T07:34:10.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000be6f4ab9ff6c9ee-0063203272-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.786 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.787 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:34:10.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:34:10.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:34:10.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:34:10.789 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.789 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.790 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:10.791 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:10.791 DEBUG:boto:Method: GET 2022-09-13T07:34:10.792 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:10.792 DEBUG:boto:Data: 2022-09-13T07:34:10.793 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.793 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.794 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.794 DEBUG:boto:Params: {} 2022-09-13T07:34:10.795 DEBUG:boto:Token: None 2022-09-13T07:34:10.795 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/?versions 2022-09-13T07:34:10.796 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:E+OF7akFwVQChca6XwGg805mmh4= 2022-09-13T07:34:10.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:E+OF7akFwVQChca6XwGg805mmh4=', 'Content-Length': '0'} 2022-09-13T07:34:10.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c80e10e086b72016-0063203272-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.800 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.802 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.802 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.803 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:10.803 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:10.804 DEBUG:boto:Method: GET 2022-09-13T07:34:10.804 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:34:10.805 DEBUG:boto:Data: 2022-09-13T07:34:10.805 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.806 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:10.806 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.807 DEBUG:boto:Params: {} 2022-09-13T07:34:10.807 DEBUG:boto:Token: None 2022-09-13T07:34:10.808 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/?versions 2022-09-13T07:34:10.809 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:E+OF7akFwVQChca6XwGg805mmh4= 2022-09-13T07:34:10.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:E+OF7akFwVQChca6XwGg805mmh4=', 'Content-Length': '0'} 2022-09-13T07:34:10.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047d35102658b027e-0063203272-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.813 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:10.815 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:10.815 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:10.816 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.816 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.817 DEBUG:boto:Method: GET 2022-09-13T07:34:10.818 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.818 DEBUG:boto:Data: 2022-09-13T07:34:10.819 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.819 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.820 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.820 DEBUG:boto:Params: {} 2022-09-13T07:34:10.821 DEBUG:boto:Token: None 2022-09-13T07:34:10.822 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.822 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:59jrzCA3Y0ntfnOy/Yx8apPNn9c= 2022-09-13T07:34:10.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:59jrzCA3Y0ntfnOy/Yx8apPNn9c=', 'Content-Length': '0'} 2022-09-13T07:34:10.825 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000083fd42bd405e548f-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.826 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:10.826 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:10.827 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.827 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.828 DEBUG:boto:Method: GET 2022-09-13T07:34:10.829 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.829 DEBUG:boto:Data: 2022-09-13T07:34:10.830 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.830 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:10.831 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.831 DEBUG:boto:Params: {} 2022-09-13T07:34:10.832 DEBUG:boto:Token: None 2022-09-13T07:34:10.832 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.833 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:59jrzCA3Y0ntfnOy/Yx8apPNn9c= 2022-09-13T07:34:10.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:59jrzCA3Y0ntfnOy/Yx8apPNn9c=', 'Content-Length': '0'} 2022-09-13T07:34:10.836 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008b20d44c8422dddd-0063203272-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.837 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:10.837 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:10.838 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.838 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.839 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.839 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.840 DEBUG:boto:Data: 2022-09-13T07:34:10.840 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.840 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.841 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.841 DEBUG:boto:Params: {} 2022-09-13T07:34:10.842 DEBUG:boto:Token: None 2022-09-13T07:34:10.843 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.843 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Fqy9ZvJC3zV3oKXwjygO453/Zk0= 2022-09-13T07:34:10.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Fqy9ZvJC3zV3oKXwjygO453/Zk0=', 'Content-Length': '0'} 2022-09-13T07:34:10.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a47a718c33822c5e-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.847 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:10.847 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:10.848 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.848 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.849 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.849 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.850 DEBUG:boto:Data: 2022-09-13T07:34:10.850 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.851 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:10.851 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.852 DEBUG:boto:Params: {} 2022-09-13T07:34:10.852 DEBUG:boto:Token: None 2022-09-13T07:34:10.853 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.853 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Fqy9ZvJC3zV3oKXwjygO453/Zk0= 2022-09-13T07:34:10.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Fqy9ZvJC3zV3oKXwjygO453/Zk0=', 'Content-Length': '0'} 2022-09-13T07:34:10.856 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000282190f046df61d5-0063203272-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:10.857 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:10.858 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:10.858 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.859 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.859 DEBUG:boto:Method: GET 2022-09-13T07:34:10.860 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.860 DEBUG:boto:Data: 2022-09-13T07:34:10.861 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.862 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.862 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.863 DEBUG:boto:Params: {} 2022-09-13T07:34:10.863 DEBUG:boto:Token: None 2022-09-13T07:34:10.864 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.865 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:59jrzCA3Y0ntfnOy/Yx8apPNn9c= 2022-09-13T07:34:10.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:59jrzCA3Y0ntfnOy/Yx8apPNn9c=', 'Content-Length': '0'} 2022-09-13T07:34:10.867 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f50f6c0b3ac2eb54-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.868 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:10.869 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:10.869 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.870 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.870 DEBUG:boto:Method: GET 2022-09-13T07:34:10.871 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.871 DEBUG:boto:Data: 2022-09-13T07:34:10.872 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.872 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:10.873 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.873 DEBUG:boto:Params: {} 2022-09-13T07:34:10.874 DEBUG:boto:Token: None 2022-09-13T07:34:10.875 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.875 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:59jrzCA3Y0ntfnOy/Yx8apPNn9c= 2022-09-13T07:34:10.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:59jrzCA3Y0ntfnOy/Yx8apPNn9c=', 'Content-Length': '0'} 2022-09-13T07:34:10.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000054458b06723fe06a-0063203272-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.879 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.879 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.880 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:10.881 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:10.881 DEBUG:boto:Method: GET 2022-09-13T07:34:10.882 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:34:10.882 DEBUG:boto:Data: 2022-09-13T07:34:10.883 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.883 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.884 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.884 DEBUG:boto:Params: {} 2022-09-13T07:34:10.885 DEBUG:boto:Token: None 2022-09-13T07:34:10.885 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.886 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d584c382a88582d-0063203272-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.930 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.931 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:10.932 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:10.932 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:10.933 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:10.934 DEBUG:boto:Method: GET 2022-09-13T07:34:10.934 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:34:10.935 DEBUG:boto:Data: 2022-09-13T07:34:10.935 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.936 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:10.936 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.937 DEBUG:boto:Params: {} 2022-09-13T07:34:10.937 DEBUG:boto:Token: None 2022-09-13T07:34:10.938 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/ 2022-09-13T07:34:10.939 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms= 2022-09-13T07:34:10.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xO2wq9UsdDngU0p/azGVZlxQ+Ms=', 'Content-Length': '0'} 2022-09-13T07:34:10.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f27d864b54053f31-0063203272-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.943 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:10.944 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:10.944 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:10.945 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.946 DEBUG:boto:Path: /yxvvuh-16/dummy 2022-09-13T07:34:10.946 DEBUG:boto:Data: 2022-09-13T07:34:10.947 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.947 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.948 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.948 DEBUG:boto:Params: {} 2022-09-13T07:34:10.949 DEBUG:boto:Token: None 2022-09-13T07:34:10.950 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/dummy 2022-09-13T07:34:10.951 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jUDTI1+XBwQ+7/YJf93LWXe5xI8= 2022-09-13T07:34:10.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jUDTI1+XBwQ+7/YJf93LWXe5xI8=', 'Content-Length': '0'} 2022-09-13T07:34:10.953 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000086c6efe4424559b1-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.954 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:10.955 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:10.955 DEBUG:boto:Method: HEAD 2022-09-13T07:34:10.956 DEBUG:boto:Path: /yxvvuh-16/dummy 2022-09-13T07:34:10.957 DEBUG:boto:Data: 2022-09-13T07:34:10.957 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.958 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:10.958 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.959 DEBUG:boto:Params: {} 2022-09-13T07:34:10.959 DEBUG:boto:Token: None 2022-09-13T07:34:10.960 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/dummy 2022-09-13T07:34:10.961 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jUDTI1+XBwQ+7/YJf93LWXe5xI8= 2022-09-13T07:34:10.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jUDTI1+XBwQ+7/YJf93LWXe5xI8=', 'Content-Length': '0'} 2022-09-13T07:34:10.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d1b76bd4fb36922c-0063203272-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:10.966 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:10.967 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:10.967 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.968 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.969 DEBUG:boto:Method: GET 2022-09-13T07:34:10.969 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.970 DEBUG:boto:Data: 2022-09-13T07:34:10.970 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.971 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:10.972 DEBUG:boto:Port: 8001 2022-09-13T07:34:10.973 DEBUG:boto:Params: {} 2022-09-13T07:34:10.974 DEBUG:boto:Token: None 2022-09-13T07:34:10.975 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.976 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:59jrzCA3Y0ntfnOy/Yx8apPNn9c= 2022-09-13T07:34:10.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:59jrzCA3Y0ntfnOy/Yx8apPNn9c=', 'Content-Length': '0'} 2022-09-13T07:34:10.979 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d8fb9fe17b900abd-0063203272-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.981 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:34:10.982 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:34:10.983 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.983 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.984 DEBUG:boto:Method: GET 2022-09-13T07:34:10.985 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.986 DEBUG:boto:Data: 2022-09-13T07:34:10.987 DEBUG:boto:Headers: {} 2022-09-13T07:34:10.989 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:10.990 DEBUG:boto:Port: 8000 2022-09-13T07:34:10.991 DEBUG:boto:Params: {} 2022-09-13T07:34:10.992 DEBUG:boto:Token: None 2022-09-13T07:34:10.993 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:10 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:34:10.994 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:59jrzCA3Y0ntfnOy/Yx8apPNn9c= 2022-09-13T07:34:10.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:59jrzCA3Y0ntfnOy/Yx8apPNn9c=', 'Content-Length': '0'} 2022-09-13T07:34:10.997 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a8b5c290471ca7dd-0063203272-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:10.999 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:11.000 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:11.001 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:11.002 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:11.003 DEBUG:boto:Method: GET 2022-09-13T07:34:11.004 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:34:11.004 DEBUG:boto:Data: 2022-09-13T07:34:11.005 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.006 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.007 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.008 DEBUG:boto:Params: {} 2022-09-13T07:34:11.009 DEBUG:boto:Token: None 2022-09-13T07:34:11.010 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/ 2022-09-13T07:34:11.012 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE= 2022-09-13T07:34:11.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE=', 'Content-Length': '0'} 2022-09-13T07:34:11.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cd820e4ef4f106e-0063203273-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.017 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:11.019 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:11.019 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:11.020 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:11.022 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:34:11.022 DEBUG:boto:Method: GET 2022-09-13T07:34:11.023 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:34:11.024 DEBUG:boto:Data: 2022-09-13T07:34:11.025 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.026 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.026 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.027 DEBUG:boto:Params: {} 2022-09-13T07:34:11.028 DEBUG:boto:Token: None 2022-09-13T07:34:11.029 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/ 2022-09-13T07:34:11.030 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE= 2022-09-13T07:34:11.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE=', 'Content-Length': '0'} 2022-09-13T07:34:11.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c87e6d33267007e2-0063203273-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.034 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:11.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:11.035 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.036 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.036 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.037 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.037 DEBUG:boto:Method: GET 2022-09-13T07:34:11.038 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.038 DEBUG:boto:Data: 2022-09-13T07:34:11.039 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.039 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.040 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.041 DEBUG:boto:Params: {} 2022-09-13T07:34:11.041 DEBUG:boto:Token: None 2022-09-13T07:34:11.042 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.042 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+jjspoIMMU3bzd5wQ0WQj16Taf4= 2022-09-13T07:34:11.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+jjspoIMMU3bzd5wQ0WQj16Taf4=', 'Content-Length': '0'} 2022-09-13T07:34:11.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058c37abbb0510ffd-0063203273-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.046 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.046 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.047 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.047 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.048 DEBUG:boto:Method: GET 2022-09-13T07:34:11.048 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.049 DEBUG:boto:Data: 2022-09-13T07:34:11.050 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.050 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.051 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.051 DEBUG:boto:Params: {} 2022-09-13T07:34:11.052 DEBUG:boto:Token: None 2022-09-13T07:34:11.052 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.053 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+jjspoIMMU3bzd5wQ0WQj16Taf4= 2022-09-13T07:34:11.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+jjspoIMMU3bzd5wQ0WQj16Taf4=', 'Content-Length': '0'} 2022-09-13T07:34:11.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000984992f0ab78346a-0063203273-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.057 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.057 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.058 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.058 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.059 DEBUG:boto:Method: HEAD 2022-09-13T07:34:11.059 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.060 DEBUG:boto:Data: 2022-09-13T07:34:11.060 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.061 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.061 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.062 DEBUG:boto:Params: {} 2022-09-13T07:34:11.062 DEBUG:boto:Token: None 2022-09-13T07:34:11.063 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.063 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MbW9imBFH+9vWEKPaT8mye+BlSs= 2022-09-13T07:34:11.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MbW9imBFH+9vWEKPaT8mye+BlSs=', 'Content-Length': '0'} 2022-09-13T07:34:11.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000311e30132554542b-0063203273-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.067 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.067 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.068 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.068 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.069 DEBUG:boto:Method: HEAD 2022-09-13T07:34:11.069 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.070 DEBUG:boto:Data: 2022-09-13T07:34:11.070 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.071 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.071 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.072 DEBUG:boto:Params: {} 2022-09-13T07:34:11.072 DEBUG:boto:Token: None 2022-09-13T07:34:11.073 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.073 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MbW9imBFH+9vWEKPaT8mye+BlSs= 2022-09-13T07:34:11.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MbW9imBFH+9vWEKPaT8mye+BlSs=', 'Content-Length': '0'} 2022-09-13T07:34:11.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbaa8536817832f0-0063203273-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:11.077 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.078 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.079 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.079 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.080 DEBUG:boto:Method: GET 2022-09-13T07:34:11.080 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.081 DEBUG:boto:Data: 2022-09-13T07:34:11.081 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.082 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.082 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.083 DEBUG:boto:Params: {} 2022-09-13T07:34:11.083 DEBUG:boto:Token: None 2022-09-13T07:34:11.084 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.084 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+jjspoIMMU3bzd5wQ0WQj16Taf4= 2022-09-13T07:34:11.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+jjspoIMMU3bzd5wQ0WQj16Taf4=', 'Content-Length': '0'} 2022-09-13T07:34:11.087 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011759a1139d58f78-0063203273-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.088 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.089 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.089 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.090 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.090 DEBUG:boto:Method: GET 2022-09-13T07:34:11.091 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.091 DEBUG:boto:Data: 2022-09-13T07:34:11.092 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.092 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.093 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.093 DEBUG:boto:Params: {} 2022-09-13T07:34:11.094 DEBUG:boto:Token: None 2022-09-13T07:34:11.094 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.095 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+jjspoIMMU3bzd5wQ0WQj16Taf4= 2022-09-13T07:34:11.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+jjspoIMMU3bzd5wQ0WQj16Taf4=', 'Content-Length': '0'} 2022-09-13T07:34:11.098 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000562a9c6fb65fe75b-0063203273-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.099 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:11.099 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:11.100 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:11.100 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:11.101 DEBUG:boto:Method: GET 2022-09-13T07:34:11.101 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:11.102 DEBUG:boto:Data: 2022-09-13T07:34:11.102 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.103 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.103 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.104 DEBUG:boto:Params: {} 2022-09-13T07:34:11.104 DEBUG:boto:Token: None 2022-09-13T07:34:11.105 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/ 2022-09-13T07:34:11.105 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE= 2022-09-13T07:34:11.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE=', 'Content-Length': '0'} 2022-09-13T07:34:11.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c98882d59c6c716c-0063203273-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.110 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:11.111 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:11.111 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:11.112 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:11.112 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:11.113 DEBUG:boto:Method: GET 2022-09-13T07:34:11.113 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:11.114 DEBUG:boto:Data: 2022-09-13T07:34:11.114 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.115 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.115 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.116 DEBUG:boto:Params: {} 2022-09-13T07:34:11.116 DEBUG:boto:Token: None 2022-09-13T07:34:11.117 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/ 2022-09-13T07:34:11.118 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE= 2022-09-13T07:34:11.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE=', 'Content-Length': '0'} 2022-09-13T07:34:11.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b6bc9b99562db8b-0063203273-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.122 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:11.122 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.123 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.123 DEBUG:boto:Method: HEAD 2022-09-13T07:34:11.124 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.124 DEBUG:boto:Data: 2022-09-13T07:34:11.125 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.125 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.126 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.126 DEBUG:boto:Params: {} 2022-09-13T07:34:11.127 DEBUG:boto:Token: None 2022-09-13T07:34:11.127 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.128 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rovKvVwLduhD9jj9zFbEpLUtG/o= 2022-09-13T07:34:11.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rovKvVwLduhD9jj9zFbEpLUtG/o=', 'Content-Length': '0'} 2022-09-13T07:34:11.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a65a0cc32a923130-0063203273-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.132 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.133 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.134 DEBUG:boto:Method: HEAD 2022-09-13T07:34:11.135 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.136 DEBUG:boto:Data: 2022-09-13T07:34:11.136 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.137 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.138 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.139 DEBUG:boto:Params: {} 2022-09-13T07:34:11.140 DEBUG:boto:Token: None 2022-09-13T07:34:11.141 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.142 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rovKvVwLduhD9jj9zFbEpLUtG/o= 2022-09-13T07:34:11.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rovKvVwLduhD9jj9zFbEpLUtG/o=', 'Content-Length': '0'} 2022-09-13T07:34:11.146 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab663671596a8e4f-0063203273-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:11.147 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.148 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.148 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.149 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.149 DEBUG:boto:Method: GET 2022-09-13T07:34:11.150 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.150 DEBUG:boto:Data: 2022-09-13T07:34:11.151 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.151 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.152 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.153 DEBUG:boto:Params: {} 2022-09-13T07:34:11.153 DEBUG:boto:Token: None 2022-09-13T07:34:11.154 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.154 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+jjspoIMMU3bzd5wQ0WQj16Taf4= 2022-09-13T07:34:11.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+jjspoIMMU3bzd5wQ0WQj16Taf4=', 'Content-Length': '0'} 2022-09-13T07:34:11.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028641e22685d0a33-0063203273-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.198 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.199 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:34:11.199 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.200 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.201 DEBUG:boto:Method: GET 2022-09-13T07:34:11.201 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.202 DEBUG:boto:Data: 2022-09-13T07:34:11.202 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.203 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.203 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.204 DEBUG:boto:Params: {} 2022-09-13T07:34:11.205 DEBUG:boto:Token: None 2022-09-13T07:34:11.205 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:34:11.206 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+jjspoIMMU3bzd5wQ0WQj16Taf4= 2022-09-13T07:34:11.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+jjspoIMMU3bzd5wQ0WQj16Taf4=', 'Content-Length': '0'} 2022-09-13T07:34:11.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000602e2c496e03d2ff-0063203273-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.210 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:11.210 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:11.211 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:11.211 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:11.212 DEBUG:boto:Method: GET 2022-09-13T07:34:11.212 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:11.213 DEBUG:boto:Data: 2022-09-13T07:34:11.213 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.214 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.215 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.215 DEBUG:boto:Params: {} 2022-09-13T07:34:11.216 DEBUG:boto:Token: None 2022-09-13T07:34:11.217 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/ 2022-09-13T07:34:11.217 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE= 2022-09-13T07:34:11.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE=', 'Content-Length': '0'} 2022-09-13T07:34:11.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008181046af49016a9-0063203273-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.221 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:11.222 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:11.223 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:11.223 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:11.224 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:11.225 DEBUG:boto:Method: GET 2022-09-13T07:34:11.226 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:34:11.227 DEBUG:boto:Data: 2022-09-13T07:34:11.228 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.229 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.229 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.230 DEBUG:boto:Params: {} 2022-09-13T07:34:11.230 DEBUG:boto:Token: None 2022-09-13T07:34:11.231 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/ 2022-09-13T07:34:11.232 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE= 2022-09-13T07:34:11.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE=', 'Content-Length': '0'} 2022-09-13T07:34:11.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000842063234419e42f-0063203273-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.237 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:11.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:11.239 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.239 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.240 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.240 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.241 DEBUG:boto:Method: GET 2022-09-13T07:34:11.241 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.242 DEBUG:boto:Data: 2022-09-13T07:34:11.242 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.243 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.244 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.244 DEBUG:boto:Params: {} 2022-09-13T07:34:11.245 DEBUG:boto:Token: None 2022-09-13T07:34:11.246 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.246 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+clD4YByaeLv0SLE95P3wSNeLTQ= 2022-09-13T07:34:11.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+clD4YByaeLv0SLE95P3wSNeLTQ=', 'Content-Length': '0'} 2022-09-13T07:34:11.249 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca15952675933067-0063203273-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.250 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.250 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.251 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.252 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.252 DEBUG:boto:Method: GET 2022-09-13T07:34:11.253 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.253 DEBUG:boto:Data: 2022-09-13T07:34:11.254 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.254 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.255 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.255 DEBUG:boto:Params: {} 2022-09-13T07:34:11.256 DEBUG:boto:Token: None 2022-09-13T07:34:11.257 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.257 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+clD4YByaeLv0SLE95P3wSNeLTQ= 2022-09-13T07:34:11.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+clD4YByaeLv0SLE95P3wSNeLTQ=', 'Content-Length': '0'} 2022-09-13T07:34:11.260 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed5afb38aade4d6e-0063203273-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.261 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.262 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.262 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.263 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.265 DEBUG:boto:Method: HEAD 2022-09-13T07:34:11.266 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.267 DEBUG:boto:Data: 2022-09-13T07:34:11.267 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.268 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.268 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.269 DEBUG:boto:Params: {} 2022-09-13T07:34:11.270 DEBUG:boto:Token: None 2022-09-13T07:34:11.270 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.271 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5/xhQgRIc4Hw8NuNfA3LfV5ylm0= 2022-09-13T07:34:11.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5/xhQgRIc4Hw8NuNfA3LfV5ylm0=', 'Content-Length': '0'} 2022-09-13T07:34:11.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085dcd4e932cb1b33-0063203273-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.275 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.275 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.276 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.276 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.277 DEBUG:boto:Method: HEAD 2022-09-13T07:34:11.277 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.278 DEBUG:boto:Data: 2022-09-13T07:34:11.278 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.279 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.279 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.280 DEBUG:boto:Params: {} 2022-09-13T07:34:11.280 DEBUG:boto:Token: None 2022-09-13T07:34:11.281 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.281 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5/xhQgRIc4Hw8NuNfA3LfV5ylm0= 2022-09-13T07:34:11.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5/xhQgRIc4Hw8NuNfA3LfV5ylm0=', 'Content-Length': '0'} 2022-09-13T07:34:11.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c77e8ccf6ca63f99-0063203273-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:11.285 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.286 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.286 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.287 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.287 DEBUG:boto:Method: GET 2022-09-13T07:34:11.288 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.289 DEBUG:boto:Data: 2022-09-13T07:34:11.289 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.290 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.290 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.291 DEBUG:boto:Params: {} 2022-09-13T07:34:11.291 DEBUG:boto:Token: None 2022-09-13T07:34:11.292 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.293 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+clD4YByaeLv0SLE95P3wSNeLTQ= 2022-09-13T07:34:11.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+clD4YByaeLv0SLE95P3wSNeLTQ=', 'Content-Length': '0'} 2022-09-13T07:34:11.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ab5591565041a89-0063203273-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.296 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.297 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.297 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.298 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.298 DEBUG:boto:Method: GET 2022-09-13T07:34:11.299 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.299 DEBUG:boto:Data: 2022-09-13T07:34:11.300 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.300 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.301 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.301 DEBUG:boto:Params: {} 2022-09-13T07:34:11.302 DEBUG:boto:Token: None 2022-09-13T07:34:11.302 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.303 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+clD4YByaeLv0SLE95P3wSNeLTQ= 2022-09-13T07:34:11.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+clD4YByaeLv0SLE95P3wSNeLTQ=', 'Content-Length': '0'} 2022-09-13T07:34:11.306 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea3e94e2bc4ab361-0063203273-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.306 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:11.307 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:11.308 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:11.308 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:11.309 DEBUG:boto:Method: GET 2022-09-13T07:34:11.309 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:11.310 DEBUG:boto:Data: 2022-09-13T07:34:11.310 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.311 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.311 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.312 DEBUG:boto:Params: {} 2022-09-13T07:34:11.312 DEBUG:boto:Token: None 2022-09-13T07:34:11.313 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/ 2022-09-13T07:34:11.313 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE= 2022-09-13T07:34:11.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE=', 'Content-Length': '0'} 2022-09-13T07:34:11.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb82d13904e4692f-0063203273-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.318 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:11.319 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:11.319 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:11.320 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:11.320 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:11.321 DEBUG:boto:Method: GET 2022-09-13T07:34:11.321 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:11.322 DEBUG:boto:Data: 2022-09-13T07:34:11.322 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.323 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.323 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.324 DEBUG:boto:Params: {} 2022-09-13T07:34:11.324 DEBUG:boto:Token: None 2022-09-13T07:34:11.325 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/ 2022-09-13T07:34:11.325 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE= 2022-09-13T07:34:11.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE=', 'Content-Length': '0'} 2022-09-13T07:34:11.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069cfb8e6e1ab5c8d-0063203273-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.332 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:11.333 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.333 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.334 DEBUG:boto:Method: HEAD 2022-09-13T07:34:11.334 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.335 DEBUG:boto:Data: 2022-09-13T07:34:11.335 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.336 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.336 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.337 DEBUG:boto:Params: {} 2022-09-13T07:34:11.337 DEBUG:boto:Token: None 2022-09-13T07:34:11.338 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.339 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SAO/hZGuB+kA4zCefYfI6iTc+3s= 2022-09-13T07:34:11.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SAO/hZGuB+kA4zCefYfI6iTc+3s=', 'Content-Length': '0'} 2022-09-13T07:34:11.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3c1bd76f9f75e8f-0063203273-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.342 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.343 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.343 DEBUG:boto:Method: HEAD 2022-09-13T07:34:11.344 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.344 DEBUG:boto:Data: 2022-09-13T07:34:11.345 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.345 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.346 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.346 DEBUG:boto:Params: {} 2022-09-13T07:34:11.347 DEBUG:boto:Token: None 2022-09-13T07:34:11.347 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.348 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SAO/hZGuB+kA4zCefYfI6iTc+3s= 2022-09-13T07:34:11.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SAO/hZGuB+kA4zCefYfI6iTc+3s=', 'Content-Length': '0'} 2022-09-13T07:34:11.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047e0d0e751e3e818-0063203273-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:11.352 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.353 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.353 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.354 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.354 DEBUG:boto:Method: GET 2022-09-13T07:34:11.355 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.355 DEBUG:boto:Data: 2022-09-13T07:34:11.356 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.356 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.357 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.357 DEBUG:boto:Params: {} 2022-09-13T07:34:11.358 DEBUG:boto:Token: None 2022-09-13T07:34:11.358 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.359 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+clD4YByaeLv0SLE95P3wSNeLTQ= 2022-09-13T07:34:11.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+clD4YByaeLv0SLE95P3wSNeLTQ=', 'Content-Length': '0'} 2022-09-13T07:34:11.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d4fdc24d6bdf801-0063203273-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.362 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.363 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:34:11.364 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.364 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.365 DEBUG:boto:Method: GET 2022-09-13T07:34:11.365 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.366 DEBUG:boto:Data: 2022-09-13T07:34:11.366 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.367 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.367 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.368 DEBUG:boto:Params: {} 2022-09-13T07:34:11.369 DEBUG:boto:Token: None 2022-09-13T07:34:11.369 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:34:11.370 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+clD4YByaeLv0SLE95P3wSNeLTQ= 2022-09-13T07:34:11.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+clD4YByaeLv0SLE95P3wSNeLTQ=', 'Content-Length': '0'} 2022-09-13T07:34:11.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033b340da2c010557-0063203273-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.414 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:11.415 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:11.415 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:11.416 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:11.416 DEBUG:boto:Method: GET 2022-09-13T07:34:11.417 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:11.417 DEBUG:boto:Data: 2022-09-13T07:34:11.418 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.418 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.419 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.419 DEBUG:boto:Params: {} 2022-09-13T07:34:11.420 DEBUG:boto:Token: None 2022-09-13T07:34:11.421 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/ 2022-09-13T07:34:11.421 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE= 2022-09-13T07:34:11.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE=', 'Content-Length': '0'} 2022-09-13T07:34:11.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4877926e22a6e94-0063203273-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.425 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:11.426 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:11.426 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:11.427 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:11.427 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:11.428 DEBUG:boto:Method: GET 2022-09-13T07:34:11.428 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:34:11.429 DEBUG:boto:Data: 2022-09-13T07:34:11.429 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.430 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.430 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.431 DEBUG:boto:Params: {} 2022-09-13T07:34:11.432 DEBUG:boto:Token: None 2022-09-13T07:34:11.432 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/ 2022-09-13T07:34:11.433 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE= 2022-09-13T07:34:11.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE=', 'Content-Length': '0'} 2022-09-13T07:34:11.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000824aac6cdfd3ae0c-0063203273-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.437 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:11.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:11.438 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.438 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.439 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.439 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.440 DEBUG:boto:Method: GET 2022-09-13T07:34:11.440 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.441 DEBUG:boto:Data: 2022-09-13T07:34:11.441 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.442 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.442 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.443 DEBUG:boto:Params: {} 2022-09-13T07:34:11.444 DEBUG:boto:Token: None 2022-09-13T07:34:11.444 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.445 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MUYmvFfmKSmCeceNoq7RU+cOTP4= 2022-09-13T07:34:11.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MUYmvFfmKSmCeceNoq7RU+cOTP4=', 'Content-Length': '0'} 2022-09-13T07:34:11.448 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c18f417c41eeb8df-0063203273-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.448 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.449 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.449 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.450 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.450 DEBUG:boto:Method: GET 2022-09-13T07:34:11.451 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.451 DEBUG:boto:Data: 2022-09-13T07:34:11.452 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.452 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.453 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.453 DEBUG:boto:Params: {} 2022-09-13T07:34:11.454 DEBUG:boto:Token: None 2022-09-13T07:34:11.455 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.455 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MUYmvFfmKSmCeceNoq7RU+cOTP4= 2022-09-13T07:34:11.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MUYmvFfmKSmCeceNoq7RU+cOTP4=', 'Content-Length': '0'} 2022-09-13T07:34:11.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000832bd74d59daf558-0063203273-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.458 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.459 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.460 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.460 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.461 DEBUG:boto:Method: HEAD 2022-09-13T07:34:11.461 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.462 DEBUG:boto:Data: 2022-09-13T07:34:11.462 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.463 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.463 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.464 DEBUG:boto:Params: {} 2022-09-13T07:34:11.464 DEBUG:boto:Token: None 2022-09-13T07:34:11.465 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.465 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ErnQWzRs9HItG6iAjs/7xftakZA= 2022-09-13T07:34:11.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ErnQWzRs9HItG6iAjs/7xftakZA=', 'Content-Length': '0'} 2022-09-13T07:34:11.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090618350a2648668-0063203273-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.469 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.469 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.470 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.470 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.471 DEBUG:boto:Method: HEAD 2022-09-13T07:34:11.471 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.472 DEBUG:boto:Data: 2022-09-13T07:34:11.472 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.473 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.473 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.474 DEBUG:boto:Params: {} 2022-09-13T07:34:11.474 DEBUG:boto:Token: None 2022-09-13T07:34:11.475 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.475 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ErnQWzRs9HItG6iAjs/7xftakZA= 2022-09-13T07:34:11.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ErnQWzRs9HItG6iAjs/7xftakZA=', 'Content-Length': '0'} 2022-09-13T07:34:11.478 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005817c4c9a093a159-0063203273-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:11.479 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.480 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.480 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.481 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.482 DEBUG:boto:Method: GET 2022-09-13T07:34:11.482 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.483 DEBUG:boto:Data: 2022-09-13T07:34:11.483 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.484 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.484 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.485 DEBUG:boto:Params: {} 2022-09-13T07:34:11.485 DEBUG:boto:Token: None 2022-09-13T07:34:11.486 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.486 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MUYmvFfmKSmCeceNoq7RU+cOTP4= 2022-09-13T07:34:11.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MUYmvFfmKSmCeceNoq7RU+cOTP4=', 'Content-Length': '0'} 2022-09-13T07:34:11.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c05067263c0998bc-0063203273-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.490 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.491 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.491 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.492 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.492 DEBUG:boto:Method: GET 2022-09-13T07:34:11.493 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.493 DEBUG:boto:Data: 2022-09-13T07:34:11.494 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.494 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.495 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.495 DEBUG:boto:Params: {} 2022-09-13T07:34:11.496 DEBUG:boto:Token: None 2022-09-13T07:34:11.496 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.497 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MUYmvFfmKSmCeceNoq7RU+cOTP4= 2022-09-13T07:34:11.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MUYmvFfmKSmCeceNoq7RU+cOTP4=', 'Content-Length': '0'} 2022-09-13T07:34:11.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d92905aec3c9ae0-0063203273-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.546 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:11.547 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:11.547 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:11.548 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:11.548 DEBUG:boto:Method: GET 2022-09-13T07:34:11.549 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:11.549 DEBUG:boto:Data: 2022-09-13T07:34:11.550 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.550 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.551 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.551 DEBUG:boto:Params: {} 2022-09-13T07:34:11.552 DEBUG:boto:Token: None 2022-09-13T07:34:11.552 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/ 2022-09-13T07:34:11.553 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE= 2022-09-13T07:34:11.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE=', 'Content-Length': '0'} 2022-09-13T07:34:11.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000368794cca45663d7-0063203273-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.557 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:11.558 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:11.559 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:11.559 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:11.560 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:11.560 DEBUG:boto:Method: GET 2022-09-13T07:34:11.561 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:11.561 DEBUG:boto:Data: 2022-09-13T07:34:11.562 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.562 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.563 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.563 DEBUG:boto:Params: {} 2022-09-13T07:34:11.564 DEBUG:boto:Token: None 2022-09-13T07:34:11.565 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/ 2022-09-13T07:34:11.565 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE= 2022-09-13T07:34:11.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE=', 'Content-Length': '0'} 2022-09-13T07:34:11.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085178a4604921e4b-0063203273-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.569 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:11.570 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.570 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.571 DEBUG:boto:Method: HEAD 2022-09-13T07:34:11.571 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.572 DEBUG:boto:Data: 2022-09-13T07:34:11.573 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.573 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.574 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.574 DEBUG:boto:Params: {} 2022-09-13T07:34:11.575 DEBUG:boto:Token: None 2022-09-13T07:34:11.575 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.576 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HXuz7hQyo8LpQSil3kKhF2b4oEk= 2022-09-13T07:34:11.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HXuz7hQyo8LpQSil3kKhF2b4oEk=', 'Content-Length': '0'} 2022-09-13T07:34:11.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5e36e577537738d-0063203273-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.579 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.580 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.580 DEBUG:boto:Method: HEAD 2022-09-13T07:34:11.581 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.581 DEBUG:boto:Data: 2022-09-13T07:34:11.582 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.582 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.583 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.583 DEBUG:boto:Params: {} 2022-09-13T07:34:11.584 DEBUG:boto:Token: None 2022-09-13T07:34:11.585 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.585 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HXuz7hQyo8LpQSil3kKhF2b4oEk= 2022-09-13T07:34:11.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HXuz7hQyo8LpQSil3kKhF2b4oEk=', 'Content-Length': '0'} 2022-09-13T07:34:11.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000beb174216dbb86f9-0063203273-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:11.589 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.590 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.591 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.591 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.592 DEBUG:boto:Method: GET 2022-09-13T07:34:11.592 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.593 DEBUG:boto:Data: 2022-09-13T07:34:11.593 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.594 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.595 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.595 DEBUG:boto:Params: {} 2022-09-13T07:34:11.596 DEBUG:boto:Token: None 2022-09-13T07:34:11.596 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.597 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MUYmvFfmKSmCeceNoq7RU+cOTP4= 2022-09-13T07:34:11.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MUYmvFfmKSmCeceNoq7RU+cOTP4=', 'Content-Length': '0'} 2022-09-13T07:34:11.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045ef4b482ab2e924-0063203273-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.600 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.601 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:34:11.601 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.602 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.603 DEBUG:boto:Method: GET 2022-09-13T07:34:11.603 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.604 DEBUG:boto:Data: 2022-09-13T07:34:11.604 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.605 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.605 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.606 DEBUG:boto:Params: {} 2022-09-13T07:34:11.606 DEBUG:boto:Token: None 2022-09-13T07:34:11.607 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:34:11.608 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MUYmvFfmKSmCeceNoq7RU+cOTP4= 2022-09-13T07:34:11.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MUYmvFfmKSmCeceNoq7RU+cOTP4=', 'Content-Length': '0'} 2022-09-13T07:34:11.611 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1cee5e0457317f8-0063203273-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.612 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:11.613 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:11.613 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:11.614 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:11.614 DEBUG:boto:Method: GET 2022-09-13T07:34:11.615 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:11.616 DEBUG:boto:Data: 2022-09-13T07:34:11.616 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.617 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:11.617 DEBUG:boto:Port: 8001 2022-09-13T07:34:11.617 DEBUG:boto:Params: {} 2022-09-13T07:34:11.618 DEBUG:boto:Token: None 2022-09-13T07:34:11.619 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/ 2022-09-13T07:34:11.620 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE= 2022-09-13T07:34:11.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE=', 'Content-Length': '0'} 2022-09-13T07:34:11.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f6db8a55b35cae1-0063203273-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.625 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:11.626 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:34:11.626 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:34:11.627 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:11.628 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:11.628 DEBUG:boto:Method: GET 2022-09-13T07:34:11.629 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:34:11.629 DEBUG:boto:Data: 2022-09-13T07:34:11.630 DEBUG:boto:Headers: {} 2022-09-13T07:34:11.630 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:11.631 DEBUG:boto:Port: 8000 2022-09-13T07:34:11.631 DEBUG:boto:Params: {} 2022-09-13T07:34:11.632 DEBUG:boto:Token: None 2022-09-13T07:34:11.633 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:11 GMT /yxvvuh-16/ 2022-09-13T07:34:11.633 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE= 2022-09-13T07:34:11.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hlGdz3XtmAmWgDCE8ZvH2q8hEPE=', 'Content-Length': '0'} 2022-09-13T07:34:11.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000477707b5115b2938-0063203273-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:11.637 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:11.638 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-16 zones={test-zone3, test-zone2} 2022-09-13T07:34:11.639 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-17 2022-09-13T07:34:11.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:11.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:11.641 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:34:11.705 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:34:11.705 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:34:11.757 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:34:11.753+0000 7fcc60524600 1 bucket sync caught up with source: 2022-09-13T07:34:11.757 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000010.24.5, , 00000000003.29.9, , 00000000007.18.9, , , , , ] 2022-09-13T07:34:11.757 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000010.24.5, , 00000000003.29.9, , 00000000007.18.9, , , , , ] 2022-09-13T07:34:11.758 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:34:11.753+0000 7fcc60524600 0 bucket checkpoint complete 2022-09-13T07:34:11.779 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-17 2022-09-13T07:34:11.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:11.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:11.781 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:34:11.848 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:34:11.849 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:34:11.901 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:11.899+0000 7f7ff2718600 1 bucket sync caught up with source: 2022-09-13T07:34:11.902 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000010.24.5, , 00000000003.29.9, , 00000000007.18.9, , , , , ] 2022-09-13T07:34:11.902 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000010.24.5, , 00000000003.29.9, , 00000000007.18.9, , , , , ] 2022-09-13T07:34:11.902 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:11.899+0000 7f7ff2718600 0 bucket checkpoint complete 2022-09-13T07:34:11.917 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-17 2022-09-13T07:34:11.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:11.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:11.920 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:34:11.978 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:34:11.979 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:34:12.022 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:12.019+0000 7fe2fbf35600 1 bucket sync caught up with source: 2022-09-13T07:34:12.022 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.40.9, , 00000000010.34.5, , 00000000008.33.9, , , , , ] 2022-09-13T07:34:12.023 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.40.9, , 00000000010.34.5, , 00000000008.33.9, , , , , ] 2022-09-13T07:34:12.023 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:12.019+0000 7fe2fbf35600 0 bucket checkpoint complete 2022-09-13T07:34:12.034 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-17 2022-09-13T07:34:12.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:12.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:12.036 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:34:12.093 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:34:12.094 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:34:12.128 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:12.125+0000 7f2edf839600 1 bucket sync caught up with source: 2022-09-13T07:34:12.129 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.40.9, , 00000000010.34.5, , 00000000008.33.9, , , , , ] 2022-09-13T07:34:12.129 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.40.9, , 00000000010.34.5, , 00000000008.33.9, , , , , ] 2022-09-13T07:34:12.130 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:12.125+0000 7f2edf839600 0 bucket checkpoint complete 2022-09-13T07:34:12.140 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-17 2022-09-13T07:34:12.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:12.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:12.143 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:34:12.200 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:34:12.201 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:34:12.249 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:12.246+0000 7f6edb913600 1 bucket sync caught up with source: 2022-09-13T07:34:12.249 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.19.9, , 00000000003.21.9, , 00000000014.56.5, , , , , ] 2022-09-13T07:34:12.249 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.19.9, , 00000000003.21.9, , 00000000014.56.5, , , , , ] 2022-09-13T07:34:12.250 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:12.246+0000 7f6edb913600 0 bucket checkpoint complete 2022-09-13T07:34:12.271 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-17 2022-09-13T07:34:12.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:12.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:12.273 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:34:12.330 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:34:12.330 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:34:12.382 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:34:12.378+0000 7f304fee6600 1 bucket sync caught up with source: 2022-09-13T07:34:12.382 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000003.19.9, , 00000000003.21.9, , 00000000014.56.5, , , , , ] 2022-09-13T07:34:12.383 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000003.19.9, , 00000000003.21.9, , 00000000014.56.5, , , , , ] 2022-09-13T07:34:12.383 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:34:12.378+0000 7f304fee6600 0 bucket checkpoint complete 2022-09-13T07:34:12.393 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-17 zones={test-zone2, test-zone1} 2022-09-13T07:34:12.394 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:12.395 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:12.395 DEBUG:boto:Method: HEAD 2022-09-13T07:34:12.396 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:34:12.396 DEBUG:boto:Data: 2022-09-13T07:34:12.397 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.397 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.398 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.398 DEBUG:boto:Params: {} 2022-09-13T07:34:12.399 DEBUG:boto:Token: None 2022-09-13T07:34:12.399 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/ 2022-09-13T07:34:12.400 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TUp31Z+8QExj36kOzjfqbf+gKhQ= 2022-09-13T07:34:12.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TUp31Z+8QExj36kOzjfqbf+gKhQ=', 'Content-Length': '0'} 2022-09-13T07:34:12.403 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005262b33ad2405b28-0063203274-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.403 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:12.404 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:12.405 DEBUG:boto:Method: HEAD 2022-09-13T07:34:12.405 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:34:12.406 DEBUG:boto:Data: 2022-09-13T07:34:12.406 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.407 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.407 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.407 DEBUG:boto:Params: {} 2022-09-13T07:34:12.408 DEBUG:boto:Token: None 2022-09-13T07:34:12.409 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/ 2022-09-13T07:34:12.409 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TUp31Z+8QExj36kOzjfqbf+gKhQ= 2022-09-13T07:34:12.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TUp31Z+8QExj36kOzjfqbf+gKhQ=', 'Content-Length': '0'} 2022-09-13T07:34:12.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fffde98a6deb3a40-0063203274-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.413 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:34:12.414 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:12.414 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:12.415 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:12.415 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:12.416 DEBUG:boto:Method: GET 2022-09-13T07:34:12.416 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:12.417 DEBUG:boto:Data: 2022-09-13T07:34:12.417 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.418 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.418 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.419 DEBUG:boto:Params: {} 2022-09-13T07:34:12.419 DEBUG:boto:Token: None 2022-09-13T07:34:12.420 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/?versions 2022-09-13T07:34:12.420 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kMjesFemMQQsnXAeOnd3vtHSkIQ= 2022-09-13T07:34:12.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kMjesFemMQQsnXAeOnd3vtHSkIQ=', 'Content-Length': '0'} 2022-09-13T07:34:12.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000366c79dabdb7056d-0063203274-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.424 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:12.425 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:34:12.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:34:12.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:34:12.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:34:12.427 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:34:12.427 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:12.428 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:12.428 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:12.429 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:12.429 DEBUG:boto:Method: GET 2022-09-13T07:34:12.430 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:12.430 DEBUG:boto:Data: 2022-09-13T07:34:12.431 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.432 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.432 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.433 DEBUG:boto:Params: {} 2022-09-13T07:34:12.433 DEBUG:boto:Token: None 2022-09-13T07:34:12.434 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/?versions 2022-09-13T07:34:12.434 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kMjesFemMQQsnXAeOnd3vtHSkIQ= 2022-09-13T07:34:12.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kMjesFemMQQsnXAeOnd3vtHSkIQ=', 'Content-Length': '0'} 2022-09-13T07:34:12.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa9ea67b81100e66-0063203274-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.438 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:12.439 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:34:12.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:34:12.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:34:12.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:34:12.441 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:12.441 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:12.442 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:12.442 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:12.443 DEBUG:boto:Method: GET 2022-09-13T07:34:12.443 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:12.444 DEBUG:boto:Data: 2022-09-13T07:34:12.444 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.445 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.445 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.446 DEBUG:boto:Params: {} 2022-09-13T07:34:12.446 DEBUG:boto:Token: None 2022-09-13T07:34:12.447 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/?versions 2022-09-13T07:34:12.447 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kMjesFemMQQsnXAeOnd3vtHSkIQ= 2022-09-13T07:34:12.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kMjesFemMQQsnXAeOnd3vtHSkIQ=', 'Content-Length': '0'} 2022-09-13T07:34:12.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e210e438fdaedd7f-0063203274-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.451 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:12.452 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:12.452 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:12.453 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:12.453 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:12.454 DEBUG:boto:Method: GET 2022-09-13T07:34:12.454 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:12.455 DEBUG:boto:Data: 2022-09-13T07:34:12.455 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.456 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.456 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.457 DEBUG:boto:Params: {} 2022-09-13T07:34:12.457 DEBUG:boto:Token: None 2022-09-13T07:34:12.458 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/?versions 2022-09-13T07:34:12.459 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kMjesFemMQQsnXAeOnd3vtHSkIQ= 2022-09-13T07:34:12.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kMjesFemMQQsnXAeOnd3vtHSkIQ=', 'Content-Length': '0'} 2022-09-13T07:34:12.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000904738e826b78a98-0063203274-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.462 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:12.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:12.463 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:12.464 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:12.464 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.465 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.465 DEBUG:boto:Method: GET 2022-09-13T07:34:12.466 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.466 DEBUG:boto:Data: 2022-09-13T07:34:12.467 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.467 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.468 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.468 DEBUG:boto:Params: {} 2022-09-13T07:34:12.469 DEBUG:boto:Token: None 2022-09-13T07:34:12.470 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.470 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jeru696SlyX/Rnx4VTHhvOK+mTU= 2022-09-13T07:34:12.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jeru696SlyX/Rnx4VTHhvOK+mTU=', 'Content-Length': '0'} 2022-09-13T07:34:12.473 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000062aff85392f2301b-0063203274-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.473 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:12.474 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:12.474 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.475 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.476 DEBUG:boto:Method: GET 2022-09-13T07:34:12.476 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.477 DEBUG:boto:Data: 2022-09-13T07:34:12.477 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.478 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.479 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.479 DEBUG:boto:Params: {} 2022-09-13T07:34:12.480 DEBUG:boto:Token: None 2022-09-13T07:34:12.480 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.481 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jeru696SlyX/Rnx4VTHhvOK+mTU= 2022-09-13T07:34:12.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jeru696SlyX/Rnx4VTHhvOK+mTU=', 'Content-Length': '0'} 2022-09-13T07:34:12.483 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008a87ffdb233f5ff1-0063203274-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.484 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:12.485 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:12.485 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.486 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.486 DEBUG:boto:Method: HEAD 2022-09-13T07:34:12.487 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.487 DEBUG:boto:Data: 2022-09-13T07:34:12.488 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.488 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.489 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.489 DEBUG:boto:Params: {} 2022-09-13T07:34:12.490 DEBUG:boto:Token: None 2022-09-13T07:34:12.491 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.491 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QY34drFQEzkdm+nN8OTqsygiulk= 2022-09-13T07:34:12.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QY34drFQEzkdm+nN8OTqsygiulk=', 'Content-Length': '0'} 2022-09-13T07:34:12.493 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000051b9e540c00de237-0063203274-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.494 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:12.495 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:12.495 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.495 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.496 DEBUG:boto:Method: HEAD 2022-09-13T07:34:12.496 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.497 DEBUG:boto:Data: 2022-09-13T07:34:12.497 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.498 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.498 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.499 DEBUG:boto:Params: {} 2022-09-13T07:34:12.500 DEBUG:boto:Token: None 2022-09-13T07:34:12.500 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.501 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QY34drFQEzkdm+nN8OTqsygiulk= 2022-09-13T07:34:12.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QY34drFQEzkdm+nN8OTqsygiulk=', 'Content-Length': '0'} 2022-09-13T07:34:12.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000923f90f8cd93801-0063203274-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:12.506 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:12.507 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:12.507 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.508 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.508 DEBUG:boto:Method: GET 2022-09-13T07:34:12.509 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.509 DEBUG:boto:Data: 2022-09-13T07:34:12.510 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.510 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.511 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.511 DEBUG:boto:Params: {} 2022-09-13T07:34:12.512 DEBUG:boto:Token: None 2022-09-13T07:34:12.512 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.513 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jeru696SlyX/Rnx4VTHhvOK+mTU= 2022-09-13T07:34:12.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jeru696SlyX/Rnx4VTHhvOK+mTU=', 'Content-Length': '0'} 2022-09-13T07:34:12.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d2c1a0908125c3f0-0063203274-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.516 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:12.517 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:12.517 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.518 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.518 DEBUG:boto:Method: GET 2022-09-13T07:34:12.519 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.519 DEBUG:boto:Data: 2022-09-13T07:34:12.520 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.521 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.521 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.522 DEBUG:boto:Params: {} 2022-09-13T07:34:12.522 DEBUG:boto:Token: None 2022-09-13T07:34:12.523 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.523 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jeru696SlyX/Rnx4VTHhvOK+mTU= 2022-09-13T07:34:12.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jeru696SlyX/Rnx4VTHhvOK+mTU=', 'Content-Length': '0'} 2022-09-13T07:34:12.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2cd3a34141f29ce-0063203274-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.527 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:12.528 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:12.529 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:12.530 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:12.531 DEBUG:boto:Method: GET 2022-09-13T07:34:12.532 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:34:12.532 DEBUG:boto:Data: 2022-09-13T07:34:12.533 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.534 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.535 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.536 DEBUG:boto:Params: {} 2022-09-13T07:34:12.536 DEBUG:boto:Token: None 2022-09-13T07:34:12.537 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/ 2022-09-13T07:34:12.538 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY= 2022-09-13T07:34:12.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY=', 'Content-Length': '0'} 2022-09-13T07:34:12.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098e5e1e6d50eea26-0063203274-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.590 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:12.591 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:12.592 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:12.592 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:12.593 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:12.594 DEBUG:boto:Method: GET 2022-09-13T07:34:12.595 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:34:12.596 DEBUG:boto:Data: 2022-09-13T07:34:12.596 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.597 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.597 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.598 DEBUG:boto:Params: {} 2022-09-13T07:34:12.598 DEBUG:boto:Token: None 2022-09-13T07:34:12.599 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/ 2022-09-13T07:34:12.600 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY= 2022-09-13T07:34:12.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY=', 'Content-Length': '0'} 2022-09-13T07:34:12.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e773d7072e906fa-0063203274-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.604 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:12.604 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:12.605 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:12.606 DEBUG:boto:Method: HEAD 2022-09-13T07:34:12.606 DEBUG:boto:Path: /yxvvuh-17/dummy 2022-09-13T07:34:12.607 DEBUG:boto:Data: 2022-09-13T07:34:12.607 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.608 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.608 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.608 DEBUG:boto:Params: {} 2022-09-13T07:34:12.609 DEBUG:boto:Token: None 2022-09-13T07:34:12.610 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/dummy 2022-09-13T07:34:12.610 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:JARr1xgAn9pihHXDrNs4lAp4pj8= 2022-09-13T07:34:12.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JARr1xgAn9pihHXDrNs4lAp4pj8=', 'Content-Length': '0'} 2022-09-13T07:34:12.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c89e8f04a1080312-0063203274-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.613 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:12.614 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:12.614 DEBUG:boto:Method: HEAD 2022-09-13T07:34:12.615 DEBUG:boto:Path: /yxvvuh-17/dummy 2022-09-13T07:34:12.616 DEBUG:boto:Data: 2022-09-13T07:34:12.616 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.617 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.617 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.618 DEBUG:boto:Params: {} 2022-09-13T07:34:12.618 DEBUG:boto:Token: None 2022-09-13T07:34:12.619 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/dummy 2022-09-13T07:34:12.620 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:JARr1xgAn9pihHXDrNs4lAp4pj8= 2022-09-13T07:34:12.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JARr1xgAn9pihHXDrNs4lAp4pj8=', 'Content-Length': '0'} 2022-09-13T07:34:12.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8568f9d7486dc7b-0063203274-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:12.624 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:12.625 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:12.625 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.626 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.626 DEBUG:boto:Method: GET 2022-09-13T07:34:12.627 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.627 DEBUG:boto:Data: 2022-09-13T07:34:12.628 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.628 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.629 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.629 DEBUG:boto:Params: {} 2022-09-13T07:34:12.630 DEBUG:boto:Token: None 2022-09-13T07:34:12.631 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.631 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jeru696SlyX/Rnx4VTHhvOK+mTU= 2022-09-13T07:34:12.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jeru696SlyX/Rnx4VTHhvOK+mTU=', 'Content-Length': '0'} 2022-09-13T07:34:12.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000da5bfd06b115297e-0063203274-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.634 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:12.635 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:12.635 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.636 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.636 DEBUG:boto:Method: GET 2022-09-13T07:34:12.637 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.637 DEBUG:boto:Data: 2022-09-13T07:34:12.638 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.639 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.639 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.640 DEBUG:boto:Params: {} 2022-09-13T07:34:12.640 DEBUG:boto:Token: None 2022-09-13T07:34:12.641 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:12.641 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jeru696SlyX/Rnx4VTHhvOK+mTU= 2022-09-13T07:34:12.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jeru696SlyX/Rnx4VTHhvOK+mTU=', 'Content-Length': '0'} 2022-09-13T07:34:12.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000054159840db28a2a2-0063203274-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.645 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:12.646 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:12.647 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:12.648 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:12.649 DEBUG:boto:Method: GET 2022-09-13T07:34:12.649 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:34:12.650 DEBUG:boto:Data: 2022-09-13T07:34:12.651 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.652 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.653 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.654 DEBUG:boto:Params: {} 2022-09-13T07:34:12.655 DEBUG:boto:Token: None 2022-09-13T07:34:12.656 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/ 2022-09-13T07:34:12.657 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY= 2022-09-13T07:34:12.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY=', 'Content-Length': '0'} 2022-09-13T07:34:12.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092144e699e010860-0063203274-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.661 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:12.662 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:12.663 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:12.663 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:12.664 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:12.664 DEBUG:boto:Method: GET 2022-09-13T07:34:12.665 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:34:12.665 DEBUG:boto:Data: 2022-09-13T07:34:12.666 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.666 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.667 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.668 DEBUG:boto:Params: {} 2022-09-13T07:34:12.669 DEBUG:boto:Token: None 2022-09-13T07:34:12.670 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/ 2022-09-13T07:34:12.671 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY= 2022-09-13T07:34:12.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY=', 'Content-Length': '0'} 2022-09-13T07:34:12.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca41855ffe340db3-0063203274-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.677 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:12.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:12.679 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.680 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.681 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.682 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.683 DEBUG:boto:Method: GET 2022-09-13T07:34:12.684 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.685 DEBUG:boto:Data: 2022-09-13T07:34:12.686 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.687 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.688 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.688 DEBUG:boto:Params: {} 2022-09-13T07:34:12.690 DEBUG:boto:Token: None 2022-09-13T07:34:12.691 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.692 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4r5LhHdZ8kyL8KeHz8L0NDMz8O8= 2022-09-13T07:34:12.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4r5LhHdZ8kyL8KeHz8L0NDMz8O8=', 'Content-Length': '0'} 2022-09-13T07:34:12.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e6e388d72cd84bc-0063203274-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.697 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.698 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.699 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.700 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.701 DEBUG:boto:Method: GET 2022-09-13T07:34:12.702 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.703 DEBUG:boto:Data: 2022-09-13T07:34:12.704 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.704 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.705 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.706 DEBUG:boto:Params: {} 2022-09-13T07:34:12.707 DEBUG:boto:Token: None 2022-09-13T07:34:12.708 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.709 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4r5LhHdZ8kyL8KeHz8L0NDMz8O8= 2022-09-13T07:34:12.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4r5LhHdZ8kyL8KeHz8L0NDMz8O8=', 'Content-Length': '0'} 2022-09-13T07:34:12.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003613c733c0a975b-0063203274-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.714 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.715 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.716 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.717 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.717 DEBUG:boto:Method: HEAD 2022-09-13T07:34:12.718 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.719 DEBUG:boto:Data: 2022-09-13T07:34:12.719 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.720 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.720 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.721 DEBUG:boto:Params: {} 2022-09-13T07:34:12.721 DEBUG:boto:Token: None 2022-09-13T07:34:12.722 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.723 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LAvtWkS3jNwmjyNy4oEg6gkEgj8= 2022-09-13T07:34:12.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LAvtWkS3jNwmjyNy4oEg6gkEgj8=', 'Content-Length': '0'} 2022-09-13T07:34:12.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7529ef9253f1e63-0063203274-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.725 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.726 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.727 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.727 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.727 DEBUG:boto:Method: HEAD 2022-09-13T07:34:12.728 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.728 DEBUG:boto:Data: 2022-09-13T07:34:12.729 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.730 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.730 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.731 DEBUG:boto:Params: {} 2022-09-13T07:34:12.731 DEBUG:boto:Token: None 2022-09-13T07:34:12.732 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.732 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LAvtWkS3jNwmjyNy4oEg6gkEgj8= 2022-09-13T07:34:12.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LAvtWkS3jNwmjyNy4oEg6gkEgj8=', 'Content-Length': '0'} 2022-09-13T07:34:12.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002182e5572b3275d1-0063203274-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:12.736 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.737 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.737 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.738 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.738 DEBUG:boto:Method: GET 2022-09-13T07:34:12.739 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.739 DEBUG:boto:Data: 2022-09-13T07:34:12.740 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.740 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.741 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.741 DEBUG:boto:Params: {} 2022-09-13T07:34:12.742 DEBUG:boto:Token: None 2022-09-13T07:34:12.742 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.743 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4r5LhHdZ8kyL8KeHz8L0NDMz8O8= 2022-09-13T07:34:12.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4r5LhHdZ8kyL8KeHz8L0NDMz8O8=', 'Content-Length': '0'} 2022-09-13T07:34:12.745 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000379574f58c0e2bc6-0063203274-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.746 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.747 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.747 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.748 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.748 DEBUG:boto:Method: GET 2022-09-13T07:34:12.749 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.749 DEBUG:boto:Data: 2022-09-13T07:34:12.750 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.751 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.752 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.752 DEBUG:boto:Params: {} 2022-09-13T07:34:12.753 DEBUG:boto:Token: None 2022-09-13T07:34:12.753 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.754 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4r5LhHdZ8kyL8KeHz8L0NDMz8O8= 2022-09-13T07:34:12.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4r5LhHdZ8kyL8KeHz8L0NDMz8O8=', 'Content-Length': '0'} 2022-09-13T07:34:12.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4ce7689ea5aaf22-0063203274-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.798 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:12.799 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:12.800 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:12.801 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:12.802 DEBUG:boto:Method: GET 2022-09-13T07:34:12.802 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:12.803 DEBUG:boto:Data: 2022-09-13T07:34:12.804 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.804 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.805 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.805 DEBUG:boto:Params: {} 2022-09-13T07:34:12.806 DEBUG:boto:Token: None 2022-09-13T07:34:12.806 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/ 2022-09-13T07:34:12.807 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY= 2022-09-13T07:34:12.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY=', 'Content-Length': '0'} 2022-09-13T07:34:12.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000101384be2291eb0f-0063203274-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.811 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:12.812 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:12.813 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:12.813 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:12.814 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:12.815 DEBUG:boto:Method: GET 2022-09-13T07:34:12.816 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:12.817 DEBUG:boto:Data: 2022-09-13T07:34:12.817 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.818 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.818 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.819 DEBUG:boto:Params: {} 2022-09-13T07:34:12.819 DEBUG:boto:Token: None 2022-09-13T07:34:12.820 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/ 2022-09-13T07:34:12.821 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY= 2022-09-13T07:34:12.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY=', 'Content-Length': '0'} 2022-09-13T07:34:12.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cf84182e949583b-0063203274-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.826 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:12.827 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.827 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.828 DEBUG:boto:Method: HEAD 2022-09-13T07:34:12.828 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.829 DEBUG:boto:Data: 2022-09-13T07:34:12.829 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.830 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.830 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.831 DEBUG:boto:Params: {} 2022-09-13T07:34:12.832 DEBUG:boto:Token: None 2022-09-13T07:34:12.832 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.833 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GXUhHQOoJF8HyM0sEcY7ENb1QLc= 2022-09-13T07:34:12.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GXUhHQOoJF8HyM0sEcY7ENb1QLc=', 'Content-Length': '0'} 2022-09-13T07:34:12.836 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006dbb3b488776f541-0063203274-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.837 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.838 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.839 DEBUG:boto:Method: HEAD 2022-09-13T07:34:12.839 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.840 DEBUG:boto:Data: 2022-09-13T07:34:12.840 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.841 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.841 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.842 DEBUG:boto:Params: {} 2022-09-13T07:34:12.843 DEBUG:boto:Token: None 2022-09-13T07:34:12.843 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.844 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GXUhHQOoJF8HyM0sEcY7ENb1QLc= 2022-09-13T07:34:12.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GXUhHQOoJF8HyM0sEcY7ENb1QLc=', 'Content-Length': '0'} 2022-09-13T07:34:12.847 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb8e5edc11b4cd2a-0063203274-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:12.849 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.849 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.850 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.850 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.851 DEBUG:boto:Method: GET 2022-09-13T07:34:12.852 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.852 DEBUG:boto:Data: 2022-09-13T07:34:12.853 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.853 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.854 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.855 DEBUG:boto:Params: {} 2022-09-13T07:34:12.856 DEBUG:boto:Token: None 2022-09-13T07:34:12.857 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.858 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4r5LhHdZ8kyL8KeHz8L0NDMz8O8= 2022-09-13T07:34:12.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4r5LhHdZ8kyL8KeHz8L0NDMz8O8=', 'Content-Length': '0'} 2022-09-13T07:34:12.861 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004154a1a931317f75-0063203274-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.862 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.863 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:12.864 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.864 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.865 DEBUG:boto:Method: GET 2022-09-13T07:34:12.866 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.866 DEBUG:boto:Data: 2022-09-13T07:34:12.867 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.867 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.868 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.868 DEBUG:boto:Params: {} 2022-09-13T07:34:12.869 DEBUG:boto:Token: None 2022-09-13T07:34:12.870 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:12.871 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4r5LhHdZ8kyL8KeHz8L0NDMz8O8= 2022-09-13T07:34:12.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4r5LhHdZ8kyL8KeHz8L0NDMz8O8=', 'Content-Length': '0'} 2022-09-13T07:34:12.873 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6099d0f4a29c104-0063203274-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.874 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:12.875 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:12.875 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:12.876 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:12.877 DEBUG:boto:Method: GET 2022-09-13T07:34:12.877 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:12.878 DEBUG:boto:Data: 2022-09-13T07:34:12.879 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.880 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.880 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.881 DEBUG:boto:Params: {} 2022-09-13T07:34:12.881 DEBUG:boto:Token: None 2022-09-13T07:34:12.882 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/ 2022-09-13T07:34:12.883 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY= 2022-09-13T07:34:12.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY=', 'Content-Length': '0'} 2022-09-13T07:34:12.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071441d3b18a7607a-0063203274-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.887 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:12.888 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:12.888 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:12.889 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:12.889 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:12.890 DEBUG:boto:Method: GET 2022-09-13T07:34:12.890 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:12.891 DEBUG:boto:Data: 2022-09-13T07:34:12.891 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.892 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.892 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.893 DEBUG:boto:Params: {} 2022-09-13T07:34:12.894 DEBUG:boto:Token: None 2022-09-13T07:34:12.894 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/ 2022-09-13T07:34:12.895 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY= 2022-09-13T07:34:12.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/H8iEyqHv1161RAg9SzvaSia/tY=', 'Content-Length': '0'} 2022-09-13T07:34:12.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e24f3a99f5fd44e-0063203274-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.899 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:12.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:12.900 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:12.901 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:12.901 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.902 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.902 DEBUG:boto:Method: GET 2022-09-13T07:34:12.903 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.903 DEBUG:boto:Data: 2022-09-13T07:34:12.904 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.904 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.905 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.905 DEBUG:boto:Params: {} 2022-09-13T07:34:12.906 DEBUG:boto:Token: None 2022-09-13T07:34:12.906 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.907 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mf9iBOdt5jdjhxEYUEASeYrOGYI= 2022-09-13T07:34:12.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mf9iBOdt5jdjhxEYUEASeYrOGYI=', 'Content-Length': '0'} 2022-09-13T07:34:12.909 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f01608902cc3f206-0063203274-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.950 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:12.951 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:12.952 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.953 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.953 DEBUG:boto:Method: GET 2022-09-13T07:34:12.954 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.954 DEBUG:boto:Data: 2022-09-13T07:34:12.955 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.955 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.956 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.957 DEBUG:boto:Params: {} 2022-09-13T07:34:12.957 DEBUG:boto:Token: None 2022-09-13T07:34:12.958 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.958 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mf9iBOdt5jdjhxEYUEASeYrOGYI= 2022-09-13T07:34:12.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mf9iBOdt5jdjhxEYUEASeYrOGYI=', 'Content-Length': '0'} 2022-09-13T07:34:12.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082abe1e3ba4907d6-0063203274-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.962 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:12.963 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:12.963 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.964 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.964 DEBUG:boto:Method: HEAD 2022-09-13T07:34:12.965 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.965 DEBUG:boto:Data: 2022-09-13T07:34:12.966 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.966 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.967 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.967 DEBUG:boto:Params: {} 2022-09-13T07:34:12.968 DEBUG:boto:Token: None 2022-09-13T07:34:12.969 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.969 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RVbw6iRhHLbygmextX5GqjqhhZY= 2022-09-13T07:34:12.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RVbw6iRhHLbygmextX5GqjqhhZY=', 'Content-Length': '0'} 2022-09-13T07:34:12.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002646fd76643950aa-0063203274-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.973 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:12.973 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:12.974 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.974 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.975 DEBUG:boto:Method: HEAD 2022-09-13T07:34:12.975 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.976 DEBUG:boto:Data: 2022-09-13T07:34:12.976 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.977 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:12.977 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.978 DEBUG:boto:Params: {} 2022-09-13T07:34:12.978 DEBUG:boto:Token: None 2022-09-13T07:34:12.979 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.979 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RVbw6iRhHLbygmextX5GqjqhhZY= 2022-09-13T07:34:12.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RVbw6iRhHLbygmextX5GqjqhhZY=', 'Content-Length': '0'} 2022-09-13T07:34:12.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a220879b82ea6c9-0063203274-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:12.983 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:12.984 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:12.986 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.986 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.987 DEBUG:boto:Method: GET 2022-09-13T07:34:12.988 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.988 DEBUG:boto:Data: 2022-09-13T07:34:12.989 DEBUG:boto:Headers: {} 2022-09-13T07:34:12.989 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:12.990 DEBUG:boto:Port: 8000 2022-09-13T07:34:12.990 DEBUG:boto:Params: {} 2022-09-13T07:34:12.991 DEBUG:boto:Token: None 2022-09-13T07:34:12.991 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:12 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.992 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mf9iBOdt5jdjhxEYUEASeYrOGYI= 2022-09-13T07:34:12.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mf9iBOdt5jdjhxEYUEASeYrOGYI=', 'Content-Length': '0'} 2022-09-13T07:34:12.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001a7c817d5e420a3-0063203274-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:12.995 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:12.996 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:12.996 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.997 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.997 DEBUG:boto:Method: GET 2022-09-13T07:34:12.998 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:12.998 DEBUG:boto:Data: 2022-09-13T07:34:12.999 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.000 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.000 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.001 DEBUG:boto:Params: {} 2022-09-13T07:34:13.001 DEBUG:boto:Token: None 2022-09-13T07:34:13.002 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.002 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NaRYHT9B2PvUGaJIzSdH26diy7I= 2022-09-13T07:34:13.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NaRYHT9B2PvUGaJIzSdH26diy7I=', 'Content-Length': '0'} 2022-09-13T07:34:13.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca553f699c1ab106-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.046 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.047 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.047 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.048 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.048 DEBUG:boto:Method: GET 2022-09-13T07:34:13.049 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.049 DEBUG:boto:Data: 2022-09-13T07:34:13.050 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.050 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:13.051 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.052 DEBUG:boto:Params: {} 2022-09-13T07:34:13.052 DEBUG:boto:Token: None 2022-09-13T07:34:13.053 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.054 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079afcebc239dfd0d-0063203275-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.057 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.058 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.059 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.059 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.060 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.060 DEBUG:boto:Method: GET 2022-09-13T07:34:13.061 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.061 DEBUG:boto:Data: 2022-09-13T07:34:13.062 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.062 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.063 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.063 DEBUG:boto:Params: {} 2022-09-13T07:34:13.064 DEBUG:boto:Token: None 2022-09-13T07:34:13.064 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.065 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002c3049c17bb6d93-0063203275-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.069 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.070 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.070 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.071 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.071 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.072 DEBUG:boto:Data: 2022-09-13T07:34:13.072 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.073 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:13.073 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.074 DEBUG:boto:Params: {} 2022-09-13T07:34:13.074 DEBUG:boto:Token: None 2022-09-13T07:34:13.075 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.075 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:glHF2+c+svDET/AUcSvUF+AT7I4= 2022-09-13T07:34:13.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:glHF2+c+svDET/AUcSvUF+AT7I4=', 'Content-Length': '0'} 2022-09-13T07:34:13.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000020b11ee41603c8a-0063203275-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.080 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.081 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.082 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.083 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.083 DEBUG:boto:Data: 2022-09-13T07:34:13.084 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.084 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.085 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.085 DEBUG:boto:Params: {} 2022-09-13T07:34:13.086 DEBUG:boto:Token: None 2022-09-13T07:34:13.086 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.087 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:glHF2+c+svDET/AUcSvUF+AT7I4= 2022-09-13T07:34:13.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:glHF2+c+svDET/AUcSvUF+AT7I4=', 'Content-Length': '0'} 2022-09-13T07:34:13.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000768e0dd7762708a4-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:13.092 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.093 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.093 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.094 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.094 DEBUG:boto:Method: GET 2022-09-13T07:34:13.095 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.095 DEBUG:boto:Data: 2022-09-13T07:34:13.096 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.096 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:13.097 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.098 DEBUG:boto:Params: {} 2022-09-13T07:34:13.098 DEBUG:boto:Token: None 2022-09-13T07:34:13.099 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.100 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NaRYHT9B2PvUGaJIzSdH26diy7I= 2022-09-13T07:34:13.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NaRYHT9B2PvUGaJIzSdH26diy7I=', 'Content-Length': '0'} 2022-09-13T07:34:13.104 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024cbae245cfd6b71-0063203275-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.105 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.106 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.106 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.107 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.107 DEBUG:boto:Method: GET 2022-09-13T07:34:13.108 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.108 DEBUG:boto:Data: 2022-09-13T07:34:13.109 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.110 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.111 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.111 DEBUG:boto:Params: {} 2022-09-13T07:34:13.112 DEBUG:boto:Token: None 2022-09-13T07:34:13.113 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.113 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NaRYHT9B2PvUGaJIzSdH26diy7I= 2022-09-13T07:34:13.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NaRYHT9B2PvUGaJIzSdH26diy7I=', 'Content-Length': '0'} 2022-09-13T07:34:13.116 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1dd4f3ea988a732-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.117 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.117 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.118 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.119 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.119 DEBUG:boto:Method: GET 2022-09-13T07:34:13.120 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.120 DEBUG:boto:Data: 2022-09-13T07:34:13.121 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.121 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:13.122 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.122 DEBUG:boto:Params: {} 2022-09-13T07:34:13.123 DEBUG:boto:Token: None 2022-09-13T07:34:13.123 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.124 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d15fd676908fef7-0063203275-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.127 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.128 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.129 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.129 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.130 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.130 DEBUG:boto:Method: GET 2022-09-13T07:34:13.131 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.131 DEBUG:boto:Data: 2022-09-13T07:34:13.132 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.132 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.133 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.133 DEBUG:boto:Params: {} 2022-09-13T07:34:13.134 DEBUG:boto:Token: None 2022-09-13T07:34:13.134 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.135 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d520313114954de-0063203275-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.139 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:13.140 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.141 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.141 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.142 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.142 DEBUG:boto:Method: GET 2022-09-13T07:34:13.143 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.143 DEBUG:boto:Data: 2022-09-13T07:34:13.144 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.144 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:13.145 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.145 DEBUG:boto:Params: {} 2022-09-13T07:34:13.146 DEBUG:boto:Token: None 2022-09-13T07:34:13.146 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.147 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t3npxCE4m0ndfX8hTufJ1GuTf/s= 2022-09-13T07:34:13.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t3npxCE4m0ndfX8hTufJ1GuTf/s=', 'Content-Length': '0'} 2022-09-13T07:34:13.149 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecc69cf893c04e56-0063203275-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.190 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.191 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.191 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.192 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.192 DEBUG:boto:Method: GET 2022-09-13T07:34:13.193 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.193 DEBUG:boto:Data: 2022-09-13T07:34:13.194 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.195 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.195 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.196 DEBUG:boto:Params: {} 2022-09-13T07:34:13.196 DEBUG:boto:Token: None 2022-09-13T07:34:13.197 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.197 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t3npxCE4m0ndfX8hTufJ1GuTf/s= 2022-09-13T07:34:13.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t3npxCE4m0ndfX8hTufJ1GuTf/s=', 'Content-Length': '0'} 2022-09-13T07:34:13.200 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000232088cccd68d7f0-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.201 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.201 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.202 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.202 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.203 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.203 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.204 DEBUG:boto:Data: 2022-09-13T07:34:13.204 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.205 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:13.205 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.206 DEBUG:boto:Params: {} 2022-09-13T07:34:13.206 DEBUG:boto:Token: None 2022-09-13T07:34:13.207 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.208 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8QVCJmr+H63hDTHsRWxOoz4EjMM= 2022-09-13T07:34:13.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8QVCJmr+H63hDTHsRWxOoz4EjMM=', 'Content-Length': '0'} 2022-09-13T07:34:13.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047d6157450dddfcf-0063203275-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.211 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.211 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.212 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.212 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.213 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.213 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.214 DEBUG:boto:Data: 2022-09-13T07:34:13.214 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.215 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.215 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.216 DEBUG:boto:Params: {} 2022-09-13T07:34:13.216 DEBUG:boto:Token: None 2022-09-13T07:34:13.217 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.217 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8QVCJmr+H63hDTHsRWxOoz4EjMM= 2022-09-13T07:34:13.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8QVCJmr+H63hDTHsRWxOoz4EjMM=', 'Content-Length': '0'} 2022-09-13T07:34:13.220 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ec8f7dadced6997-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:13.222 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.222 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.223 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.223 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.224 DEBUG:boto:Method: GET 2022-09-13T07:34:13.224 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.225 DEBUG:boto:Data: 2022-09-13T07:34:13.225 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.226 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:13.226 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.227 DEBUG:boto:Params: {} 2022-09-13T07:34:13.228 DEBUG:boto:Token: None 2022-09-13T07:34:13.228 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.229 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t3npxCE4m0ndfX8hTufJ1GuTf/s= 2022-09-13T07:34:13.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t3npxCE4m0ndfX8hTufJ1GuTf/s=', 'Content-Length': '0'} 2022-09-13T07:34:13.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000182fc441ef3cdb89-0063203275-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.232 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.233 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.233 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.234 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.234 DEBUG:boto:Method: GET 2022-09-13T07:34:13.235 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.235 DEBUG:boto:Data: 2022-09-13T07:34:13.236 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.237 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.237 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.238 DEBUG:boto:Params: {} 2022-09-13T07:34:13.238 DEBUG:boto:Token: None 2022-09-13T07:34:13.239 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.239 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t3npxCE4m0ndfX8hTufJ1GuTf/s= 2022-09-13T07:34:13.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t3npxCE4m0ndfX8hTufJ1GuTf/s=', 'Content-Length': '0'} 2022-09-13T07:34:13.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e384c27233e44676-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.243 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.244 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.244 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:13.245 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:13.245 DEBUG:boto:Method: GET 2022-09-13T07:34:13.246 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:13.246 DEBUG:boto:Data: 2022-09-13T07:34:13.247 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.247 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:13.248 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.248 DEBUG:boto:Params: {} 2022-09-13T07:34:13.249 DEBUG:boto:Token: None 2022-09-13T07:34:13.249 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.250 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac59cfbac7453962-0063203275-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.254 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.254 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.255 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.256 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:13.256 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:13.257 DEBUG:boto:Method: GET 2022-09-13T07:34:13.257 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:13.258 DEBUG:boto:Data: 2022-09-13T07:34:13.258 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.259 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.259 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.260 DEBUG:boto:Params: {} 2022-09-13T07:34:13.260 DEBUG:boto:Token: None 2022-09-13T07:34:13.261 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.261 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045834db2c8b81afe-0063203275-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.310 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.311 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.312 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.312 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.313 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.313 DEBUG:boto:Data: 2022-09-13T07:34:13.314 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.314 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:13.315 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.315 DEBUG:boto:Params: {} 2022-09-13T07:34:13.316 DEBUG:boto:Token: None 2022-09-13T07:34:13.316 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.317 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4plfj6YgoO/8/iHW/CYiSJAnoiQ= 2022-09-13T07:34:13.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4plfj6YgoO/8/iHW/CYiSJAnoiQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f75bced2fb6391da-0063203275-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.322 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.322 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.323 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.323 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.324 DEBUG:boto:Data: 2022-09-13T07:34:13.324 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.325 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.325 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.326 DEBUG:boto:Params: {} 2022-09-13T07:34:13.327 DEBUG:boto:Token: None 2022-09-13T07:34:13.327 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.328 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4plfj6YgoO/8/iHW/CYiSJAnoiQ= 2022-09-13T07:34:13.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4plfj6YgoO/8/iHW/CYiSJAnoiQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008553cd67ec57ec5d-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:13.333 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.334 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.334 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.335 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.335 DEBUG:boto:Method: GET 2022-09-13T07:34:13.336 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.337 DEBUG:boto:Data: 2022-09-13T07:34:13.337 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.338 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:13.338 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.339 DEBUG:boto:Params: {} 2022-09-13T07:34:13.339 DEBUG:boto:Token: None 2022-09-13T07:34:13.340 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.341 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t3npxCE4m0ndfX8hTufJ1GuTf/s= 2022-09-13T07:34:13.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t3npxCE4m0ndfX8hTufJ1GuTf/s=', 'Content-Length': '0'} 2022-09-13T07:34:13.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f977492a2705424-0063203275-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.344 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.345 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:13.346 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.346 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.347 DEBUG:boto:Method: GET 2022-09-13T07:34:13.347 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.348 DEBUG:boto:Data: 2022-09-13T07:34:13.348 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.349 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.349 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.350 DEBUG:boto:Params: {} 2022-09-13T07:34:13.350 DEBUG:boto:Token: None 2022-09-13T07:34:13.351 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:13.352 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t3npxCE4m0ndfX8hTufJ1GuTf/s= 2022-09-13T07:34:13.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t3npxCE4m0ndfX8hTufJ1GuTf/s=', 'Content-Length': '0'} 2022-09-13T07:34:13.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b47aa2456c7f3ee7-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.355 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.355 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.356 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:13.357 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:13.357 DEBUG:boto:Method: GET 2022-09-13T07:34:13.358 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:13.358 DEBUG:boto:Data: 2022-09-13T07:34:13.359 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.359 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:13.360 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.360 DEBUG:boto:Params: {} 2022-09-13T07:34:13.361 DEBUG:boto:Token: None 2022-09-13T07:34:13.361 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.362 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007510e07bf9c2182a-0063203275-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.365 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.366 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.367 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.367 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:13.368 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:13.368 DEBUG:boto:Method: GET 2022-09-13T07:34:13.369 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:13.369 DEBUG:boto:Data: 2022-09-13T07:34:13.370 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.371 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.371 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.372 DEBUG:boto:Params: {} 2022-09-13T07:34:13.372 DEBUG:boto:Token: None 2022-09-13T07:34:13.373 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.373 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075f0c11bca6ae750-0063203275-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.377 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.378 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-17 zones={test-zone2, test-zone1} 2022-09-13T07:34:13.379 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-17 zones={test-zone3, test-zone1} 2022-09-13T07:34:13.379 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.380 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.380 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.381 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:34:13.381 DEBUG:boto:Data: 2022-09-13T07:34:13.382 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.382 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.383 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.383 DEBUG:boto:Params: {} 2022-09-13T07:34:13.384 DEBUG:boto:Token: None 2022-09-13T07:34:13.384 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.385 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oaxktW/huLm3NRgf10VJXcfyCFs= 2022-09-13T07:34:13.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oaxktW/huLm3NRgf10VJXcfyCFs=', 'Content-Length': '0'} 2022-09-13T07:34:13.388 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e149bd69db417637-0063203275-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.389 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.389 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.390 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.390 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:34:13.391 DEBUG:boto:Data: 2022-09-13T07:34:13.391 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.392 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.392 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.393 DEBUG:boto:Params: {} 2022-09-13T07:34:13.393 DEBUG:boto:Token: None 2022-09-13T07:34:13.394 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.394 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oaxktW/huLm3NRgf10VJXcfyCFs= 2022-09-13T07:34:13.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oaxktW/huLm3NRgf10VJXcfyCFs=', 'Content-Length': '0'} 2022-09-13T07:34:13.397 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f47bae5df6172df-0063203275-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.398 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:34:13.398 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.399 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.399 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:13.400 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:13.400 DEBUG:boto:Method: GET 2022-09-13T07:34:13.401 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:13.401 DEBUG:boto:Data: 2022-09-13T07:34:13.402 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.402 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.403 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.403 DEBUG:boto:Params: {} 2022-09-13T07:34:13.404 DEBUG:boto:Token: None 2022-09-13T07:34:13.405 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/?versions 2022-09-13T07:34:13.405 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:riVLRL4ysYad+zD7R+Ybxo5o5Ls= 2022-09-13T07:34:13.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:riVLRL4ysYad+zD7R+Ybxo5o5Ls=', 'Content-Length': '0'} 2022-09-13T07:34:13.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bb955fb1a94545a-0063203275-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.409 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.410 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:34:13.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:34:13.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:34:13.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:34:13.412 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:34:13.412 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.413 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.413 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:13.414 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:13.414 DEBUG:boto:Method: GET 2022-09-13T07:34:13.415 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:13.415 DEBUG:boto:Data: 2022-09-13T07:34:13.416 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.416 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.417 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.417 DEBUG:boto:Params: {} 2022-09-13T07:34:13.418 DEBUG:boto:Token: None 2022-09-13T07:34:13.419 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/?versions 2022-09-13T07:34:13.419 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:riVLRL4ysYad+zD7R+Ybxo5o5Ls= 2022-09-13T07:34:13.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:riVLRL4ysYad+zD7R+Ybxo5o5Ls=', 'Content-Length': '0'} 2022-09-13T07:34:13.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcd9918a6c61c707-0063203275-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.423 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.424 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:34:13.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:34:13.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:34:13.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:34:13.426 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.427 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.427 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:13.428 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:13.428 DEBUG:boto:Method: GET 2022-09-13T07:34:13.429 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:13.430 DEBUG:boto:Data: 2022-09-13T07:34:13.430 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.431 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.431 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.432 DEBUG:boto:Params: {} 2022-09-13T07:34:13.432 DEBUG:boto:Token: None 2022-09-13T07:34:13.433 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/?versions 2022-09-13T07:34:13.433 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:riVLRL4ysYad+zD7R+Ybxo5o5Ls= 2022-09-13T07:34:13.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:riVLRL4ysYad+zD7R+Ybxo5o5Ls=', 'Content-Length': '0'} 2022-09-13T07:34:13.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd2dd47722fb6048-0063203275-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.437 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.438 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.439 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.440 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:13.440 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:13.441 DEBUG:boto:Method: GET 2022-09-13T07:34:13.441 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:13.442 DEBUG:boto:Data: 2022-09-13T07:34:13.442 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.443 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.443 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.444 DEBUG:boto:Params: {} 2022-09-13T07:34:13.444 DEBUG:boto:Token: None 2022-09-13T07:34:13.445 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/?versions 2022-09-13T07:34:13.446 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:riVLRL4ysYad+zD7R+Ybxo5o5Ls= 2022-09-13T07:34:13.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:riVLRL4ysYad+zD7R+Ybxo5o5Ls=', 'Content-Length': '0'} 2022-09-13T07:34:13.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000503bfe9b64a28458-0063203275-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.450 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:13.452 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:13.452 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:13.453 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.453 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.454 DEBUG:boto:Method: GET 2022-09-13T07:34:13.455 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.455 DEBUG:boto:Data: 2022-09-13T07:34:13.456 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.456 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.457 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.457 DEBUG:boto:Params: {} 2022-09-13T07:34:13.458 DEBUG:boto:Token: None 2022-09-13T07:34:13.459 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.459 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pgt2tiTGwfGygMrAlfZjlrjyGqQ= 2022-09-13T07:34:13.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pgt2tiTGwfGygMrAlfZjlrjyGqQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073d4e366ebe92fd4-0063203275-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.463 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:13.464 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:13.464 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.465 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.466 DEBUG:boto:Method: GET 2022-09-13T07:34:13.467 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.468 DEBUG:boto:Data: 2022-09-13T07:34:13.469 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.470 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.471 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.472 DEBUG:boto:Params: {} 2022-09-13T07:34:13.472 DEBUG:boto:Token: None 2022-09-13T07:34:13.473 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.473 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pgt2tiTGwfGygMrAlfZjlrjyGqQ= 2022-09-13T07:34:13.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pgt2tiTGwfGygMrAlfZjlrjyGqQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008376112eeaee1553-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.477 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:13.477 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:13.478 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.478 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.479 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.479 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.480 DEBUG:boto:Data: 2022-09-13T07:34:13.480 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.481 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.481 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.481 DEBUG:boto:Params: {} 2022-09-13T07:34:13.482 DEBUG:boto:Token: None 2022-09-13T07:34:13.483 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.483 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2oEkNw8VFOA9lWL2v0pkkhqJgyA= 2022-09-13T07:34:13.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2oEkNw8VFOA9lWL2v0pkkhqJgyA=', 'Content-Length': '0'} 2022-09-13T07:34:13.486 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000194b5d17606c4ae3-0063203275-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.486 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:13.487 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:13.487 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.488 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.488 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.489 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.489 DEBUG:boto:Data: 2022-09-13T07:34:13.490 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.490 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.491 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.491 DEBUG:boto:Params: {} 2022-09-13T07:34:13.492 DEBUG:boto:Token: None 2022-09-13T07:34:13.492 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.493 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2oEkNw8VFOA9lWL2v0pkkhqJgyA= 2022-09-13T07:34:13.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2oEkNw8VFOA9lWL2v0pkkhqJgyA=', 'Content-Length': '0'} 2022-09-13T07:34:13.495 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000094ac5212a7d194e3-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:13.496 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:13.497 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:13.497 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.498 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.498 DEBUG:boto:Method: GET 2022-09-13T07:34:13.499 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.499 DEBUG:boto:Data: 2022-09-13T07:34:13.500 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.500 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.501 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.501 DEBUG:boto:Params: {} 2022-09-13T07:34:13.502 DEBUG:boto:Token: None 2022-09-13T07:34:13.503 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.503 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pgt2tiTGwfGygMrAlfZjlrjyGqQ= 2022-09-13T07:34:13.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pgt2tiTGwfGygMrAlfZjlrjyGqQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.505 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c696652cdd42608c-0063203275-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.506 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:13.507 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:13.507 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.508 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.508 DEBUG:boto:Method: GET 2022-09-13T07:34:13.509 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.509 DEBUG:boto:Data: 2022-09-13T07:34:13.510 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.510 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.511 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.511 DEBUG:boto:Params: {} 2022-09-13T07:34:13.512 DEBUG:boto:Token: None 2022-09-13T07:34:13.512 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.513 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pgt2tiTGwfGygMrAlfZjlrjyGqQ= 2022-09-13T07:34:13.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pgt2tiTGwfGygMrAlfZjlrjyGqQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.516 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007a7efbcb638fb137-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.516 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.517 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.517 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:13.518 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:13.519 DEBUG:boto:Method: GET 2022-09-13T07:34:13.519 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:34:13.520 DEBUG:boto:Data: 2022-09-13T07:34:13.520 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.521 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.521 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.522 DEBUG:boto:Params: {} 2022-09-13T07:34:13.523 DEBUG:boto:Token: None 2022-09-13T07:34:13.523 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.524 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e26943ccd022f8fc-0063203275-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.570 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.571 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.572 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.572 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:13.573 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:13.573 DEBUG:boto:Method: GET 2022-09-13T07:34:13.574 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:34:13.574 DEBUG:boto:Data: 2022-09-13T07:34:13.575 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.575 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.576 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.576 DEBUG:boto:Params: {} 2022-09-13T07:34:13.577 DEBUG:boto:Token: None 2022-09-13T07:34:13.578 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.578 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c6f42e22765a8fa-0063203275-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.582 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.583 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:13.584 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:13.584 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.584 DEBUG:boto:Path: /yxvvuh-17/dummy 2022-09-13T07:34:13.585 DEBUG:boto:Data: 2022-09-13T07:34:13.585 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.586 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.586 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.587 DEBUG:boto:Params: {} 2022-09-13T07:34:13.587 DEBUG:boto:Token: None 2022-09-13T07:34:13.588 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/dummy 2022-09-13T07:34:13.588 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fXg+1pfMWnAa1Xf3WNECMWB9LBk= 2022-09-13T07:34:13.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fXg+1pfMWnAa1Xf3WNECMWB9LBk=', 'Content-Length': '0'} 2022-09-13T07:34:13.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000081489903854b55aa-0063203275-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.592 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:13.592 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:13.593 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.593 DEBUG:boto:Path: /yxvvuh-17/dummy 2022-09-13T07:34:13.594 DEBUG:boto:Data: 2022-09-13T07:34:13.594 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.595 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.595 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.596 DEBUG:boto:Params: {} 2022-09-13T07:34:13.597 DEBUG:boto:Token: None 2022-09-13T07:34:13.597 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/dummy 2022-09-13T07:34:13.598 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fXg+1pfMWnAa1Xf3WNECMWB9LBk= 2022-09-13T07:34:13.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fXg+1pfMWnAa1Xf3WNECMWB9LBk=', 'Content-Length': '0'} 2022-09-13T07:34:13.601 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d7799c7d043ea0a4-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:13.603 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:13.604 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:13.605 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.605 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.606 DEBUG:boto:Method: GET 2022-09-13T07:34:13.606 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.607 DEBUG:boto:Data: 2022-09-13T07:34:13.607 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.608 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.608 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.609 DEBUG:boto:Params: {} 2022-09-13T07:34:13.609 DEBUG:boto:Token: None 2022-09-13T07:34:13.610 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.611 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pgt2tiTGwfGygMrAlfZjlrjyGqQ= 2022-09-13T07:34:13.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pgt2tiTGwfGygMrAlfZjlrjyGqQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c664180e9141bbad-0063203275-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.614 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:13.615 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:13.615 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.616 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.616 DEBUG:boto:Method: GET 2022-09-13T07:34:13.617 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.617 DEBUG:boto:Data: 2022-09-13T07:34:13.618 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.618 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.619 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.619 DEBUG:boto:Params: {} 2022-09-13T07:34:13.620 DEBUG:boto:Token: None 2022-09-13T07:34:13.620 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:13.621 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pgt2tiTGwfGygMrAlfZjlrjyGqQ= 2022-09-13T07:34:13.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pgt2tiTGwfGygMrAlfZjlrjyGqQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002ca8a54fb5582895-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.624 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.625 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.625 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:13.626 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:13.626 DEBUG:boto:Method: GET 2022-09-13T07:34:13.627 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:34:13.627 DEBUG:boto:Data: 2022-09-13T07:34:13.628 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.628 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.629 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.630 DEBUG:boto:Params: {} 2022-09-13T07:34:13.631 DEBUG:boto:Token: None 2022-09-13T07:34:13.632 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.633 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5bd76d93c99131a-0063203275-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.638 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.639 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.639 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.640 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:13.640 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:13.641 DEBUG:boto:Method: GET 2022-09-13T07:34:13.642 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:34:13.643 DEBUG:boto:Data: 2022-09-13T07:34:13.644 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.644 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.645 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.646 DEBUG:boto:Params: {} 2022-09-13T07:34:13.647 DEBUG:boto:Token: None 2022-09-13T07:34:13.648 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.649 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a771aab99141684-0063203275-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.653 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:13.654 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.654 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.655 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.655 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.656 DEBUG:boto:Method: GET 2022-09-13T07:34:13.656 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.657 DEBUG:boto:Data: 2022-09-13T07:34:13.658 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.658 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.659 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.660 DEBUG:boto:Params: {} 2022-09-13T07:34:13.660 DEBUG:boto:Token: None 2022-09-13T07:34:13.661 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.661 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UTlgRRkMMlRrM0KU4ClD7E/51qQ= 2022-09-13T07:34:13.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UTlgRRkMMlRrM0KU4ClD7E/51qQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.664 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000426974732913ccf5-0063203275-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.665 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.665 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.666 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.666 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.667 DEBUG:boto:Method: GET 2022-09-13T07:34:13.668 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.668 DEBUG:boto:Data: 2022-09-13T07:34:13.669 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.669 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.670 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.670 DEBUG:boto:Params: {} 2022-09-13T07:34:13.671 DEBUG:boto:Token: None 2022-09-13T07:34:13.672 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.672 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UTlgRRkMMlRrM0KU4ClD7E/51qQ= 2022-09-13T07:34:13.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UTlgRRkMMlRrM0KU4ClD7E/51qQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fff8c076486a2ba-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.676 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.677 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.677 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.678 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.678 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.679 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.679 DEBUG:boto:Data: 2022-09-13T07:34:13.679 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.680 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.680 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.681 DEBUG:boto:Params: {} 2022-09-13T07:34:13.682 DEBUG:boto:Token: None 2022-09-13T07:34:13.682 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.683 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eFAhLxPkGesoKqKwlOx9XcCq7tE= 2022-09-13T07:34:13.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFAhLxPkGesoKqKwlOx9XcCq7tE=', 'Content-Length': '0'} 2022-09-13T07:34:13.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e11c37c802b730da-0063203275-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.686 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.687 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.687 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.687 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.688 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.688 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.689 DEBUG:boto:Data: 2022-09-13T07:34:13.689 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.690 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.690 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.691 DEBUG:boto:Params: {} 2022-09-13T07:34:13.691 DEBUG:boto:Token: None 2022-09-13T07:34:13.692 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.693 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eFAhLxPkGesoKqKwlOx9XcCq7tE= 2022-09-13T07:34:13.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFAhLxPkGesoKqKwlOx9XcCq7tE=', 'Content-Length': '0'} 2022-09-13T07:34:13.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003debc6998cc8f954-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:13.697 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.697 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.698 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.698 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.699 DEBUG:boto:Method: GET 2022-09-13T07:34:13.699 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.700 DEBUG:boto:Data: 2022-09-13T07:34:13.700 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.701 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.701 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.702 DEBUG:boto:Params: {} 2022-09-13T07:34:13.703 DEBUG:boto:Token: None 2022-09-13T07:34:13.703 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.704 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UTlgRRkMMlRrM0KU4ClD7E/51qQ= 2022-09-13T07:34:13.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UTlgRRkMMlRrM0KU4ClD7E/51qQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.707 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a4a63c7ca91a36f-0063203275-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.707 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.708 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.708 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.709 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.709 DEBUG:boto:Method: GET 2022-09-13T07:34:13.710 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.710 DEBUG:boto:Data: 2022-09-13T07:34:13.711 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.712 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.712 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.713 DEBUG:boto:Params: {} 2022-09-13T07:34:13.713 DEBUG:boto:Token: None 2022-09-13T07:34:13.714 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.714 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UTlgRRkMMlRrM0KU4ClD7E/51qQ= 2022-09-13T07:34:13.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UTlgRRkMMlRrM0KU4ClD7E/51qQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a94eb58c37a9d952-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.718 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.718 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.719 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:13.720 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:13.720 DEBUG:boto:Method: GET 2022-09-13T07:34:13.721 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:13.721 DEBUG:boto:Data: 2022-09-13T07:34:13.722 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.722 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.723 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.723 DEBUG:boto:Params: {} 2022-09-13T07:34:13.724 DEBUG:boto:Token: None 2022-09-13T07:34:13.724 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.725 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bf9083c453417cd-0063203275-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.770 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.771 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.772 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.772 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:13.773 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:13.773 DEBUG:boto:Method: GET 2022-09-13T07:34:13.774 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:13.774 DEBUG:boto:Data: 2022-09-13T07:34:13.775 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.775 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.776 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.776 DEBUG:boto:Params: {} 2022-09-13T07:34:13.777 DEBUG:boto:Token: None 2022-09-13T07:34:13.777 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.778 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e792eacb1ea13db-0063203275-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.782 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.783 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.783 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.784 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.784 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.785 DEBUG:boto:Data: 2022-09-13T07:34:13.785 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.786 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.786 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.787 DEBUG:boto:Params: {} 2022-09-13T07:34:13.787 DEBUG:boto:Token: None 2022-09-13T07:34:13.788 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.788 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bdlVU+8vYYnE7d1fkDnrWOVVmto= 2022-09-13T07:34:13.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bdlVU+8vYYnE7d1fkDnrWOVVmto=', 'Content-Length': '0'} 2022-09-13T07:34:13.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000887f88c5b7dcf124-0063203275-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.792 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.793 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.793 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.794 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.794 DEBUG:boto:Data: 2022-09-13T07:34:13.795 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.795 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.796 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.796 DEBUG:boto:Params: {} 2022-09-13T07:34:13.797 DEBUG:boto:Token: None 2022-09-13T07:34:13.797 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.798 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bdlVU+8vYYnE7d1fkDnrWOVVmto= 2022-09-13T07:34:13.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bdlVU+8vYYnE7d1fkDnrWOVVmto=', 'Content-Length': '0'} 2022-09-13T07:34:13.800 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f1bd1ec1a76c35d-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:13.802 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.802 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.803 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.803 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.804 DEBUG:boto:Method: GET 2022-09-13T07:34:13.804 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.805 DEBUG:boto:Data: 2022-09-13T07:34:13.805 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.806 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.806 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.807 DEBUG:boto:Params: {} 2022-09-13T07:34:13.807 DEBUG:boto:Token: None 2022-09-13T07:34:13.808 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.808 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UTlgRRkMMlRrM0KU4ClD7E/51qQ= 2022-09-13T07:34:13.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UTlgRRkMMlRrM0KU4ClD7E/51qQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041085ee194fa1afc-0063203275-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.812 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.812 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:13.813 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.813 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.814 DEBUG:boto:Method: GET 2022-09-13T07:34:13.814 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.815 DEBUG:boto:Data: 2022-09-13T07:34:13.815 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.816 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.816 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.817 DEBUG:boto:Params: {} 2022-09-13T07:34:13.817 DEBUG:boto:Token: None 2022-09-13T07:34:13.818 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:13.818 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UTlgRRkMMlRrM0KU4ClD7E/51qQ= 2022-09-13T07:34:13.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UTlgRRkMMlRrM0KU4ClD7E/51qQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da21e973e84747c4-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.822 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.822 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.823 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:13.823 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:13.824 DEBUG:boto:Method: GET 2022-09-13T07:34:13.824 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:13.825 DEBUG:boto:Data: 2022-09-13T07:34:13.825 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.826 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.826 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.827 DEBUG:boto:Params: {} 2022-09-13T07:34:13.827 DEBUG:boto:Token: None 2022-09-13T07:34:13.828 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.828 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000937608f786b7a0d7-0063203275-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.832 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.833 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.833 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.834 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:13.834 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:13.835 DEBUG:boto:Method: GET 2022-09-13T07:34:13.835 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:13.836 DEBUG:boto:Data: 2022-09-13T07:34:13.836 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.837 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.837 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.838 DEBUG:boto:Params: {} 2022-09-13T07:34:13.838 DEBUG:boto:Token: None 2022-09-13T07:34:13.839 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.840 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071d0335ff07ab6bc-0063203275-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.843 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:13.845 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.845 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.846 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.846 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.847 DEBUG:boto:Method: GET 2022-09-13T07:34:13.847 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.848 DEBUG:boto:Data: 2022-09-13T07:34:13.848 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.849 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.849 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.850 DEBUG:boto:Params: {} 2022-09-13T07:34:13.850 DEBUG:boto:Token: None 2022-09-13T07:34:13.851 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.851 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NaRYHT9B2PvUGaJIzSdH26diy7I= 2022-09-13T07:34:13.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NaRYHT9B2PvUGaJIzSdH26diy7I=', 'Content-Length': '0'} 2022-09-13T07:34:13.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015ddc38e4d2ccac5-0063203275-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.855 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.855 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.856 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.856 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.857 DEBUG:boto:Method: GET 2022-09-13T07:34:13.858 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.858 DEBUG:boto:Data: 2022-09-13T07:34:13.859 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.859 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.860 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.860 DEBUG:boto:Params: {} 2022-09-13T07:34:13.861 DEBUG:boto:Token: None 2022-09-13T07:34:13.861 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.862 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NaRYHT9B2PvUGaJIzSdH26diy7I= 2022-09-13T07:34:13.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NaRYHT9B2PvUGaJIzSdH26diy7I=', 'Content-Length': '0'} 2022-09-13T07:34:13.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e1bc1ab68222533-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.865 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.866 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.866 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.867 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.867 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.868 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.869 DEBUG:boto:Data: 2022-09-13T07:34:13.869 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.869 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.870 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.870 DEBUG:boto:Params: {} 2022-09-13T07:34:13.871 DEBUG:boto:Token: None 2022-09-13T07:34:13.871 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.872 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eeijmsu4N+DCnXd0JVMXduyoV/0= 2022-09-13T07:34:13.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eeijmsu4N+DCnXd0JVMXduyoV/0=', 'Content-Length': '0'} 2022-09-13T07:34:13.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b589d777831d3671-0063203275-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.875 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.876 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.876 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.877 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.877 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.878 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.878 DEBUG:boto:Data: 2022-09-13T07:34:13.879 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.879 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.880 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.880 DEBUG:boto:Params: {} 2022-09-13T07:34:13.881 DEBUG:boto:Token: None 2022-09-13T07:34:13.881 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.882 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eeijmsu4N+DCnXd0JVMXduyoV/0= 2022-09-13T07:34:13.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eeijmsu4N+DCnXd0JVMXduyoV/0=', 'Content-Length': '0'} 2022-09-13T07:34:13.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071fd89353081857c-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:13.886 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.887 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.887 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.888 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.888 DEBUG:boto:Method: GET 2022-09-13T07:34:13.889 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.889 DEBUG:boto:Data: 2022-09-13T07:34:13.890 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.891 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.892 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.893 DEBUG:boto:Params: {} 2022-09-13T07:34:13.893 DEBUG:boto:Token: None 2022-09-13T07:34:13.894 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.894 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NaRYHT9B2PvUGaJIzSdH26diy7I= 2022-09-13T07:34:13.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NaRYHT9B2PvUGaJIzSdH26diy7I=', 'Content-Length': '0'} 2022-09-13T07:34:13.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000676997096cd13790-0063203275-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.938 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.939 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.939 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.940 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.940 DEBUG:boto:Method: GET 2022-09-13T07:34:13.941 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.942 DEBUG:boto:Data: 2022-09-13T07:34:13.942 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.943 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.943 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.944 DEBUG:boto:Params: {} 2022-09-13T07:34:13.945 DEBUG:boto:Token: None 2022-09-13T07:34:13.945 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.946 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NaRYHT9B2PvUGaJIzSdH26diy7I= 2022-09-13T07:34:13.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NaRYHT9B2PvUGaJIzSdH26diy7I=', 'Content-Length': '0'} 2022-09-13T07:34:13.949 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009cef0252ab499182-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.950 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.950 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.951 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.951 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.952 DEBUG:boto:Method: GET 2022-09-13T07:34:13.952 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.953 DEBUG:boto:Data: 2022-09-13T07:34:13.953 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.954 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.954 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.955 DEBUG:boto:Params: {} 2022-09-13T07:34:13.955 DEBUG:boto:Token: None 2022-09-13T07:34:13.956 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.957 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085b98cc09aab85da-0063203275-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.961 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.961 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:13.962 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:13.963 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.963 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.964 DEBUG:boto:Method: GET 2022-09-13T07:34:13.965 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:13.965 DEBUG:boto:Data: 2022-09-13T07:34:13.966 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.966 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.967 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.967 DEBUG:boto:Params: {} 2022-09-13T07:34:13.968 DEBUG:boto:Token: None 2022-09-13T07:34:13.968 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/ 2022-09-13T07:34:13.969 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ= 2022-09-13T07:34:13.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Hl5WQ6HpxrOUFQMDLWUtNoMcFBQ=', 'Content-Length': '0'} 2022-09-13T07:34:13.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023817756837e6a38-0063203275-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.973 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:13.974 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.974 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.975 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.975 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.976 DEBUG:boto:Data: 2022-09-13T07:34:13.976 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.977 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.977 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.978 DEBUG:boto:Params: {} 2022-09-13T07:34:13.978 DEBUG:boto:Token: None 2022-09-13T07:34:13.979 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.979 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:glHF2+c+svDET/AUcSvUF+AT7I4= 2022-09-13T07:34:13.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:glHF2+c+svDET/AUcSvUF+AT7I4=', 'Content-Length': '0'} 2022-09-13T07:34:13.983 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057cd6ce5001c09da-0063203275-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.983 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.984 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.985 DEBUG:boto:Method: HEAD 2022-09-13T07:34:13.985 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.985 DEBUG:boto:Data: 2022-09-13T07:34:13.986 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.986 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:13.987 DEBUG:boto:Port: 8000 2022-09-13T07:34:13.987 DEBUG:boto:Params: {} 2022-09-13T07:34:13.988 DEBUG:boto:Token: None 2022-09-13T07:34:13.988 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:13 GMT /yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.989 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:glHF2+c+svDET/AUcSvUF+AT7I4= 2022-09-13T07:34:13.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:glHF2+c+svDET/AUcSvUF+AT7I4=', 'Content-Length': '0'} 2022-09-13T07:34:13.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013fbcb7c1fe99bfd-0063203275-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:13.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:13.994 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.995 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:13.995 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.996 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.996 DEBUG:boto:Method: GET 2022-09-13T07:34:13.997 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:13.997 DEBUG:boto:Data: 2022-09-13T07:34:13.998 DEBUG:boto:Headers: {} 2022-09-13T07:34:13.998 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:13.999 DEBUG:boto:Port: 8001 2022-09-13T07:34:13.999 DEBUG:boto:Params: {} 2022-09-13T07:34:14.000 DEBUG:boto:Token: None 2022-09-13T07:34:14.001 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.001 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2r7eJbEVhwT7OGAClF62KbGEmic= 2022-09-13T07:34:14.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2r7eJbEVhwT7OGAClF62KbGEmic=', 'Content-Length': '0'} 2022-09-13T07:34:14.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000623418810b9ac543-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.005 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:14.005 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:14.006 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.006 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.007 DEBUG:boto:Method: GET 2022-09-13T07:34:14.007 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.008 DEBUG:boto:Data: 2022-09-13T07:34:14.008 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.009 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:14.009 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.010 DEBUG:boto:Params: {} 2022-09-13T07:34:14.011 DEBUG:boto:Token: None 2022-09-13T07:34:14.011 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.012 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2r7eJbEVhwT7OGAClF62KbGEmic= 2022-09-13T07:34:14.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2r7eJbEVhwT7OGAClF62KbGEmic=', 'Content-Length': '0'} 2022-09-13T07:34:14.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e3a17f5970e402d-0063203276-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.016 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.016 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.017 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:14.018 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:14.018 DEBUG:boto:Method: GET 2022-09-13T07:34:14.019 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:14.019 DEBUG:boto:Data: 2022-09-13T07:34:14.020 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.020 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.021 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.021 DEBUG:boto:Params: {} 2022-09-13T07:34:14.022 DEBUG:boto:Token: None 2022-09-13T07:34:14.022 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.023 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q= 2022-09-13T07:34:14.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q=', 'Content-Length': '0'} 2022-09-13T07:34:14.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088d9d527870d2bd5-0063203276-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.027 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.028 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.028 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.029 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:14.029 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:14.030 DEBUG:boto:Method: GET 2022-09-13T07:34:14.030 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:14.031 DEBUG:boto:Data: 2022-09-13T07:34:14.031 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.032 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:14.032 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.033 DEBUG:boto:Params: {} 2022-09-13T07:34:14.034 DEBUG:boto:Token: None 2022-09-13T07:34:14.034 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.035 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q= 2022-09-13T07:34:14.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q=', 'Content-Length': '0'} 2022-09-13T07:34:14.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d345d4cfce195d53-0063203276-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.039 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:14.040 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.041 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.042 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.042 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.043 DEBUG:boto:Method: GET 2022-09-13T07:34:14.043 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.044 DEBUG:boto:Data: 2022-09-13T07:34:14.044 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.045 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.045 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.046 DEBUG:boto:Params: {} 2022-09-13T07:34:14.046 DEBUG:boto:Token: None 2022-09-13T07:34:14.047 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.047 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Qsg6135AGKz7dfPoJSAIUEk6X4U= 2022-09-13T07:34:14.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Qsg6135AGKz7dfPoJSAIUEk6X4U=', 'Content-Length': '0'} 2022-09-13T07:34:14.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7c731f44e27fbff-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.094 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.095 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.096 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.097 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.098 DEBUG:boto:Method: GET 2022-09-13T07:34:14.099 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.100 DEBUG:boto:Data: 2022-09-13T07:34:14.101 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.102 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:14.103 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.104 DEBUG:boto:Params: {} 2022-09-13T07:34:14.105 DEBUG:boto:Token: None 2022-09-13T07:34:14.106 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.107 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Qsg6135AGKz7dfPoJSAIUEk6X4U= 2022-09-13T07:34:14.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Qsg6135AGKz7dfPoJSAIUEk6X4U=', 'Content-Length': '0'} 2022-09-13T07:34:14.110 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001cc463d0cfdc820-0063203276-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.111 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.111 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.112 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.112 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.113 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.113 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.114 DEBUG:boto:Data: 2022-09-13T07:34:14.115 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.115 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.116 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.116 DEBUG:boto:Params: {} 2022-09-13T07:34:14.117 DEBUG:boto:Token: None 2022-09-13T07:34:14.117 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.118 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3yL87KE1Pkzn7P3DNTQDw430Aoo= 2022-09-13T07:34:14.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3yL87KE1Pkzn7P3DNTQDw430Aoo=', 'Content-Length': '0'} 2022-09-13T07:34:14.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000280e02c6bf790d42-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.121 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.122 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.122 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.123 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.123 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.124 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.125 DEBUG:boto:Data: 2022-09-13T07:34:14.125 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.126 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:14.126 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.127 DEBUG:boto:Params: {} 2022-09-13T07:34:14.128 DEBUG:boto:Token: None 2022-09-13T07:34:14.128 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.129 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3yL87KE1Pkzn7P3DNTQDw430Aoo= 2022-09-13T07:34:14.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3yL87KE1Pkzn7P3DNTQDw430Aoo=', 'Content-Length': '0'} 2022-09-13T07:34:14.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a1cea5798488af0-0063203276-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:14.135 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.136 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.137 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.138 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.139 DEBUG:boto:Method: GET 2022-09-13T07:34:14.140 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.141 DEBUG:boto:Data: 2022-09-13T07:34:14.142 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.143 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.144 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.145 DEBUG:boto:Params: {} 2022-09-13T07:34:14.145 DEBUG:boto:Token: None 2022-09-13T07:34:14.146 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.146 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Qsg6135AGKz7dfPoJSAIUEk6X4U= 2022-09-13T07:34:14.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Qsg6135AGKz7dfPoJSAIUEk6X4U=', 'Content-Length': '0'} 2022-09-13T07:34:14.149 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd458ab04c1537da-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.150 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.151 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.151 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.152 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.153 DEBUG:boto:Method: GET 2022-09-13T07:34:14.153 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.154 DEBUG:boto:Data: 2022-09-13T07:34:14.154 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.155 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:14.155 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.156 DEBUG:boto:Params: {} 2022-09-13T07:34:14.156 DEBUG:boto:Token: None 2022-09-13T07:34:14.157 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.158 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Qsg6135AGKz7dfPoJSAIUEk6X4U= 2022-09-13T07:34:14.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Qsg6135AGKz7dfPoJSAIUEk6X4U=', 'Content-Length': '0'} 2022-09-13T07:34:14.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d9a536fa74fe565-0063203276-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.202 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.203 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.204 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:14.204 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:14.205 DEBUG:boto:Method: GET 2022-09-13T07:34:14.205 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:14.206 DEBUG:boto:Data: 2022-09-13T07:34:14.206 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.207 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.207 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.208 DEBUG:boto:Params: {} 2022-09-13T07:34:14.209 DEBUG:boto:Token: None 2022-09-13T07:34:14.209 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.210 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q= 2022-09-13T07:34:14.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q=', 'Content-Length': '0'} 2022-09-13T07:34:14.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b613cc69c3561c98-0063203276-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.214 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.215 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.216 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.216 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:14.217 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:14.217 DEBUG:boto:Method: GET 2022-09-13T07:34:14.218 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:14.218 DEBUG:boto:Data: 2022-09-13T07:34:14.219 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.219 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:14.220 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.220 DEBUG:boto:Params: {} 2022-09-13T07:34:14.221 DEBUG:boto:Token: None 2022-09-13T07:34:14.222 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.222 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q= 2022-09-13T07:34:14.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q=', 'Content-Length': '0'} 2022-09-13T07:34:14.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8e3928ba5c96f6e-0063203276-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.228 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.229 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.230 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.230 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.231 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.231 DEBUG:boto:Data: 2022-09-13T07:34:14.232 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.232 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.233 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.233 DEBUG:boto:Params: {} 2022-09-13T07:34:14.234 DEBUG:boto:Token: None 2022-09-13T07:34:14.234 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.235 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SyDAcvNDU8SZTfGGCMPpxqvYBvs= 2022-09-13T07:34:14.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SyDAcvNDU8SZTfGGCMPpxqvYBvs=', 'Content-Length': '0'} 2022-09-13T07:34:14.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000343709c0c1e74da5-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.240 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.240 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.241 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.241 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.242 DEBUG:boto:Data: 2022-09-13T07:34:14.243 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.243 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:14.244 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.244 DEBUG:boto:Params: {} 2022-09-13T07:34:14.245 DEBUG:boto:Token: None 2022-09-13T07:34:14.245 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.246 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SyDAcvNDU8SZTfGGCMPpxqvYBvs= 2022-09-13T07:34:14.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SyDAcvNDU8SZTfGGCMPpxqvYBvs=', 'Content-Length': '0'} 2022-09-13T07:34:14.249 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc65dd22d59e47ad-0063203276-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:14.251 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.251 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.252 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.253 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.253 DEBUG:boto:Method: GET 2022-09-13T07:34:14.255 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.255 DEBUG:boto:Data: 2022-09-13T07:34:14.256 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.256 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.257 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.257 DEBUG:boto:Params: {} 2022-09-13T07:34:14.258 DEBUG:boto:Token: None 2022-09-13T07:34:14.258 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.259 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Qsg6135AGKz7dfPoJSAIUEk6X4U= 2022-09-13T07:34:14.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Qsg6135AGKz7dfPoJSAIUEk6X4U=', 'Content-Length': '0'} 2022-09-13T07:34:14.262 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a0a07e228f19885-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.262 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.263 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:14.264 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.265 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.266 DEBUG:boto:Method: GET 2022-09-13T07:34:14.267 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.268 DEBUG:boto:Data: 2022-09-13T07:34:14.269 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.270 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:14.270 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.271 DEBUG:boto:Params: {} 2022-09-13T07:34:14.272 DEBUG:boto:Token: None 2022-09-13T07:34:14.273 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:14.274 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Qsg6135AGKz7dfPoJSAIUEk6X4U= 2022-09-13T07:34:14.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Qsg6135AGKz7dfPoJSAIUEk6X4U=', 'Content-Length': '0'} 2022-09-13T07:34:14.278 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000155825a5c6bf2b87-0063203276-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.280 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.282 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.283 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:14.283 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:14.284 DEBUG:boto:Method: GET 2022-09-13T07:34:14.285 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:14.285 DEBUG:boto:Data: 2022-09-13T07:34:14.286 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.286 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.287 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.287 DEBUG:boto:Params: {} 2022-09-13T07:34:14.288 DEBUG:boto:Token: None 2022-09-13T07:34:14.289 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.289 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q= 2022-09-13T07:34:14.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q=', 'Content-Length': '0'} 2022-09-13T07:34:14.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cd7184ca79d2ef6-0063203276-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.293 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.294 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.295 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.295 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:14.296 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:14.296 DEBUG:boto:Method: GET 2022-09-13T07:34:14.297 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:14.297 DEBUG:boto:Data: 2022-09-13T07:34:14.298 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.298 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:34:14.299 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.299 DEBUG:boto:Params: {} 2022-09-13T07:34:14.300 DEBUG:boto:Token: None 2022-09-13T07:34:14.301 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.301 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q= 2022-09-13T07:34:14.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q=', 'Content-Length': '0'} 2022-09-13T07:34:14.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008932e78eb53732f6-0063203276-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.346 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.347 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-17 zones={test-zone3, test-zone1} 2022-09-13T07:34:14.348 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-17 zones={test-zone3, test-zone2} 2022-09-13T07:34:14.348 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.349 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.350 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.351 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:34:14.352 DEBUG:boto:Data: 2022-09-13T07:34:14.353 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.354 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.355 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.356 DEBUG:boto:Params: {} 2022-09-13T07:34:14.357 DEBUG:boto:Token: None 2022-09-13T07:34:14.358 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.359 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ogfx+DS7oX69D8QAdubH9GbZEro= 2022-09-13T07:34:14.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ogfx+DS7oX69D8QAdubH9GbZEro=', 'Content-Length': '0'} 2022-09-13T07:34:14.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a275b53931e79e3-0063203276-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.363 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.364 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.364 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.365 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:34:14.365 DEBUG:boto:Data: 2022-09-13T07:34:14.366 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.366 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.367 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.367 DEBUG:boto:Params: {} 2022-09-13T07:34:14.368 DEBUG:boto:Token: None 2022-09-13T07:34:14.369 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.369 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ogfx+DS7oX69D8QAdubH9GbZEro= 2022-09-13T07:34:14.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ogfx+DS7oX69D8QAdubH9GbZEro=', 'Content-Length': '0'} 2022-09-13T07:34:14.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e693219df0c8cd4-0063203276-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.373 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:34:14.374 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.374 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.375 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:14.375 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:14.376 DEBUG:boto:Method: GET 2022-09-13T07:34:14.376 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:14.377 DEBUG:boto:Data: 2022-09-13T07:34:14.377 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.378 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.378 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.379 DEBUG:boto:Params: {} 2022-09-13T07:34:14.380 DEBUG:boto:Token: None 2022-09-13T07:34:14.380 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/?versions 2022-09-13T07:34:14.381 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4xUt/txAXLDfTaPqKhcvQZKCybs= 2022-09-13T07:34:14.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4xUt/txAXLDfTaPqKhcvQZKCybs=', 'Content-Length': '0'} 2022-09-13T07:34:14.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1bfecef17bc9431-0063203276-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.385 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.386 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:34:14.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:34:14.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:34:14.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:34:14.388 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:34:14.388 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.389 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.389 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:14.390 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:14.390 DEBUG:boto:Method: GET 2022-09-13T07:34:14.391 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:14.391 DEBUG:boto:Data: 2022-09-13T07:34:14.392 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.392 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.393 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.393 DEBUG:boto:Params: {} 2022-09-13T07:34:14.394 DEBUG:boto:Token: None 2022-09-13T07:34:14.394 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/?versions 2022-09-13T07:34:14.395 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4xUt/txAXLDfTaPqKhcvQZKCybs= 2022-09-13T07:34:14.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4xUt/txAXLDfTaPqKhcvQZKCybs=', 'Content-Length': '0'} 2022-09-13T07:34:14.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033f426b9f465185a-0063203276-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.399 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.400 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:34:14.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:34:14.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:34:14.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:34:14.402 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.402 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.403 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:14.403 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:14.404 DEBUG:boto:Method: GET 2022-09-13T07:34:14.404 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:14.405 DEBUG:boto:Data: 2022-09-13T07:34:14.405 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.406 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.406 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.407 DEBUG:boto:Params: {} 2022-09-13T07:34:14.407 DEBUG:boto:Token: None 2022-09-13T07:34:14.408 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/?versions 2022-09-13T07:34:14.408 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4xUt/txAXLDfTaPqKhcvQZKCybs= 2022-09-13T07:34:14.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4xUt/txAXLDfTaPqKhcvQZKCybs=', 'Content-Length': '0'} 2022-09-13T07:34:14.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069a7921dca3709a4-0063203276-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.412 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.413 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.414 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.414 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:14.415 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:14.416 DEBUG:boto:Method: GET 2022-09-13T07:34:14.416 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:34:14.417 DEBUG:boto:Data: 2022-09-13T07:34:14.417 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.417 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.418 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.418 DEBUG:boto:Params: {} 2022-09-13T07:34:14.419 DEBUG:boto:Token: None 2022-09-13T07:34:14.420 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/?versions 2022-09-13T07:34:14.421 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4xUt/txAXLDfTaPqKhcvQZKCybs= 2022-09-13T07:34:14.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4xUt/txAXLDfTaPqKhcvQZKCybs=', 'Content-Length': '0'} 2022-09-13T07:34:14.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ec16f12dfc23c48-0063203276-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.425 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:14.426 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:14.427 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:14.427 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.428 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.428 DEBUG:boto:Method: GET 2022-09-13T07:34:14.429 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.429 DEBUG:boto:Data: 2022-09-13T07:34:14.430 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.430 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.431 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.431 DEBUG:boto:Params: {} 2022-09-13T07:34:14.432 DEBUG:boto:Token: None 2022-09-13T07:34:14.433 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.433 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aDgZ+FFT4Pt/WYJfAokkHXFBawY= 2022-09-13T07:34:14.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aDgZ+FFT4Pt/WYJfAokkHXFBawY=', 'Content-Length': '0'} 2022-09-13T07:34:14.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e5ed6a89f42927df-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.437 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:14.437 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:14.438 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.438 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.439 DEBUG:boto:Method: GET 2022-09-13T07:34:14.439 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.440 DEBUG:boto:Data: 2022-09-13T07:34:14.440 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.441 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.441 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.442 DEBUG:boto:Params: {} 2022-09-13T07:34:14.442 DEBUG:boto:Token: None 2022-09-13T07:34:14.443 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.444 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aDgZ+FFT4Pt/WYJfAokkHXFBawY= 2022-09-13T07:34:14.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aDgZ+FFT4Pt/WYJfAokkHXFBawY=', 'Content-Length': '0'} 2022-09-13T07:34:14.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d79571464a2d3f80-0063203276-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.447 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:14.448 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:14.448 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.449 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.449 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.450 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.450 DEBUG:boto:Data: 2022-09-13T07:34:14.451 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.451 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.451 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.452 DEBUG:boto:Params: {} 2022-09-13T07:34:14.452 DEBUG:boto:Token: None 2022-09-13T07:34:14.453 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.454 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7DUt3d5MvnOQeas7TvFdCN+5pKs= 2022-09-13T07:34:14.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7DUt3d5MvnOQeas7TvFdCN+5pKs=', 'Content-Length': '0'} 2022-09-13T07:34:14.456 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000058629d85831e3a58-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.457 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:14.458 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:14.459 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.459 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.460 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.461 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.461 DEBUG:boto:Data: 2022-09-13T07:34:14.462 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.462 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.463 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.463 DEBUG:boto:Params: {} 2022-09-13T07:34:14.464 DEBUG:boto:Token: None 2022-09-13T07:34:14.464 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.465 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7DUt3d5MvnOQeas7TvFdCN+5pKs= 2022-09-13T07:34:14.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7DUt3d5MvnOQeas7TvFdCN+5pKs=', 'Content-Length': '0'} 2022-09-13T07:34:14.469 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b028bfeec60189f8-0063203276-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:14.472 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:14.473 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:14.473 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.474 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.475 DEBUG:boto:Method: GET 2022-09-13T07:34:14.476 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.477 DEBUG:boto:Data: 2022-09-13T07:34:14.478 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.479 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.480 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.480 DEBUG:boto:Params: {} 2022-09-13T07:34:14.481 DEBUG:boto:Token: None 2022-09-13T07:34:14.481 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.482 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aDgZ+FFT4Pt/WYJfAokkHXFBawY= 2022-09-13T07:34:14.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aDgZ+FFT4Pt/WYJfAokkHXFBawY=', 'Content-Length': '0'} 2022-09-13T07:34:14.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000769b0b4d9f9baecd-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.487 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:14.487 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:14.488 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.489 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.489 DEBUG:boto:Method: GET 2022-09-13T07:34:14.490 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.491 DEBUG:boto:Data: 2022-09-13T07:34:14.491 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.492 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.492 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.493 DEBUG:boto:Params: {} 2022-09-13T07:34:14.493 DEBUG:boto:Token: None 2022-09-13T07:34:14.494 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.495 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aDgZ+FFT4Pt/WYJfAokkHXFBawY= 2022-09-13T07:34:14.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aDgZ+FFT4Pt/WYJfAokkHXFBawY=', 'Content-Length': '0'} 2022-09-13T07:34:14.497 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000026389ffb863c9bb3-0063203276-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.498 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.499 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.499 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:14.500 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:14.500 DEBUG:boto:Method: GET 2022-09-13T07:34:14.501 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:34:14.501 DEBUG:boto:Data: 2022-09-13T07:34:14.502 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.502 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.503 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.503 DEBUG:boto:Params: {} 2022-09-13T07:34:14.504 DEBUG:boto:Token: None 2022-09-13T07:34:14.505 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.505 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q= 2022-09-13T07:34:14.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q=', 'Content-Length': '0'} 2022-09-13T07:34:14.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e6d335e2cdb9c45-0063203276-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.550 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.551 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.552 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.552 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:14.553 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:14.553 DEBUG:boto:Method: GET 2022-09-13T07:34:14.554 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:34:14.554 DEBUG:boto:Data: 2022-09-13T07:34:14.566 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.567 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.568 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.568 DEBUG:boto:Params: {} 2022-09-13T07:34:14.569 DEBUG:boto:Token: None 2022-09-13T07:34:14.569 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.570 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q= 2022-09-13T07:34:14.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q=', 'Content-Length': '0'} 2022-09-13T07:34:14.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b26a94ea1d058dd6-0063203276-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.574 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.575 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:14.577 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:14.577 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.578 DEBUG:boto:Path: /yxvvuh-17/dummy 2022-09-13T07:34:14.579 DEBUG:boto:Data: 2022-09-13T07:34:14.579 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.580 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.581 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.581 DEBUG:boto:Params: {} 2022-09-13T07:34:14.582 DEBUG:boto:Token: None 2022-09-13T07:34:14.583 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/dummy 2022-09-13T07:34:14.584 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fkStoxO2Yv+MGnbcyUIahIDx4HY= 2022-09-13T07:34:14.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fkStoxO2Yv+MGnbcyUIahIDx4HY=', 'Content-Length': '0'} 2022-09-13T07:34:14.588 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b509130ff9637982-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.589 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:14.590 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:14.590 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.591 DEBUG:boto:Path: /yxvvuh-17/dummy 2022-09-13T07:34:14.591 DEBUG:boto:Data: 2022-09-13T07:34:14.592 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.593 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.593 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.594 DEBUG:boto:Params: {} 2022-09-13T07:34:14.594 DEBUG:boto:Token: None 2022-09-13T07:34:14.595 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/dummy 2022-09-13T07:34:14.596 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fkStoxO2Yv+MGnbcyUIahIDx4HY= 2022-09-13T07:34:14.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fkStoxO2Yv+MGnbcyUIahIDx4HY=', 'Content-Length': '0'} 2022-09-13T07:34:14.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e0dc780e512d96d4-0063203276-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:34:14.600 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:14.601 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:14.601 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.602 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.602 DEBUG:boto:Method: GET 2022-09-13T07:34:14.603 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.604 DEBUG:boto:Data: 2022-09-13T07:34:14.604 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.605 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.605 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.606 DEBUG:boto:Params: {} 2022-09-13T07:34:14.606 DEBUG:boto:Token: None 2022-09-13T07:34:14.607 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.607 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aDgZ+FFT4Pt/WYJfAokkHXFBawY= 2022-09-13T07:34:14.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aDgZ+FFT4Pt/WYJfAokkHXFBawY=', 'Content-Length': '0'} 2022-09-13T07:34:14.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab2f3020a14747ac-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.611 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:34:14.611 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:34:14.612 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.613 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.613 DEBUG:boto:Method: GET 2022-09-13T07:34:14.614 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.614 DEBUG:boto:Data: 2022-09-13T07:34:14.615 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.615 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.616 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.617 DEBUG:boto:Params: {} 2022-09-13T07:34:14.617 DEBUG:boto:Token: None 2022-09-13T07:34:14.618 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:34:14.618 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aDgZ+FFT4Pt/WYJfAokkHXFBawY= 2022-09-13T07:34:14.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aDgZ+FFT4Pt/WYJfAokkHXFBawY=', 'Content-Length': '0'} 2022-09-13T07:34:14.621 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004efd55a5780fc9f5-0063203276-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.622 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.622 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.623 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:14.623 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:14.624 DEBUG:boto:Method: GET 2022-09-13T07:34:14.624 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:34:14.625 DEBUG:boto:Data: 2022-09-13T07:34:14.626 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.626 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.627 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.627 DEBUG:boto:Params: {} 2022-09-13T07:34:14.628 DEBUG:boto:Token: None 2022-09-13T07:34:14.628 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.629 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q= 2022-09-13T07:34:14.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q=', 'Content-Length': '0'} 2022-09-13T07:34:14.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cbe7410a4735395-0063203276-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.633 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.634 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.635 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.636 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:14.637 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:34:14.637 DEBUG:boto:Method: GET 2022-09-13T07:34:14.638 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:34:14.639 DEBUG:boto:Data: 2022-09-13T07:34:14.640 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.641 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.642 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.642 DEBUG:boto:Params: {} 2022-09-13T07:34:14.643 DEBUG:boto:Token: None 2022-09-13T07:34:14.644 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.645 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q= 2022-09-13T07:34:14.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q=', 'Content-Length': '0'} 2022-09-13T07:34:14.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032d37c4769bb3a99-0063203276-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.650 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:14.651 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.652 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.652 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.653 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.653 DEBUG:boto:Method: GET 2022-09-13T07:34:14.654 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.654 DEBUG:boto:Data: 2022-09-13T07:34:14.655 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.655 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.656 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.656 DEBUG:boto:Params: {} 2022-09-13T07:34:14.657 DEBUG:boto:Token: None 2022-09-13T07:34:14.657 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.658 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kOzxfux8sKG6RH/p47ynrTK8evA= 2022-09-13T07:34:14.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kOzxfux8sKG6RH/p47ynrTK8evA=', 'Content-Length': '0'} 2022-09-13T07:34:14.661 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c44e8c90f0ad5ce-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.661 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.662 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.662 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.663 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.663 DEBUG:boto:Method: GET 2022-09-13T07:34:14.664 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.664 DEBUG:boto:Data: 2022-09-13T07:34:14.665 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.665 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.666 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.666 DEBUG:boto:Params: {} 2022-09-13T07:34:14.667 DEBUG:boto:Token: None 2022-09-13T07:34:14.668 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.668 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kOzxfux8sKG6RH/p47ynrTK8evA= 2022-09-13T07:34:14.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kOzxfux8sKG6RH/p47ynrTK8evA=', 'Content-Length': '0'} 2022-09-13T07:34:14.671 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d81cca4533da48e-0063203276-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.672 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.672 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.673 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.673 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.674 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.674 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.675 DEBUG:boto:Data: 2022-09-13T07:34:14.675 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.676 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.677 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.678 DEBUG:boto:Params: {} 2022-09-13T07:34:14.679 DEBUG:boto:Token: None 2022-09-13T07:34:14.680 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.681 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iNdTku2VeBjm5mu32V/EaUW+vzg= 2022-09-13T07:34:14.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iNdTku2VeBjm5mu32V/EaUW+vzg=', 'Content-Length': '0'} 2022-09-13T07:34:14.684 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008afee3cd1c421a5f-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.685 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.685 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.686 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.687 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.687 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.688 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.688 DEBUG:boto:Data: 2022-09-13T07:34:14.689 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.689 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.690 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.690 DEBUG:boto:Params: {} 2022-09-13T07:34:14.691 DEBUG:boto:Token: None 2022-09-13T07:34:14.691 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.692 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iNdTku2VeBjm5mu32V/EaUW+vzg= 2022-09-13T07:34:14.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iNdTku2VeBjm5mu32V/EaUW+vzg=', 'Content-Length': '0'} 2022-09-13T07:34:14.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039814b8d4909fdaf-0063203276-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:14.696 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.697 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.697 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.698 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.699 DEBUG:boto:Method: GET 2022-09-13T07:34:14.699 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.700 DEBUG:boto:Data: 2022-09-13T07:34:14.700 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.701 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.701 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.702 DEBUG:boto:Params: {} 2022-09-13T07:34:14.702 DEBUG:boto:Token: None 2022-09-13T07:34:14.703 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.703 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kOzxfux8sKG6RH/p47ynrTK8evA= 2022-09-13T07:34:14.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kOzxfux8sKG6RH/p47ynrTK8evA=', 'Content-Length': '0'} 2022-09-13T07:34:14.706 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc8cf3c5a1809e6c-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.707 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.708 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.708 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.709 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.709 DEBUG:boto:Method: GET 2022-09-13T07:34:14.710 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.710 DEBUG:boto:Data: 2022-09-13T07:34:14.711 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.711 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.712 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.712 DEBUG:boto:Params: {} 2022-09-13T07:34:14.713 DEBUG:boto:Token: None 2022-09-13T07:34:14.714 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.714 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kOzxfux8sKG6RH/p47ynrTK8evA= 2022-09-13T07:34:14.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kOzxfux8sKG6RH/p47ynrTK8evA=', 'Content-Length': '0'} 2022-09-13T07:34:14.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000deb9f5828a6675d1-0063203276-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.758 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.759 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.760 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:14.761 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:14.762 DEBUG:boto:Method: GET 2022-09-13T07:34:14.763 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:14.764 DEBUG:boto:Data: 2022-09-13T07:34:14.765 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.766 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.767 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.768 DEBUG:boto:Params: {} 2022-09-13T07:34:14.769 DEBUG:boto:Token: None 2022-09-13T07:34:14.770 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.771 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q= 2022-09-13T07:34:14.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q=', 'Content-Length': '0'} 2022-09-13T07:34:14.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f02eb439d3bd8f44-0063203276-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.777 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.778 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.779 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.780 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:14.781 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:14.782 DEBUG:boto:Method: GET 2022-09-13T07:34:14.783 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:14.785 DEBUG:boto:Data: 2022-09-13T07:34:14.785 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.786 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.786 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.787 DEBUG:boto:Params: {} 2022-09-13T07:34:14.787 DEBUG:boto:Token: None 2022-09-13T07:34:14.788 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.789 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q= 2022-09-13T07:34:14.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q=', 'Content-Length': '0'} 2022-09-13T07:34:14.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001144740f852dce57-0063203276-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.793 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.794 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.794 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.795 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.795 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.796 DEBUG:boto:Data: 2022-09-13T07:34:14.796 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.797 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.797 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.798 DEBUG:boto:Params: {} 2022-09-13T07:34:14.798 DEBUG:boto:Token: None 2022-09-13T07:34:14.799 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.799 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NR9JEK+1N2Z/KhP1lE4Z9A4Hr2M= 2022-09-13T07:34:14.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NR9JEK+1N2Z/KhP1lE4Z9A4Hr2M=', 'Content-Length': '0'} 2022-09-13T07:34:14.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e71026aa0e7d4ce-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.803 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.804 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.805 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.806 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.806 DEBUG:boto:Data: 2022-09-13T07:34:14.807 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.807 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.808 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.808 DEBUG:boto:Params: {} 2022-09-13T07:34:14.809 DEBUG:boto:Token: None 2022-09-13T07:34:14.809 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.810 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NR9JEK+1N2Z/KhP1lE4Z9A4Hr2M= 2022-09-13T07:34:14.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NR9JEK+1N2Z/KhP1lE4Z9A4Hr2M=', 'Content-Length': '0'} 2022-09-13T07:34:14.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007555657e1da09104-0063203276-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:34:14.814 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.815 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.816 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.816 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.817 DEBUG:boto:Method: GET 2022-09-13T07:34:14.817 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.818 DEBUG:boto:Data: 2022-09-13T07:34:14.818 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.819 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.819 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.820 DEBUG:boto:Params: {} 2022-09-13T07:34:14.820 DEBUG:boto:Token: None 2022-09-13T07:34:14.821 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.822 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kOzxfux8sKG6RH/p47ynrTK8evA= 2022-09-13T07:34:14.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kOzxfux8sKG6RH/p47ynrTK8evA=', 'Content-Length': '0'} 2022-09-13T07:34:14.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d25bf422dc85e361-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.866 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.867 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:34:14.867 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.868 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.868 DEBUG:boto:Method: GET 2022-09-13T07:34:14.869 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.869 DEBUG:boto:Data: 2022-09-13T07:34:14.870 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.871 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.871 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.872 DEBUG:boto:Params: {} 2022-09-13T07:34:14.872 DEBUG:boto:Token: None 2022-09-13T07:34:14.873 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:34:14.874 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kOzxfux8sKG6RH/p47ynrTK8evA= 2022-09-13T07:34:14.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kOzxfux8sKG6RH/p47ynrTK8evA=', 'Content-Length': '0'} 2022-09-13T07:34:14.876 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:06 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009b1d9fc4ff5426f-0063203276-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.877 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.878 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.879 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:14.880 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:14.880 DEBUG:boto:Method: GET 2022-09-13T07:34:14.881 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:14.882 DEBUG:boto:Data: 2022-09-13T07:34:14.883 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.884 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.885 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.885 DEBUG:boto:Params: {} 2022-09-13T07:34:14.886 DEBUG:boto:Token: None 2022-09-13T07:34:14.887 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.888 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q= 2022-09-13T07:34:14.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q=', 'Content-Length': '0'} 2022-09-13T07:34:14.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000092bc1138a731531-0063203276-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.892 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.893 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:14.893 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:14.894 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:14.894 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:14.895 DEBUG:boto:Method: GET 2022-09-13T07:34:14.895 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:34:14.896 DEBUG:boto:Data: 2022-09-13T07:34:14.896 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.897 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.897 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.898 DEBUG:boto:Params: {} 2022-09-13T07:34:14.899 DEBUG:boto:Token: None 2022-09-13T07:34:14.900 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/ 2022-09-13T07:34:14.900 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q= 2022-09-13T07:34:14.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/nbW8b6jQKOyxOMJeuCW/7q/0Q=', 'Content-Length': '0'} 2022-09-13T07:34:14.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f96a5f0d0766f2ba-0063203276-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.905 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:14.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:14.906 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:14.907 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:14.907 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.908 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.908 DEBUG:boto:Method: GET 2022-09-13T07:34:14.909 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.909 DEBUG:boto:Data: 2022-09-13T07:34:14.910 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.911 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.911 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.912 DEBUG:boto:Params: {} 2022-09-13T07:34:14.912 DEBUG:boto:Token: None 2022-09-13T07:34:14.913 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.914 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2r7eJbEVhwT7OGAClF62KbGEmic= 2022-09-13T07:34:14.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2r7eJbEVhwT7OGAClF62KbGEmic=', 'Content-Length': '0'} 2022-09-13T07:34:14.917 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013779642d2760dfc-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.918 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:14.919 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:14.920 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.921 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.922 DEBUG:boto:Method: GET 2022-09-13T07:34:14.923 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.924 DEBUG:boto:Data: 2022-09-13T07:34:14.925 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.926 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.926 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.927 DEBUG:boto:Params: {} 2022-09-13T07:34:14.929 DEBUG:boto:Token: None 2022-09-13T07:34:14.930 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.931 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2r7eJbEVhwT7OGAClF62KbGEmic= 2022-09-13T07:34:14.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2r7eJbEVhwT7OGAClF62KbGEmic=', 'Content-Length': '0'} 2022-09-13T07:34:14.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a76a176daeaee31-0063203276-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.936 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:14.937 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:14.938 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.939 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.940 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.941 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.941 DEBUG:boto:Data: 2022-09-13T07:34:14.942 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.943 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.944 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.944 DEBUG:boto:Params: {} 2022-09-13T07:34:14.945 DEBUG:boto:Token: None 2022-09-13T07:34:14.946 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.947 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qPl1y0BukZu80kGSKCCZX3jv8Z4= 2022-09-13T07:34:14.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qPl1y0BukZu80kGSKCCZX3jv8Z4=', 'Content-Length': '0'} 2022-09-13T07:34:14.949 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce1c24687ac7639c-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.950 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:14.951 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:14.951 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.952 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.952 DEBUG:boto:Method: HEAD 2022-09-13T07:34:14.953 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.953 DEBUG:boto:Data: 2022-09-13T07:34:14.954 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.954 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.955 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.956 DEBUG:boto:Params: {} 2022-09-13T07:34:14.957 DEBUG:boto:Token: None 2022-09-13T07:34:14.958 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.959 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qPl1y0BukZu80kGSKCCZX3jv8Z4= 2022-09-13T07:34:14.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qPl1y0BukZu80kGSKCCZX3jv8Z4=', 'Content-Length': '0'} 2022-09-13T07:34:14.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5e6aae51e235cf7-0063203276-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:14.965 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:14.966 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:14.966 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.967 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.968 DEBUG:boto:Method: GET 2022-09-13T07:34:14.968 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.969 DEBUG:boto:Data: 2022-09-13T07:34:14.969 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.970 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:14.970 DEBUG:boto:Port: 8001 2022-09-13T07:34:14.971 DEBUG:boto:Params: {} 2022-09-13T07:34:14.971 DEBUG:boto:Token: None 2022-09-13T07:34:14.972 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.973 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2r7eJbEVhwT7OGAClF62KbGEmic= 2022-09-13T07:34:14.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2r7eJbEVhwT7OGAClF62KbGEmic=', 'Content-Length': '0'} 2022-09-13T07:34:14.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d6d8b4acff266fc-0063203276-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:14.977 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:14.978 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:14.979 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.980 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.981 DEBUG:boto:Method: GET 2022-09-13T07:34:14.982 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.983 DEBUG:boto:Data: 2022-09-13T07:34:14.984 DEBUG:boto:Headers: {} 2022-09-13T07:34:14.985 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:14.986 DEBUG:boto:Port: 8000 2022-09-13T07:34:14.987 DEBUG:boto:Params: {} 2022-09-13T07:34:14.988 DEBUG:boto:Token: None 2022-09-13T07:34:14.989 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:14 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:14.990 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2r7eJbEVhwT7OGAClF62KbGEmic= 2022-09-13T07:34:14.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2r7eJbEVhwT7OGAClF62KbGEmic=', 'Content-Length': '0'} 2022-09-13T07:34:14.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000721f92d3a54468f0-0063203276-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.038 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:15.039 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:15.039 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:15.040 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:15.041 DEBUG:boto:Method: GET 2022-09-13T07:34:15.041 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:15.042 DEBUG:boto:Data: 2022-09-13T07:34:15.043 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.043 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:15.044 DEBUG:boto:Port: 8001 2022-09-13T07:34:15.044 DEBUG:boto:Params: {} 2022-09-13T07:34:15.045 DEBUG:boto:Token: None 2022-09-13T07:34:15.046 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/ 2022-09-13T07:34:15.046 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U= 2022-09-13T07:34:15.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U=', 'Content-Length': '0'} 2022-09-13T07:34:15.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007134e61b4b0c52d0-0063203277-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.050 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:15.051 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:15.052 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:15.052 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:15.053 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:15.053 DEBUG:boto:Method: GET 2022-09-13T07:34:15.054 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:15.055 DEBUG:boto:Data: 2022-09-13T07:34:15.056 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.056 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:15.057 DEBUG:boto:Port: 8000 2022-09-13T07:34:15.058 DEBUG:boto:Params: {} 2022-09-13T07:34:15.059 DEBUG:boto:Token: None 2022-09-13T07:34:15.059 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/ 2022-09-13T07:34:15.060 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U= 2022-09-13T07:34:15.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U=', 'Content-Length': '0'} 2022-09-13T07:34:15.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dedd21db9452b8e-0063203277-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.066 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:15.066 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:15.067 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:15.068 DEBUG:boto:Method: HEAD 2022-09-13T07:34:15.068 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2 2022-09-13T07:34:15.069 DEBUG:boto:Data: 2022-09-13T07:34:15.070 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.070 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:15.071 DEBUG:boto:Port: 8001 2022-09-13T07:34:15.071 DEBUG:boto:Params: {} 2022-09-13T07:34:15.072 DEBUG:boto:Token: None 2022-09-13T07:34:15.072 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/obj-test-zone2 2022-09-13T07:34:15.073 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3IBDQfNIutZv4WDB+0wPUDuC+5s= 2022-09-13T07:34:15.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3IBDQfNIutZv4WDB+0wPUDuC+5s=', 'Content-Length': '0'} 2022-09-13T07:34:15.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e21342a534af6cb9-0063203277-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.077 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:15.078 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:15.078 DEBUG:boto:Method: HEAD 2022-09-13T07:34:15.079 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2 2022-09-13T07:34:15.079 DEBUG:boto:Data: 2022-09-13T07:34:15.080 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.080 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:15.081 DEBUG:boto:Port: 8000 2022-09-13T07:34:15.081 DEBUG:boto:Params: {} 2022-09-13T07:34:15.082 DEBUG:boto:Token: None 2022-09-13T07:34:15.082 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/obj-test-zone2 2022-09-13T07:34:15.083 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3IBDQfNIutZv4WDB+0wPUDuC+5s= 2022-09-13T07:34:15.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3IBDQfNIutZv4WDB+0wPUDuC+5s=', 'Content-Length': '0'} 2022-09-13T07:34:15.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8a68295f4f84222-0063203277-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:34:15.087 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:15.088 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:15.088 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:15.089 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:15.089 DEBUG:boto:Method: GET 2022-09-13T07:34:15.090 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:15.090 DEBUG:boto:Data: 2022-09-13T07:34:15.091 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.091 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:15.092 DEBUG:boto:Port: 8001 2022-09-13T07:34:15.092 DEBUG:boto:Params: {} 2022-09-13T07:34:15.093 DEBUG:boto:Token: None 2022-09-13T07:34:15.093 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:15.094 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hzHNqya0l8x2TaND5YDPpA+zKoQ= 2022-09-13T07:34:15.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hzHNqya0l8x2TaND5YDPpA+zKoQ=', 'Content-Length': '0'} 2022-09-13T07:34:15.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001431b9134816ddb-0063203277-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.097 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:15.098 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:34:15.098 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:15.099 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:15.099 DEBUG:boto:Method: GET 2022-09-13T07:34:15.100 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:15.100 DEBUG:boto:Data: 2022-09-13T07:34:15.101 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.101 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:15.102 DEBUG:boto:Port: 8000 2022-09-13T07:34:15.102 DEBUG:boto:Params: {} 2022-09-13T07:34:15.103 DEBUG:boto:Token: None 2022-09-13T07:34:15.104 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:34:15.104 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hzHNqya0l8x2TaND5YDPpA+zKoQ= 2022-09-13T07:34:15.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hzHNqya0l8x2TaND5YDPpA+zKoQ=', 'Content-Length': '0'} 2022-09-13T07:34:15.107 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6453989dc77ac2a-0063203277-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.108 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:15.108 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:15.109 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:15.109 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:15.110 DEBUG:boto:Method: GET 2022-09-13T07:34:15.110 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:15.111 DEBUG:boto:Data: 2022-09-13T07:34:15.111 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.112 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:15.112 DEBUG:boto:Port: 8001 2022-09-13T07:34:15.113 DEBUG:boto:Params: {} 2022-09-13T07:34:15.113 DEBUG:boto:Token: None 2022-09-13T07:34:15.114 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/ 2022-09-13T07:34:15.114 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U= 2022-09-13T07:34:15.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U=', 'Content-Length': '0'} 2022-09-13T07:34:15.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048d608f714764db4-0063203277-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.118 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:15.119 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:15.119 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:15.120 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:15.120 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:15.121 DEBUG:boto:Method: GET 2022-09-13T07:34:15.122 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:34:15.122 DEBUG:boto:Data: 2022-09-13T07:34:15.123 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.123 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:15.124 DEBUG:boto:Port: 8000 2022-09-13T07:34:15.124 DEBUG:boto:Params: {} 2022-09-13T07:34:15.125 DEBUG:boto:Token: None 2022-09-13T07:34:15.125 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/ 2022-09-13T07:34:15.126 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U= 2022-09-13T07:34:15.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U=', 'Content-Length': '0'} 2022-09-13T07:34:15.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0cf5e5339387f8a-0063203277-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.174 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:15.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:15.176 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.176 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.177 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.177 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.178 DEBUG:boto:Method: GET 2022-09-13T07:34:15.178 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.179 DEBUG:boto:Data: 2022-09-13T07:34:15.179 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.180 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:15.180 DEBUG:boto:Port: 8001 2022-09-13T07:34:15.181 DEBUG:boto:Params: {} 2022-09-13T07:34:15.181 DEBUG:boto:Token: None 2022-09-13T07:34:15.182 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.183 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zz4GCjUdsddTZO8/zpnXmeo6OFk= 2022-09-13T07:34:15.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zz4GCjUdsddTZO8/zpnXmeo6OFk=', 'Content-Length': '0'} 2022-09-13T07:34:15.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c4ce9c129db0c1e-0063203277-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.186 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.187 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.187 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.188 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.188 DEBUG:boto:Method: GET 2022-09-13T07:34:15.189 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.189 DEBUG:boto:Data: 2022-09-13T07:34:15.190 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.190 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:15.191 DEBUG:boto:Port: 8000 2022-09-13T07:34:15.191 DEBUG:boto:Params: {} 2022-09-13T07:34:15.192 DEBUG:boto:Token: None 2022-09-13T07:34:15.193 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.193 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zz4GCjUdsddTZO8/zpnXmeo6OFk= 2022-09-13T07:34:15.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zz4GCjUdsddTZO8/zpnXmeo6OFk=', 'Content-Length': '0'} 2022-09-13T07:34:15.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099085b14830839f0-0063203277-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.197 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.197 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.199 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.199 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.200 DEBUG:boto:Method: HEAD 2022-09-13T07:34:15.200 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.201 DEBUG:boto:Data: 2022-09-13T07:34:15.201 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.202 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:15.202 DEBUG:boto:Port: 8001 2022-09-13T07:34:15.203 DEBUG:boto:Params: {} 2022-09-13T07:34:15.203 DEBUG:boto:Token: None 2022-09-13T07:34:15.204 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.204 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fsoC+QdWx9qIBt5dZJ8LlEghCCE= 2022-09-13T07:34:15.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fsoC+QdWx9qIBt5dZJ8LlEghCCE=', 'Content-Length': '0'} 2022-09-13T07:34:15.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb652acff0299d88-0063203277-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.208 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.208 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.209 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.209 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.210 DEBUG:boto:Method: HEAD 2022-09-13T07:34:15.211 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.211 DEBUG:boto:Data: 2022-09-13T07:34:15.211 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.212 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:15.212 DEBUG:boto:Port: 8000 2022-09-13T07:34:15.213 DEBUG:boto:Params: {} 2022-09-13T07:34:15.213 DEBUG:boto:Token: None 2022-09-13T07:34:15.214 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.215 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fsoC+QdWx9qIBt5dZJ8LlEghCCE= 2022-09-13T07:34:15.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fsoC+QdWx9qIBt5dZJ8LlEghCCE=', 'Content-Length': '0'} 2022-09-13T07:34:15.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b51953844f179ea-0063203277-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:15.218 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.219 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.219 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.220 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.221 DEBUG:boto:Method: GET 2022-09-13T07:34:15.221 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.222 DEBUG:boto:Data: 2022-09-13T07:34:15.222 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.223 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:15.223 DEBUG:boto:Port: 8001 2022-09-13T07:34:15.224 DEBUG:boto:Params: {} 2022-09-13T07:34:15.224 DEBUG:boto:Token: None 2022-09-13T07:34:15.225 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.226 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zz4GCjUdsddTZO8/zpnXmeo6OFk= 2022-09-13T07:34:15.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zz4GCjUdsddTZO8/zpnXmeo6OFk=', 'Content-Length': '0'} 2022-09-13T07:34:15.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033108871b89a7ea1-0063203277-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.270 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.271 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.271 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.272 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.273 DEBUG:boto:Method: GET 2022-09-13T07:34:15.273 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.274 DEBUG:boto:Data: 2022-09-13T07:34:15.274 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.275 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:15.275 DEBUG:boto:Port: 8000 2022-09-13T07:34:15.276 DEBUG:boto:Params: {} 2022-09-13T07:34:15.276 DEBUG:boto:Token: None 2022-09-13T07:34:15.277 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.277 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zz4GCjUdsddTZO8/zpnXmeo6OFk= 2022-09-13T07:34:15.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zz4GCjUdsddTZO8/zpnXmeo6OFk=', 'Content-Length': '0'} 2022-09-13T07:34:15.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b939b173907e3111-0063203277-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.281 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:15.281 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:15.282 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:15.283 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:15.283 DEBUG:boto:Method: GET 2022-09-13T07:34:15.284 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:15.284 DEBUG:boto:Data: 2022-09-13T07:34:15.285 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.285 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:15.286 DEBUG:boto:Port: 8001 2022-09-13T07:34:15.286 DEBUG:boto:Params: {} 2022-09-13T07:34:15.287 DEBUG:boto:Token: None 2022-09-13T07:34:15.287 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/ 2022-09-13T07:34:15.288 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U= 2022-09-13T07:34:15.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U=', 'Content-Length': '0'} 2022-09-13T07:34:15.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008544dcb5c2db0134-0063203277-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.292 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:15.292 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:15.293 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:15.293 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:15.294 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:15.294 DEBUG:boto:Method: GET 2022-09-13T07:34:15.295 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:15.295 DEBUG:boto:Data: 2022-09-13T07:34:15.296 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.296 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:15.297 DEBUG:boto:Port: 8000 2022-09-13T07:34:15.297 DEBUG:boto:Params: {} 2022-09-13T07:34:15.298 DEBUG:boto:Token: None 2022-09-13T07:34:15.299 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/ 2022-09-13T07:34:15.300 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U= 2022-09-13T07:34:15.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U=', 'Content-Length': '0'} 2022-09-13T07:34:15.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a56e1f932c1724b-0063203277-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.303 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:15.304 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.305 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.305 DEBUG:boto:Method: HEAD 2022-09-13T07:34:15.306 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.306 DEBUG:boto:Data: 2022-09-13T07:34:15.307 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.307 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:15.308 DEBUG:boto:Port: 8001 2022-09-13T07:34:15.308 DEBUG:boto:Params: {} 2022-09-13T07:34:15.309 DEBUG:boto:Token: None 2022-09-13T07:34:15.309 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.310 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8AXQ+Q7I5ICw8SAhhdpeK+l4pek= 2022-09-13T07:34:15.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8AXQ+Q7I5ICw8SAhhdpeK+l4pek=', 'Content-Length': '0'} 2022-09-13T07:34:15.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034a9abee8c0777c1-0063203277-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.314 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.314 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.315 DEBUG:boto:Method: HEAD 2022-09-13T07:34:15.315 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.316 DEBUG:boto:Data: 2022-09-13T07:34:15.316 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.317 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:15.317 DEBUG:boto:Port: 8000 2022-09-13T07:34:15.318 DEBUG:boto:Params: {} 2022-09-13T07:34:15.318 DEBUG:boto:Token: None 2022-09-13T07:34:15.319 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.319 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8AXQ+Q7I5ICw8SAhhdpeK+l4pek= 2022-09-13T07:34:15.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8AXQ+Q7I5ICw8SAhhdpeK+l4pek=', 'Content-Length': '0'} 2022-09-13T07:34:15.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ede0c12ec5f9494-0063203277-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:34:15.324 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.324 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.325 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.325 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.326 DEBUG:boto:Method: GET 2022-09-13T07:34:15.326 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.327 DEBUG:boto:Data: 2022-09-13T07:34:15.327 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.328 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:15.328 DEBUG:boto:Port: 8001 2022-09-13T07:34:15.329 DEBUG:boto:Params: {} 2022-09-13T07:34:15.329 DEBUG:boto:Token: None 2022-09-13T07:34:15.330 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.331 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zz4GCjUdsddTZO8/zpnXmeo6OFk= 2022-09-13T07:34:15.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zz4GCjUdsddTZO8/zpnXmeo6OFk=', 'Content-Length': '0'} 2022-09-13T07:34:15.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009f8dcdbbaf475d3-0063203277-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.335 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.335 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:34:15.336 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.336 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.337 DEBUG:boto:Method: GET 2022-09-13T07:34:15.337 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.338 DEBUG:boto:Data: 2022-09-13T07:34:15.338 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.339 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:15.339 DEBUG:boto:Port: 8000 2022-09-13T07:34:15.340 DEBUG:boto:Params: {} 2022-09-13T07:34:15.340 DEBUG:boto:Token: None 2022-09-13T07:34:15.341 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:34:15.341 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zz4GCjUdsddTZO8/zpnXmeo6OFk= 2022-09-13T07:34:15.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zz4GCjUdsddTZO8/zpnXmeo6OFk=', 'Content-Length': '0'} 2022-09-13T07:34:15.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e3a533af57496e4-0063203277-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.390 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:15.391 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:15.392 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:15.393 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:15.393 DEBUG:boto:Method: GET 2022-09-13T07:34:15.394 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:15.395 DEBUG:boto:Data: 2022-09-13T07:34:15.396 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.397 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:34:15.398 DEBUG:boto:Port: 8001 2022-09-13T07:34:15.399 DEBUG:boto:Params: {} 2022-09-13T07:34:15.400 DEBUG:boto:Token: None 2022-09-13T07:34:15.401 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/ 2022-09-13T07:34:15.402 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U= 2022-09-13T07:34:15.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U=', 'Content-Length': '0'} 2022-09-13T07:34:15.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b37d4ff30231065-0063203277-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.407 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:15.408 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:34:15.409 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:34:15.409 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:15.410 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:15.410 DEBUG:boto:Method: GET 2022-09-13T07:34:15.411 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:34:15.412 DEBUG:boto:Data: 2022-09-13T07:34:15.412 DEBUG:boto:Headers: {} 2022-09-13T07:34:15.413 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:34:15.413 DEBUG:boto:Port: 8000 2022-09-13T07:34:15.414 DEBUG:boto:Params: {} 2022-09-13T07:34:15.415 DEBUG:boto:Token: None 2022-09-13T07:34:15.415 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:34:15 GMT /yxvvuh-17/ 2022-09-13T07:34:15.416 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U= 2022-09-13T07:34:15.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:34:15 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j/BhfM4jpM/2BniTuwHGMV7GY9U=', 'Content-Length': '0'} 2022-09-13T07:34:15.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f057c02a60d7e77-0063203277-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:34:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:34:15.420 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:34:15.420 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-17 zones={test-zone3, test-zone2} 2022-09-13T07:34:15.421 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-18 2022-09-13T07:34:15.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:15.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:15.423 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:34:15.486 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:34:15.487 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:34:15.527 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:34:15.523+0000 7fb9586ce600 1 bucket sync caught up with source: 2022-09-13T07:34:15.527 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000010.24.5, , 00000000004.17.9, , 00000000007.35.9, , , , , ] 2022-09-13T07:34:15.528 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000010.24.5, , 00000000004.17.9, , 00000000007.35.9, , , , , ] 2022-09-13T07:34:15.528 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:34:15.523+0000 7fb9586ce600 0 bucket checkpoint complete 2022-09-13T07:34:15.539 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-18 2022-09-13T07:34:15.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:15.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:15.541 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:34:15.607 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:34:15.608 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:34:15.655 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:15.652+0000 7f2eec572600 1 bucket sync caught up with source: 2022-09-13T07:34:15.655 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000010.24.5, , 00000000004.17.9, , 00000000007.35.9, , , , , ] 2022-09-13T07:34:15.655 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000010.24.5, , 00000000004.17.9, , 00000000007.35.9, , , , , ] 2022-09-13T07:34:15.656 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:15.652+0000 7f2eec572600 0 bucket checkpoint complete 2022-09-13T07:34:15.673 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-18 2022-09-13T07:34:15.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:15.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:15.675 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:34:15.733 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:34:15.734 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:34:15.781 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:15.778+0000 7fe694f19600 1 bucket sync caught up with source: 2022-09-13T07:34:15.781 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.19.9, , 00000000010.36.5, , 00000000007.44.9, , , , , ] 2022-09-13T07:34:15.782 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.19.9, , 00000000010.36.5, , 00000000007.44.9, , , , , ] 2022-09-13T07:34:15.782 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:15.778+0000 7fe694f19600 0 bucket checkpoint complete 2022-09-13T07:34:15.794 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-18 2022-09-13T07:34:15.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:15.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:15.797 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:34:15.854 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:34:15.855 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:34:15.896 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:15.893+0000 7f05c3c93600 1 bucket sync caught up with source: 2022-09-13T07:34:15.896 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.19.9, , 00000000010.36.5, , 00000000007.44.9, , , , , ] 2022-09-13T07:34:15.897 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.19.9, , 00000000010.36.5, , 00000000007.44.9, , , , , ] 2022-09-13T07:34:15.897 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:15.893+0000 7f05c3c93600 0 bucket checkpoint complete 2022-09-13T07:34:15.912 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-18 2022-09-13T07:34:15.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:15.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:34:15.915 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:34:15.973 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:34:15.973 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:34:16.020 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:16.017+0000 7fed61b47600 1 waiting for incremental sync to catch up: 2022-09-13T07:34:16.021 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.20.9, , 00000000004.22.9, , 00000000010.60.11, , , , , ] 2022-09-13T07:34:16.021 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.20.9, , 00000000004.22.9, , 00000000016.72.9, , , , , ] 2022-09-13T07:34:21.022 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:21.019+0000 7fed61b47600 1 waiting for incremental sync to catch up: 2022-09-13T07:34:21.022 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.20.9, , 00000000004.22.9, , 00000000010.60.11, , , , , ] 2022-09-13T07:34:21.023 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.20.9, , 00000000004.22.9, , 00000000016.72.9, , , , , ] 2022-09-13T07:34:26.024 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:26.020+0000 7fed61b47600 1 waiting for incremental sync to catch up: 2022-09-13T07:34:26.025 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.20.9, , 00000000004.22.9, , 00000000010.60.11, , , , , ] 2022-09-13T07:34:26.025 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.20.9, , 00000000004.22.9, , 00000000016.72.9, , , , , ] 2022-09-13T07:34:31.025 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:31.021+0000 7fed61b47600 1 waiting for incremental sync to catch up: 2022-09-13T07:34:31.025 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.20.9, , 00000000004.22.9, , 00000000010.60.11, , , , , ] 2022-09-13T07:34:31.026 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.20.9, , 00000000004.22.9, , 00000000016.72.9, , , , , ] 2022-09-13T07:34:36.027 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:36.023+0000 7fed61b47600 1 waiting for incremental sync to catch up: 2022-09-13T07:34:36.027 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.20.9, , 00000000004.22.9, , 00000000010.60.11, , , , , ] 2022-09-13T07:34:36.027 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.20.9, , 00000000004.22.9, , 00000000016.72.9, , , , , ] 2022-09-13T07:34:41.028 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:41.024+0000 7fed61b47600 1 waiting for incremental sync to catch up: 2022-09-13T07:34:41.029 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.20.9, , 00000000004.22.9, , 00000000010.60.11, , , , , ] 2022-09-13T07:34:41.030 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.20.9, , 00000000004.22.9, , 00000000016.72.9, , , , , ] 2022-09-13T07:34:46.030 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:46.026+0000 7fed61b47600 1 waiting for incremental sync to catch up: 2022-09-13T07:34:46.031 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.20.9, , 00000000004.22.9, , 00000000010.60.11, , , , , ] 2022-09-13T07:34:46.031 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.20.9, , 00000000004.22.9, , 00000000016.72.9, , , , , ] 2022-09-13T07:34:51.032 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:51.028+0000 7fed61b47600 1 waiting for incremental sync to catch up: 2022-09-13T07:34:51.033 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.20.9, , 00000000004.22.9, , 00000000010.60.11, , , , , ] 2022-09-13T07:34:51.034 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.20.9, , 00000000004.22.9, , 00000000016.72.9, , , , , ] 2022-09-13T07:34:56.034 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:34:56.029+0000 7fed61b47600 1 waiting for incremental sync to catch up: 2022-09-13T07:34:56.034 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.20.9, , 00000000004.22.9, , 00000000010.60.11, , , , , ] 2022-09-13T07:34:56.035 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.20.9, , 00000000004.22.9, , 00000000016.72.9, , , , , ] 2022-09-13T07:35:01.035 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:01.031+0000 7fed61b47600 1 waiting for incremental sync to catch up: 2022-09-13T07:35:01.035 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.20.9, , 00000000004.22.9, , 00000000010.60.11, , , , , ] 2022-09-13T07:35:01.036 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.20.9, , 00000000004.22.9, , 00000000016.72.9, , , , , ] 2022-09-13T07:35:06.037 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:06.033+0000 7fed61b47600 1 waiting for incremental sync to catch up: 2022-09-13T07:35:06.037 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.20.9, , 00000000004.22.9, , 00000000010.60.11, , , , , ] 2022-09-13T07:35:06.038 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.20.9, , 00000000004.22.9, , 00000000016.72.9, , , , , ] 2022-09-13T07:35:11.038 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:11.034+0000 7fed61b47600 1 bucket sync caught up with source: 2022-09-13T07:35:11.039 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.20.9, , 00000000004.22.9, , 00000000016.72.9, , , , , ] 2022-09-13T07:35:11.039 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000003.20.9, , 00000000004.22.9, , 00000000016.72.9, , , , , ] 2022-09-13T07:35:11.040 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:11.034+0000 7fed61b47600 0 bucket checkpoint complete 2022-09-13T07:35:11.066 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-18 2022-09-13T07:35:11.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:11.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:11.068 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:35:11.132 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:35:11.133 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:35:11.185 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:35:11.181+0000 7f6530549600 1 bucket sync caught up with source: 2022-09-13T07:35:11.186 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000003.20.9, , 00000000004.22.9, , 00000000016.72.9, , , , , ] 2022-09-13T07:35:11.186 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000003.20.9, , 00000000004.22.9, , 00000000016.72.9, , , , , ] 2022-09-13T07:35:11.186 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:35:11.181+0000 7f6530549600 0 bucket checkpoint complete 2022-09-13T07:35:11.205 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-18 zones={test-zone2, test-zone1} 2022-09-13T07:35:11.206 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.207 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.207 DEBUG:boto:Method: HEAD 2022-09-13T07:35:11.208 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:35:11.209 DEBUG:boto:Data: 2022-09-13T07:35:11.210 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.210 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.211 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.211 DEBUG:boto:Params: {} 2022-09-13T07:35:11.212 DEBUG:boto:Token: None 2022-09-13T07:35:11.213 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/ 2022-09-13T07:35:11.213 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jasAbK+Q2WkZBTUB4dH9SLdm1lM= 2022-09-13T07:35:11.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jasAbK+Q2WkZBTUB4dH9SLdm1lM=', 'Content-Length': '0'} 2022-09-13T07:35:11.217 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8d9945b25d33448-00632032af-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.218 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.219 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.220 DEBUG:boto:Method: HEAD 2022-09-13T07:35:11.221 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:35:11.222 DEBUG:boto:Data: 2022-09-13T07:35:11.223 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.224 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.225 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.226 DEBUG:boto:Params: {} 2022-09-13T07:35:11.227 DEBUG:boto:Token: None 2022-09-13T07:35:11.228 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/ 2022-09-13T07:35:11.229 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jasAbK+Q2WkZBTUB4dH9SLdm1lM= 2022-09-13T07:35:11.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jasAbK+Q2WkZBTUB4dH9SLdm1lM=', 'Content-Length': '0'} 2022-09-13T07:35:11.233 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011dd94b38d252a5c-00632032af-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.234 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:35:11.235 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.236 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.237 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:11.237 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:11.238 DEBUG:boto:Method: GET 2022-09-13T07:35:11.238 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:11.239 DEBUG:boto:Data: 2022-09-13T07:35:11.239 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.240 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.240 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.241 DEBUG:boto:Params: {} 2022-09-13T07:35:11.241 DEBUG:boto:Token: None 2022-09-13T07:35:11.242 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/?versions 2022-09-13T07:35:11.243 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Zl2FzyDWvCkew0yGFb/RQXP9IGQ= 2022-09-13T07:35:11.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Zl2FzyDWvCkew0yGFb/RQXP9IGQ=', 'Content-Length': '0'} 2022-09-13T07:35:11.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de6c98d15d9ad5b2-00632032af-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.247 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:11.249 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:35:11.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:35:11.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:35:11.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:35:11.253 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:35:11.254 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.255 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.256 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:11.256 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:11.257 DEBUG:boto:Method: GET 2022-09-13T07:35:11.257 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:11.258 DEBUG:boto:Data: 2022-09-13T07:35:11.258 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.259 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.259 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.260 DEBUG:boto:Params: {} 2022-09-13T07:35:11.260 DEBUG:boto:Token: None 2022-09-13T07:35:11.261 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/?versions 2022-09-13T07:35:11.261 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Zl2FzyDWvCkew0yGFb/RQXP9IGQ= 2022-09-13T07:35:11.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Zl2FzyDWvCkew0yGFb/RQXP9IGQ=', 'Content-Length': '0'} 2022-09-13T07:35:11.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000277fd87956bb29c9-00632032af-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.266 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:11.267 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:35:11.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:35:11.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:35:11.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:35:11.269 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.269 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.270 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:11.270 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:11.271 DEBUG:boto:Method: GET 2022-09-13T07:35:11.271 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:11.272 DEBUG:boto:Data: 2022-09-13T07:35:11.272 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.273 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.273 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.274 DEBUG:boto:Params: {} 2022-09-13T07:35:11.274 DEBUG:boto:Token: None 2022-09-13T07:35:11.275 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/?versions 2022-09-13T07:35:11.276 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Zl2FzyDWvCkew0yGFb/RQXP9IGQ= 2022-09-13T07:35:11.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Zl2FzyDWvCkew0yGFb/RQXP9IGQ=', 'Content-Length': '0'} 2022-09-13T07:35:11.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c09db4abe10d5222-00632032af-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.279 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:11.280 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.281 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.282 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:11.282 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:11.283 DEBUG:boto:Method: GET 2022-09-13T07:35:11.283 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:11.284 DEBUG:boto:Data: 2022-09-13T07:35:11.284 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.285 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.285 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.286 DEBUG:boto:Params: {} 2022-09-13T07:35:11.286 DEBUG:boto:Token: None 2022-09-13T07:35:11.287 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/?versions 2022-09-13T07:35:11.288 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Zl2FzyDWvCkew0yGFb/RQXP9IGQ= 2022-09-13T07:35:11.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Zl2FzyDWvCkew0yGFb/RQXP9IGQ=', 'Content-Length': '0'} 2022-09-13T07:35:11.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab7b55612e5a0e1f-00632032af-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.292 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:11.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:11.293 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:11.294 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:11.295 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.295 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.296 DEBUG:boto:Method: GET 2022-09-13T07:35:11.296 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.297 DEBUG:boto:Data: 2022-09-13T07:35:11.297 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.298 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.298 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.299 DEBUG:boto:Params: {} 2022-09-13T07:35:11.300 DEBUG:boto:Token: None 2022-09-13T07:35:11.301 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.301 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:X/Wym3jX+ArmtWlUuWLWLV2DTS0= 2022-09-13T07:35:11.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:X/Wym3jX+ArmtWlUuWLWLV2DTS0=', 'Content-Length': '0'} 2022-09-13T07:35:11.304 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006421dc1bc56d6166-00632032af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.305 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:11.305 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:11.306 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.306 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.307 DEBUG:boto:Method: GET 2022-09-13T07:35:11.308 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.308 DEBUG:boto:Data: 2022-09-13T07:35:11.309 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.310 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.310 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.311 DEBUG:boto:Params: {} 2022-09-13T07:35:11.311 DEBUG:boto:Token: None 2022-09-13T07:35:11.312 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.313 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:X/Wym3jX+ArmtWlUuWLWLV2DTS0= 2022-09-13T07:35:11.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:X/Wym3jX+ArmtWlUuWLWLV2DTS0=', 'Content-Length': '0'} 2022-09-13T07:35:11.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003cdd2759c3112fbe-00632032af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.316 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:11.317 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:11.317 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.318 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.318 DEBUG:boto:Method: HEAD 2022-09-13T07:35:11.319 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.319 DEBUG:boto:Data: 2022-09-13T07:35:11.320 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.320 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.321 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.321 DEBUG:boto:Params: {} 2022-09-13T07:35:11.322 DEBUG:boto:Token: None 2022-09-13T07:35:11.322 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.323 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:sXetnhEbftNbwVAA2BKyo7/I5Rw= 2022-09-13T07:35:11.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:sXetnhEbftNbwVAA2BKyo7/I5Rw=', 'Content-Length': '0'} 2022-09-13T07:35:11.325 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000d094bb3866d0b68-00632032af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.326 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:11.327 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:11.327 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.328 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.328 DEBUG:boto:Method: HEAD 2022-09-13T07:35:11.329 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.329 DEBUG:boto:Data: 2022-09-13T07:35:11.330 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.330 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.331 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.331 DEBUG:boto:Params: {} 2022-09-13T07:35:11.332 DEBUG:boto:Token: None 2022-09-13T07:35:11.332 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.333 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:sXetnhEbftNbwVAA2BKyo7/I5Rw= 2022-09-13T07:35:11.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:sXetnhEbftNbwVAA2BKyo7/I5Rw=', 'Content-Length': '0'} 2022-09-13T07:35:11.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cdb2e46400989463-00632032af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:11.337 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:11.337 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:11.338 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.338 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.339 DEBUG:boto:Method: GET 2022-09-13T07:35:11.339 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.340 DEBUG:boto:Data: 2022-09-13T07:35:11.340 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.341 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.341 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.342 DEBUG:boto:Params: {} 2022-09-13T07:35:11.342 DEBUG:boto:Token: None 2022-09-13T07:35:11.343 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.344 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:X/Wym3jX+ArmtWlUuWLWLV2DTS0= 2022-09-13T07:35:11.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:X/Wym3jX+ArmtWlUuWLWLV2DTS0=', 'Content-Length': '0'} 2022-09-13T07:35:11.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001dbc8c81946783bb-00632032af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.347 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:11.348 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:11.348 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.349 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.349 DEBUG:boto:Method: GET 2022-09-13T07:35:11.350 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.350 DEBUG:boto:Data: 2022-09-13T07:35:11.351 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.351 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.352 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.353 DEBUG:boto:Params: {} 2022-09-13T07:35:11.353 DEBUG:boto:Token: None 2022-09-13T07:35:11.354 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.354 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:X/Wym3jX+ArmtWlUuWLWLV2DTS0= 2022-09-13T07:35:11.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:X/Wym3jX+ArmtWlUuWLWLV2DTS0=', 'Content-Length': '0'} 2022-09-13T07:35:11.357 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec72e166a2905fcc-00632032af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.358 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.358 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.359 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:11.359 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:11.360 DEBUG:boto:Method: GET 2022-09-13T07:35:11.361 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:35:11.361 DEBUG:boto:Data: 2022-09-13T07:35:11.362 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.362 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.363 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.363 DEBUG:boto:Params: {} 2022-09-13T07:35:11.364 DEBUG:boto:Token: None 2022-09-13T07:35:11.365 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/ 2022-09-13T07:35:11.365 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs= 2022-09-13T07:35:11.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs=', 'Content-Length': '0'} 2022-09-13T07:35:11.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000127c4bdef2c16012-00632032af-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.410 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:11.411 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.412 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.412 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:11.413 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:11.413 DEBUG:boto:Method: GET 2022-09-13T07:35:11.414 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:35:11.414 DEBUG:boto:Data: 2022-09-13T07:35:11.415 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.416 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.416 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.417 DEBUG:boto:Params: {} 2022-09-13T07:35:11.418 DEBUG:boto:Token: None 2022-09-13T07:35:11.418 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/ 2022-09-13T07:35:11.419 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs= 2022-09-13T07:35:11.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs=', 'Content-Length': '0'} 2022-09-13T07:35:11.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000760120f13030db98-00632032af-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.424 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:11.425 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:11.426 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:11.427 DEBUG:boto:Method: HEAD 2022-09-13T07:35:11.428 DEBUG:boto:Path: /yxvvuh-18/dummy 2022-09-13T07:35:11.428 DEBUG:boto:Data: 2022-09-13T07:35:11.429 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.430 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.431 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.431 DEBUG:boto:Params: {} 2022-09-13T07:35:11.432 DEBUG:boto:Token: None 2022-09-13T07:35:11.433 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/dummy 2022-09-13T07:35:11.433 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:e+v3JGELqZkBmlBOMxgt/uztpTE= 2022-09-13T07:35:11.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:e+v3JGELqZkBmlBOMxgt/uztpTE=', 'Content-Length': '0'} 2022-09-13T07:35:11.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fc3714b8178a3bad-00632032af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.437 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:11.437 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:11.438 DEBUG:boto:Method: HEAD 2022-09-13T07:35:11.439 DEBUG:boto:Path: /yxvvuh-18/dummy 2022-09-13T07:35:11.439 DEBUG:boto:Data: 2022-09-13T07:35:11.439 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.440 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.440 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.441 DEBUG:boto:Params: {} 2022-09-13T07:35:11.442 DEBUG:boto:Token: None 2022-09-13T07:35:11.442 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/dummy 2022-09-13T07:35:11.443 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:e+v3JGELqZkBmlBOMxgt/uztpTE= 2022-09-13T07:35:11.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:e+v3JGELqZkBmlBOMxgt/uztpTE=', 'Content-Length': '0'} 2022-09-13T07:35:11.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa76011c48b1bac8-00632032af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:11.447 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:11.448 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:11.448 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.449 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.450 DEBUG:boto:Method: GET 2022-09-13T07:35:11.450 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.451 DEBUG:boto:Data: 2022-09-13T07:35:11.451 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.452 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.452 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.453 DEBUG:boto:Params: {} 2022-09-13T07:35:11.453 DEBUG:boto:Token: None 2022-09-13T07:35:11.454 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.455 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:X/Wym3jX+ArmtWlUuWLWLV2DTS0= 2022-09-13T07:35:11.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:X/Wym3jX+ArmtWlUuWLWLV2DTS0=', 'Content-Length': '0'} 2022-09-13T07:35:11.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000924c1ea3bdb12903-00632032af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.459 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:11.459 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:11.460 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.460 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.461 DEBUG:boto:Method: GET 2022-09-13T07:35:11.461 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.462 DEBUG:boto:Data: 2022-09-13T07:35:11.462 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.463 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.463 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.464 DEBUG:boto:Params: {} 2022-09-13T07:35:11.465 DEBUG:boto:Token: None 2022-09-13T07:35:11.465 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:11.466 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:X/Wym3jX+ArmtWlUuWLWLV2DTS0= 2022-09-13T07:35:11.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:X/Wym3jX+ArmtWlUuWLWLV2DTS0=', 'Content-Length': '0'} 2022-09-13T07:35:11.469 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b7a304c400927635-00632032af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.470 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.470 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.471 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:11.471 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:11.472 DEBUG:boto:Method: GET 2022-09-13T07:35:11.472 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:35:11.473 DEBUG:boto:Data: 2022-09-13T07:35:11.473 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.474 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.475 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.475 DEBUG:boto:Params: {} 2022-09-13T07:35:11.476 DEBUG:boto:Token: None 2022-09-13T07:35:11.476 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/ 2022-09-13T07:35:11.477 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs= 2022-09-13T07:35:11.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs=', 'Content-Length': '0'} 2022-09-13T07:35:11.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd74fc7c21078103-00632032af-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.480 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:11.481 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.482 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.482 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:11.483 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:11.484 DEBUG:boto:Method: GET 2022-09-13T07:35:11.485 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:35:11.486 DEBUG:boto:Data: 2022-09-13T07:35:11.487 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.488 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.489 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.490 DEBUG:boto:Params: {} 2022-09-13T07:35:11.491 DEBUG:boto:Token: None 2022-09-13T07:35:11.492 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/ 2022-09-13T07:35:11.493 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs= 2022-09-13T07:35:11.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs=', 'Content-Length': '0'} 2022-09-13T07:35:11.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ca1cf3ff3d6044b-00632032af-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.539 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:11.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:11.540 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.541 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.542 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.542 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.543 DEBUG:boto:Method: GET 2022-09-13T07:35:11.543 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.544 DEBUG:boto:Data: 2022-09-13T07:35:11.545 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.545 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.546 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.546 DEBUG:boto:Params: {} 2022-09-13T07:35:11.547 DEBUG:boto:Token: None 2022-09-13T07:35:11.548 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.549 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:L5ybcHmXojzwKKldQrp9ylvUfH4= 2022-09-13T07:35:11.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:L5ybcHmXojzwKKldQrp9ylvUfH4=', 'Content-Length': '0'} 2022-09-13T07:35:11.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e91b93087e399b33-00632032af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.552 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.553 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.554 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.554 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.555 DEBUG:boto:Method: GET 2022-09-13T07:35:11.555 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.556 DEBUG:boto:Data: 2022-09-13T07:35:11.556 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.557 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.557 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.558 DEBUG:boto:Params: {} 2022-09-13T07:35:11.559 DEBUG:boto:Token: None 2022-09-13T07:35:11.559 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.560 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:L5ybcHmXojzwKKldQrp9ylvUfH4= 2022-09-13T07:35:11.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:L5ybcHmXojzwKKldQrp9ylvUfH4=', 'Content-Length': '0'} 2022-09-13T07:35:11.563 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088f0855d29c77be3-00632032af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.564 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.564 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.565 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.565 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.566 DEBUG:boto:Method: HEAD 2022-09-13T07:35:11.566 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.567 DEBUG:boto:Data: 2022-09-13T07:35:11.567 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.568 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.568 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.569 DEBUG:boto:Params: {} 2022-09-13T07:35:11.569 DEBUG:boto:Token: None 2022-09-13T07:35:11.570 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.570 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4+RIPxZEwseJotC7UzAIhlfndk8= 2022-09-13T07:35:11.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4+RIPxZEwseJotC7UzAIhlfndk8=', 'Content-Length': '0'} 2022-09-13T07:35:11.575 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c869f97829703fa-00632032af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.575 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.576 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.576 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.577 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.578 DEBUG:boto:Method: HEAD 2022-09-13T07:35:11.579 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.580 DEBUG:boto:Data: 2022-09-13T07:35:11.581 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.582 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.583 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.583 DEBUG:boto:Params: {} 2022-09-13T07:35:11.584 DEBUG:boto:Token: None 2022-09-13T07:35:11.584 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.585 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4+RIPxZEwseJotC7UzAIhlfndk8= 2022-09-13T07:35:11.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4+RIPxZEwseJotC7UzAIhlfndk8=', 'Content-Length': '0'} 2022-09-13T07:35:11.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d40c98693da574fe-00632032af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:11.589 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.590 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.591 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.591 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.592 DEBUG:boto:Method: GET 2022-09-13T07:35:11.592 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.593 DEBUG:boto:Data: 2022-09-13T07:35:11.593 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.594 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.594 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.595 DEBUG:boto:Params: {} 2022-09-13T07:35:11.595 DEBUG:boto:Token: None 2022-09-13T07:35:11.596 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.596 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:L5ybcHmXojzwKKldQrp9ylvUfH4= 2022-09-13T07:35:11.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:L5ybcHmXojzwKKldQrp9ylvUfH4=', 'Content-Length': '0'} 2022-09-13T07:35:11.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d1a8a95fe983774-00632032af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.600 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.600 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.601 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.601 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.602 DEBUG:boto:Method: GET 2022-09-13T07:35:11.602 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.603 DEBUG:boto:Data: 2022-09-13T07:35:11.603 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.604 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.605 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.605 DEBUG:boto:Params: {} 2022-09-13T07:35:11.606 DEBUG:boto:Token: None 2022-09-13T07:35:11.606 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.607 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:L5ybcHmXojzwKKldQrp9ylvUfH4= 2022-09-13T07:35:11.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:L5ybcHmXojzwKKldQrp9ylvUfH4=', 'Content-Length': '0'} 2022-09-13T07:35:11.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081d51f7716b74e1d-00632032af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.610 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.611 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.611 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:11.612 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:11.612 DEBUG:boto:Method: GET 2022-09-13T07:35:11.613 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:11.613 DEBUG:boto:Data: 2022-09-13T07:35:11.614 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.614 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.615 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.615 DEBUG:boto:Params: {} 2022-09-13T07:35:11.616 DEBUG:boto:Token: None 2022-09-13T07:35:11.617 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/ 2022-09-13T07:35:11.617 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs= 2022-09-13T07:35:11.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs=', 'Content-Length': '0'} 2022-09-13T07:35:11.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b07945719199e983-00632032af-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.621 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:11.622 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.622 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.623 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:11.623 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:11.624 DEBUG:boto:Method: GET 2022-09-13T07:35:11.625 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:11.625 DEBUG:boto:Data: 2022-09-13T07:35:11.626 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.626 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.627 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.627 DEBUG:boto:Params: {} 2022-09-13T07:35:11.628 DEBUG:boto:Token: None 2022-09-13T07:35:11.629 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/ 2022-09-13T07:35:11.630 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs= 2022-09-13T07:35:11.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs=', 'Content-Length': '0'} 2022-09-13T07:35:11.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068b38343a0112fa6-00632032af-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.634 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:11.635 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.635 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.636 DEBUG:boto:Method: HEAD 2022-09-13T07:35:11.637 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.637 DEBUG:boto:Data: 2022-09-13T07:35:11.638 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.639 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.639 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.640 DEBUG:boto:Params: {} 2022-09-13T07:35:11.640 DEBUG:boto:Token: None 2022-09-13T07:35:11.641 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.642 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1lmJmNcXvp4MECVTH8Fi9XFprKU= 2022-09-13T07:35:11.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1lmJmNcXvp4MECVTH8Fi9XFprKU=', 'Content-Length': '0'} 2022-09-13T07:35:11.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036e69c9760cb85e7-00632032af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.646 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.646 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.647 DEBUG:boto:Method: HEAD 2022-09-13T07:35:11.647 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.648 DEBUG:boto:Data: 2022-09-13T07:35:11.648 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.649 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.649 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.650 DEBUG:boto:Params: {} 2022-09-13T07:35:11.651 DEBUG:boto:Token: None 2022-09-13T07:35:11.651 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.652 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1lmJmNcXvp4MECVTH8Fi9XFprKU= 2022-09-13T07:35:11.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1lmJmNcXvp4MECVTH8Fi9XFprKU=', 'Content-Length': '0'} 2022-09-13T07:35:11.656 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002071b74191a62202-00632032af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:11.657 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.658 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.659 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.660 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.661 DEBUG:boto:Method: GET 2022-09-13T07:35:11.661 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.662 DEBUG:boto:Data: 2022-09-13T07:35:11.663 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.664 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.665 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.666 DEBUG:boto:Params: {} 2022-09-13T07:35:11.667 DEBUG:boto:Token: None 2022-09-13T07:35:11.668 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.669 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:L5ybcHmXojzwKKldQrp9ylvUfH4= 2022-09-13T07:35:11.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:L5ybcHmXojzwKKldQrp9ylvUfH4=', 'Content-Length': '0'} 2022-09-13T07:35:11.673 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba486ccf68a06ebf-00632032af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.674 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.675 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:11.676 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.677 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.678 DEBUG:boto:Method: GET 2022-09-13T07:35:11.679 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.680 DEBUG:boto:Data: 2022-09-13T07:35:11.681 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.682 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.683 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.684 DEBUG:boto:Params: {} 2022-09-13T07:35:11.685 DEBUG:boto:Token: None 2022-09-13T07:35:11.687 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:11.688 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:L5ybcHmXojzwKKldQrp9ylvUfH4= 2022-09-13T07:35:11.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:L5ybcHmXojzwKKldQrp9ylvUfH4=', 'Content-Length': '0'} 2022-09-13T07:35:11.692 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003005d9f46039d2bb-00632032af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.734 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.735 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.736 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:11.737 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:11.738 DEBUG:boto:Method: GET 2022-09-13T07:35:11.739 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:11.740 DEBUG:boto:Data: 2022-09-13T07:35:11.741 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.743 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.744 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.745 DEBUG:boto:Params: {} 2022-09-13T07:35:11.745 DEBUG:boto:Token: None 2022-09-13T07:35:11.746 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/ 2022-09-13T07:35:11.747 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs= 2022-09-13T07:35:11.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs=', 'Content-Length': '0'} 2022-09-13T07:35:11.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf176662562b64e2-00632032af-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.751 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:11.752 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.753 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.753 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:11.754 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:11.754 DEBUG:boto:Method: GET 2022-09-13T07:35:11.755 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:11.755 DEBUG:boto:Data: 2022-09-13T07:35:11.756 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.756 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.757 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.758 DEBUG:boto:Params: {} 2022-09-13T07:35:11.758 DEBUG:boto:Token: None 2022-09-13T07:35:11.759 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/ 2022-09-13T07:35:11.760 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs= 2022-09-13T07:35:11.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs=', 'Content-Length': '0'} 2022-09-13T07:35:11.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008077867f4c622d06-00632032af-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.765 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:11.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:11.766 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.767 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.767 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.768 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.769 DEBUG:boto:Method: GET 2022-09-13T07:35:11.769 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.770 DEBUG:boto:Data: 2022-09-13T07:35:11.770 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.771 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.771 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.772 DEBUG:boto:Params: {} 2022-09-13T07:35:11.773 DEBUG:boto:Token: None 2022-09-13T07:35:11.773 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.774 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BqN/Gk48RTjiEl8folg5pN7uDjE= 2022-09-13T07:35:11.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BqN/Gk48RTjiEl8folg5pN7uDjE=', 'Content-Length': '0'} 2022-09-13T07:35:11.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1a338495effabbd-00632032af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.818 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.819 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.819 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.820 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.820 DEBUG:boto:Method: GET 2022-09-13T07:35:11.821 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.821 DEBUG:boto:Data: 2022-09-13T07:35:11.822 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.822 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.823 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.823 DEBUG:boto:Params: {} 2022-09-13T07:35:11.824 DEBUG:boto:Token: None 2022-09-13T07:35:11.825 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.825 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BqN/Gk48RTjiEl8folg5pN7uDjE= 2022-09-13T07:35:11.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BqN/Gk48RTjiEl8folg5pN7uDjE=', 'Content-Length': '0'} 2022-09-13T07:35:11.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000319209030ed3da02-00632032af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.829 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.830 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.830 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.831 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.832 DEBUG:boto:Method: HEAD 2022-09-13T07:35:11.832 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.833 DEBUG:boto:Data: 2022-09-13T07:35:11.833 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.833 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.834 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.835 DEBUG:boto:Params: {} 2022-09-13T07:35:11.835 DEBUG:boto:Token: None 2022-09-13T07:35:11.836 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.836 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DFIianUS10pnOnXKNyVPNvN+TP0= 2022-09-13T07:35:11.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DFIianUS10pnOnXKNyVPNvN+TP0=', 'Content-Length': '0'} 2022-09-13T07:35:11.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000320edffe94ecb1b5-00632032af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.840 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.841 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.841 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.841 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.842 DEBUG:boto:Method: HEAD 2022-09-13T07:35:11.842 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.843 DEBUG:boto:Data: 2022-09-13T07:35:11.843 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.844 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.844 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.845 DEBUG:boto:Params: {} 2022-09-13T07:35:11.845 DEBUG:boto:Token: None 2022-09-13T07:35:11.846 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.847 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DFIianUS10pnOnXKNyVPNvN+TP0= 2022-09-13T07:35:11.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DFIianUS10pnOnXKNyVPNvN+TP0=', 'Content-Length': '0'} 2022-09-13T07:35:11.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000927cb0e52692b2fe-00632032af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:11.851 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.851 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.852 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.852 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.853 DEBUG:boto:Method: GET 2022-09-13T07:35:11.854 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.854 DEBUG:boto:Data: 2022-09-13T07:35:11.855 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.855 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.856 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.856 DEBUG:boto:Params: {} 2022-09-13T07:35:11.857 DEBUG:boto:Token: None 2022-09-13T07:35:11.858 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.858 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BqN/Gk48RTjiEl8folg5pN7uDjE= 2022-09-13T07:35:11.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BqN/Gk48RTjiEl8folg5pN7uDjE=', 'Content-Length': '0'} 2022-09-13T07:35:11.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3c1dfde8b9d4d92-00632032af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.863 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.863 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.864 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.865 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.865 DEBUG:boto:Method: GET 2022-09-13T07:35:11.866 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.866 DEBUG:boto:Data: 2022-09-13T07:35:11.867 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.868 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.868 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.869 DEBUG:boto:Params: {} 2022-09-13T07:35:11.869 DEBUG:boto:Token: None 2022-09-13T07:35:11.870 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.871 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BqN/Gk48RTjiEl8folg5pN7uDjE= 2022-09-13T07:35:11.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BqN/Gk48RTjiEl8folg5pN7uDjE=', 'Content-Length': '0'} 2022-09-13T07:35:11.873 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075b9b6fb90535832-00632032af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.874 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.875 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.875 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:11.876 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:11.876 DEBUG:boto:Method: GET 2022-09-13T07:35:11.877 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:11.878 DEBUG:boto:Data: 2022-09-13T07:35:11.878 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.879 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.880 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.880 DEBUG:boto:Params: {} 2022-09-13T07:35:11.881 DEBUG:boto:Token: None 2022-09-13T07:35:11.882 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/ 2022-09-13T07:35:11.883 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs= 2022-09-13T07:35:11.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs=', 'Content-Length': '0'} 2022-09-13T07:35:11.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f3bc64c3dc1f04e-00632032af-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.888 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:11.889 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:11.891 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:11.892 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:11.892 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:11.893 DEBUG:boto:Method: GET 2022-09-13T07:35:11.894 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:11.895 DEBUG:boto:Data: 2022-09-13T07:35:11.896 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.897 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.898 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.899 DEBUG:boto:Params: {} 2022-09-13T07:35:11.900 DEBUG:boto:Token: None 2022-09-13T07:35:11.901 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/ 2022-09-13T07:35:11.902 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs= 2022-09-13T07:35:11.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MfSeWmNEZA3fLN3YL4SMTLtweGs=', 'Content-Length': '0'} 2022-09-13T07:35:11.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd0bf2f162c9311f-00632032af-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.906 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:11.907 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.908 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.908 DEBUG:boto:Method: HEAD 2022-09-13T07:35:11.909 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.909 DEBUG:boto:Data: 2022-09-13T07:35:11.910 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.910 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.911 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.911 DEBUG:boto:Params: {} 2022-09-13T07:35:11.912 DEBUG:boto:Token: None 2022-09-13T07:35:11.912 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.913 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xq0y9B93qrPcjOitMy/9HWmCxtw= 2022-09-13T07:35:11.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xq0y9B93qrPcjOitMy/9HWmCxtw=', 'Content-Length': '0'} 2022-09-13T07:35:11.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d304a3fadf44407e-00632032af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.916 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.917 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.917 DEBUG:boto:Method: HEAD 2022-09-13T07:35:11.918 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.919 DEBUG:boto:Data: 2022-09-13T07:35:11.920 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.921 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.922 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.923 DEBUG:boto:Params: {} 2022-09-13T07:35:11.924 DEBUG:boto:Token: None 2022-09-13T07:35:11.925 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.926 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xq0y9B93qrPcjOitMy/9HWmCxtw= 2022-09-13T07:35:11.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xq0y9B93qrPcjOitMy/9HWmCxtw=', 'Content-Length': '0'} 2022-09-13T07:35:11.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8b5a8271991ff0c-00632032af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:11.932 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.933 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.934 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.935 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.936 DEBUG:boto:Method: GET 2022-09-13T07:35:11.937 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.938 DEBUG:boto:Data: 2022-09-13T07:35:11.939 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.940 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:11.941 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.941 DEBUG:boto:Params: {} 2022-09-13T07:35:11.943 DEBUG:boto:Token: None 2022-09-13T07:35:11.944 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.945 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BqN/Gk48RTjiEl8folg5pN7uDjE= 2022-09-13T07:35:11.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BqN/Gk48RTjiEl8folg5pN7uDjE=', 'Content-Length': '0'} 2022-09-13T07:35:11.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032d4b7678b57b1da-00632032af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:11.990 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.991 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:11.992 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.992 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.993 DEBUG:boto:Method: GET 2022-09-13T07:35:11.993 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.994 DEBUG:boto:Data: 2022-09-13T07:35:11.995 DEBUG:boto:Headers: {} 2022-09-13T07:35:11.995 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:11.995 DEBUG:boto:Port: 8000 2022-09-13T07:35:11.996 DEBUG:boto:Params: {} 2022-09-13T07:35:11.997 DEBUG:boto:Token: None 2022-09-13T07:35:11.997 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:11 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:11.998 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BqN/Gk48RTjiEl8folg5pN7uDjE= 2022-09-13T07:35:11.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BqN/Gk48RTjiEl8folg5pN7uDjE=', 'Content-Length': '0'} 2022-09-13T07:35:12.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fefe9fc80184815-00632032af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.002 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.003 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.004 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:12.005 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:12.006 DEBUG:boto:Method: GET 2022-09-13T07:35:12.007 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:12.008 DEBUG:boto:Data: 2022-09-13T07:35:12.009 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.010 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:12.011 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.011 DEBUG:boto:Params: {} 2022-09-13T07:35:12.013 DEBUG:boto:Token: None 2022-09-13T07:35:12.014 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.015 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e713343c9f4bce25-00632032b0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.020 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.021 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.022 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.023 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:12.024 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:12.025 DEBUG:boto:Method: GET 2022-09-13T07:35:12.026 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:12.027 DEBUG:boto:Data: 2022-09-13T07:35:12.028 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.028 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.029 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.030 DEBUG:boto:Params: {} 2022-09-13T07:35:12.031 DEBUG:boto:Token: None 2022-09-13T07:35:12.032 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.033 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a7b73fbd409d7a8-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.039 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:12.041 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.042 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.043 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.044 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.045 DEBUG:boto:Method: GET 2022-09-13T07:35:12.045 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.046 DEBUG:boto:Data: 2022-09-13T07:35:12.047 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.048 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:12.049 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.050 DEBUG:boto:Params: {} 2022-09-13T07:35:12.051 DEBUG:boto:Token: None 2022-09-13T07:35:12.052 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.053 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:B8oBIiTtnf5J7MKu7ginEtyzkks= 2022-09-13T07:35:12.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:B8oBIiTtnf5J7MKu7ginEtyzkks=', 'Content-Length': '0'} 2022-09-13T07:35:12.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002205b32d40567833-00632032b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.058 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.059 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.060 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.060 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.061 DEBUG:boto:Method: GET 2022-09-13T07:35:12.062 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.063 DEBUG:boto:Data: 2022-09-13T07:35:12.064 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.065 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.065 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.066 DEBUG:boto:Params: {} 2022-09-13T07:35:12.067 DEBUG:boto:Token: None 2022-09-13T07:35:12.068 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.069 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:B8oBIiTtnf5J7MKu7ginEtyzkks= 2022-09-13T07:35:12.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:B8oBIiTtnf5J7MKu7ginEtyzkks=', 'Content-Length': '0'} 2022-09-13T07:35:12.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f3724b81dbd23a1-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.073 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.074 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.074 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.075 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.075 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.076 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.076 DEBUG:boto:Data: 2022-09-13T07:35:12.077 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.077 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:12.078 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.078 DEBUG:boto:Params: {} 2022-09-13T07:35:12.079 DEBUG:boto:Token: None 2022-09-13T07:35:12.079 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.080 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PEGRN1+N89XPTislz5ou9TWeWUM= 2022-09-13T07:35:12.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PEGRN1+N89XPTislz5ou9TWeWUM=', 'Content-Length': '0'} 2022-09-13T07:35:12.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c581810c5c90d3b-00632032b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.083 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.083 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.083 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.084 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.084 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.085 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.085 DEBUG:boto:Data: 2022-09-13T07:35:12.086 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.086 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.087 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.087 DEBUG:boto:Params: {} 2022-09-13T07:35:12.088 DEBUG:boto:Token: None 2022-09-13T07:35:12.088 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.089 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PEGRN1+N89XPTislz5ou9TWeWUM= 2022-09-13T07:35:12.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PEGRN1+N89XPTislz5ou9TWeWUM=', 'Content-Length': '0'} 2022-09-13T07:35:12.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b84c6debe25a5473-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:12.093 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.093 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.094 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.094 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.095 DEBUG:boto:Method: GET 2022-09-13T07:35:12.096 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.096 DEBUG:boto:Data: 2022-09-13T07:35:12.097 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.097 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:12.097 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.098 DEBUG:boto:Params: {} 2022-09-13T07:35:12.099 DEBUG:boto:Token: None 2022-09-13T07:35:12.099 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.100 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:B8oBIiTtnf5J7MKu7ginEtyzkks= 2022-09-13T07:35:12.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:B8oBIiTtnf5J7MKu7ginEtyzkks=', 'Content-Length': '0'} 2022-09-13T07:35:12.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094bc170b3e2befee-00632032b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.146 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.147 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.148 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.148 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.149 DEBUG:boto:Method: GET 2022-09-13T07:35:12.149 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.150 DEBUG:boto:Data: 2022-09-13T07:35:12.151 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.151 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.152 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.152 DEBUG:boto:Params: {} 2022-09-13T07:35:12.153 DEBUG:boto:Token: None 2022-09-13T07:35:12.153 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.154 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:B8oBIiTtnf5J7MKu7ginEtyzkks= 2022-09-13T07:35:12.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:B8oBIiTtnf5J7MKu7ginEtyzkks=', 'Content-Length': '0'} 2022-09-13T07:35:12.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fd1b7c5a3aaea87-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.158 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.159 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.160 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:12.161 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:12.162 DEBUG:boto:Method: GET 2022-09-13T07:35:12.163 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:12.164 DEBUG:boto:Data: 2022-09-13T07:35:12.165 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.166 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:12.167 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.167 DEBUG:boto:Params: {} 2022-09-13T07:35:12.169 DEBUG:boto:Token: None 2022-09-13T07:35:12.170 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.170 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008646077c2614f31d-00632032b0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.175 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.175 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.176 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.176 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:12.177 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:12.177 DEBUG:boto:Method: GET 2022-09-13T07:35:12.178 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:12.179 DEBUG:boto:Data: 2022-09-13T07:35:12.179 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.180 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.180 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.181 DEBUG:boto:Params: {} 2022-09-13T07:35:12.181 DEBUG:boto:Token: None 2022-09-13T07:35:12.182 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.183 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b48f154d695f7dc-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.186 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.188 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.188 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.189 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.189 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.190 DEBUG:boto:Data: 2022-09-13T07:35:12.190 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.191 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:12.191 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.192 DEBUG:boto:Params: {} 2022-09-13T07:35:12.192 DEBUG:boto:Token: None 2022-09-13T07:35:12.193 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.193 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eFYghxmnjqL1ho3wv5VoNPrn3Lc= 2022-09-13T07:35:12.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFYghxmnjqL1ho3wv5VoNPrn3Lc=', 'Content-Length': '0'} 2022-09-13T07:35:12.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000519926fc7539d4c8-00632032b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.197 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.198 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.198 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.199 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.199 DEBUG:boto:Data: 2022-09-13T07:35:12.200 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.201 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.202 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.203 DEBUG:boto:Params: {} 2022-09-13T07:35:12.204 DEBUG:boto:Token: None 2022-09-13T07:35:12.205 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.206 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eFYghxmnjqL1ho3wv5VoNPrn3Lc= 2022-09-13T07:35:12.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFYghxmnjqL1ho3wv5VoNPrn3Lc=', 'Content-Length': '0'} 2022-09-13T07:35:12.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fd781521b7a12aa-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:12.211 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.212 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.213 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.213 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.214 DEBUG:boto:Method: GET 2022-09-13T07:35:12.214 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.215 DEBUG:boto:Data: 2022-09-13T07:35:12.215 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.216 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:12.216 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.217 DEBUG:boto:Params: {} 2022-09-13T07:35:12.218 DEBUG:boto:Token: None 2022-09-13T07:35:12.218 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.219 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:B8oBIiTtnf5J7MKu7ginEtyzkks= 2022-09-13T07:35:12.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:B8oBIiTtnf5J7MKu7ginEtyzkks=', 'Content-Length': '0'} 2022-09-13T07:35:12.222 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b83d64172bf9bf77-00632032b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.222 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.223 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:12.223 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.224 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.225 DEBUG:boto:Method: GET 2022-09-13T07:35:12.225 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.226 DEBUG:boto:Data: 2022-09-13T07:35:12.226 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.227 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.227 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.228 DEBUG:boto:Params: {} 2022-09-13T07:35:12.229 DEBUG:boto:Token: None 2022-09-13T07:35:12.229 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:12.230 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:B8oBIiTtnf5J7MKu7ginEtyzkks= 2022-09-13T07:35:12.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:B8oBIiTtnf5J7MKu7ginEtyzkks=', 'Content-Length': '0'} 2022-09-13T07:35:12.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058791651096d47c2-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.274 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.275 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.276 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:12.277 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:12.278 DEBUG:boto:Method: GET 2022-09-13T07:35:12.279 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:12.280 DEBUG:boto:Data: 2022-09-13T07:35:12.281 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.282 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:12.285 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.285 DEBUG:boto:Params: {} 2022-09-13T07:35:12.286 DEBUG:boto:Token: None 2022-09-13T07:35:12.287 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.287 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db4493a079613f4b-00632032b0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.292 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.293 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.293 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.294 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:12.294 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:12.296 DEBUG:boto:Method: GET 2022-09-13T07:35:12.297 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:12.297 DEBUG:boto:Data: 2022-09-13T07:35:12.298 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.298 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.299 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.300 DEBUG:boto:Params: {} 2022-09-13T07:35:12.300 DEBUG:boto:Token: None 2022-09-13T07:35:12.301 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.302 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f7807a8849e13c8-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.306 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.307 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-18 zones={test-zone2, test-zone1} 2022-09-13T07:35:12.308 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-18 zones={test-zone3, test-zone1} 2022-09-13T07:35:12.308 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.309 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.309 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.310 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:35:12.311 DEBUG:boto:Data: 2022-09-13T07:35:12.311 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.312 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.312 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.313 DEBUG:boto:Params: {} 2022-09-13T07:35:12.313 DEBUG:boto:Token: None 2022-09-13T07:35:12.314 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.315 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vTHPgwzC1P2FJPk1EJPDDVfaV0Q= 2022-09-13T07:35:12.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vTHPgwzC1P2FJPk1EJPDDVfaV0Q=', 'Content-Length': '0'} 2022-09-13T07:35:12.318 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e365db585ab32ad2-00632032b0-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.318 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.319 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.319 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.320 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:35:12.320 DEBUG:boto:Data: 2022-09-13T07:35:12.321 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.321 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.322 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.323 DEBUG:boto:Params: {} 2022-09-13T07:35:12.323 DEBUG:boto:Token: None 2022-09-13T07:35:12.324 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.324 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vTHPgwzC1P2FJPk1EJPDDVfaV0Q= 2022-09-13T07:35:12.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vTHPgwzC1P2FJPk1EJPDDVfaV0Q=', 'Content-Length': '0'} 2022-09-13T07:35:12.328 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b75d61a8c9fa7f20-00632032b0-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.329 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:35:12.330 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.330 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.331 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:12.331 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:12.332 DEBUG:boto:Method: GET 2022-09-13T07:35:12.332 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:12.333 DEBUG:boto:Data: 2022-09-13T07:35:12.333 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.334 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.334 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.335 DEBUG:boto:Params: {} 2022-09-13T07:35:12.335 DEBUG:boto:Token: None 2022-09-13T07:35:12.336 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/?versions 2022-09-13T07:35:12.337 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5JW7bQbmIi87ywX2u8W1EnZq6x8= 2022-09-13T07:35:12.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5JW7bQbmIi87ywX2u8W1EnZq6x8=', 'Content-Length': '0'} 2022-09-13T07:35:12.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a5d935481723736-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.341 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.342 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:35:12.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:35:12.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:35:12.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:35:12.344 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:35:12.344 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.345 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.345 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:12.346 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:12.346 DEBUG:boto:Method: GET 2022-09-13T07:35:12.347 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:12.347 DEBUG:boto:Data: 2022-09-13T07:35:12.348 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.348 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.349 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.349 DEBUG:boto:Params: {} 2022-09-13T07:35:12.350 DEBUG:boto:Token: None 2022-09-13T07:35:12.351 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/?versions 2022-09-13T07:35:12.351 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5JW7bQbmIi87ywX2u8W1EnZq6x8= 2022-09-13T07:35:12.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5JW7bQbmIi87ywX2u8W1EnZq6x8=', 'Content-Length': '0'} 2022-09-13T07:35:12.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e53714bf718dbb1b-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.355 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.356 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:35:12.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:35:12.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:35:12.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:35:12.358 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.359 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.359 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:12.360 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:12.361 DEBUG:boto:Method: GET 2022-09-13T07:35:12.361 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:12.362 DEBUG:boto:Data: 2022-09-13T07:35:12.362 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.363 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.363 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.364 DEBUG:boto:Params: {} 2022-09-13T07:35:12.364 DEBUG:boto:Token: None 2022-09-13T07:35:12.365 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/?versions 2022-09-13T07:35:12.365 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5JW7bQbmIi87ywX2u8W1EnZq6x8= 2022-09-13T07:35:12.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5JW7bQbmIi87ywX2u8W1EnZq6x8=', 'Content-Length': '0'} 2022-09-13T07:35:12.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c2e4c0f164fb29a-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.369 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.370 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.371 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.371 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:12.372 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:12.373 DEBUG:boto:Method: GET 2022-09-13T07:35:12.373 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:12.374 DEBUG:boto:Data: 2022-09-13T07:35:12.374 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.375 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.375 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.376 DEBUG:boto:Params: {} 2022-09-13T07:35:12.376 DEBUG:boto:Token: None 2022-09-13T07:35:12.377 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/?versions 2022-09-13T07:35:12.378 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5JW7bQbmIi87ywX2u8W1EnZq6x8= 2022-09-13T07:35:12.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5JW7bQbmIi87ywX2u8W1EnZq6x8=', 'Content-Length': '0'} 2022-09-13T07:35:12.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a831841bf4ed3dce-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.382 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:12.383 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:12.384 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:12.384 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.385 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.385 DEBUG:boto:Method: GET 2022-09-13T07:35:12.386 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.387 DEBUG:boto:Data: 2022-09-13T07:35:12.387 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.388 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.388 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.389 DEBUG:boto:Params: {} 2022-09-13T07:35:12.389 DEBUG:boto:Token: None 2022-09-13T07:35:12.390 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.390 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:57I66G21fnct3JBqWQuKr6sVOsk= 2022-09-13T07:35:12.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:57I66G21fnct3JBqWQuKr6sVOsk=', 'Content-Length': '0'} 2022-09-13T07:35:12.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ea6e6a28c4412bce-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.394 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:12.395 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:12.395 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.396 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.396 DEBUG:boto:Method: GET 2022-09-13T07:35:12.397 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.398 DEBUG:boto:Data: 2022-09-13T07:35:12.399 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.400 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.401 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.402 DEBUG:boto:Params: {} 2022-09-13T07:35:12.403 DEBUG:boto:Token: None 2022-09-13T07:35:12.404 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.405 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:57I66G21fnct3JBqWQuKr6sVOsk= 2022-09-13T07:35:12.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:57I66G21fnct3JBqWQuKr6sVOsk=', 'Content-Length': '0'} 2022-09-13T07:35:12.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a6b78a9a1cfd96b6-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.409 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:12.409 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:12.410 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.410 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.411 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.412 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.412 DEBUG:boto:Data: 2022-09-13T07:35:12.413 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.413 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.414 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.414 DEBUG:boto:Params: {} 2022-09-13T07:35:12.415 DEBUG:boto:Token: None 2022-09-13T07:35:12.415 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.416 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1EOUAE3jUMvzpZUAj7BHmQoZOv0= 2022-09-13T07:35:12.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1EOUAE3jUMvzpZUAj7BHmQoZOv0=', 'Content-Length': '0'} 2022-09-13T07:35:12.419 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007ea5a11b5d536e99-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.419 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:12.420 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:12.420 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.421 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.421 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.422 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.423 DEBUG:boto:Data: 2022-09-13T07:35:12.423 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.424 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.424 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.425 DEBUG:boto:Params: {} 2022-09-13T07:35:12.425 DEBUG:boto:Token: None 2022-09-13T07:35:12.426 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.426 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1EOUAE3jUMvzpZUAj7BHmQoZOv0= 2022-09-13T07:35:12.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1EOUAE3jUMvzpZUAj7BHmQoZOv0=', 'Content-Length': '0'} 2022-09-13T07:35:12.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006a234f6dbfdbb343-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:12.430 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:12.431 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:12.431 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.432 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.432 DEBUG:boto:Method: GET 2022-09-13T07:35:12.433 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.433 DEBUG:boto:Data: 2022-09-13T07:35:12.434 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.434 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.435 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.435 DEBUG:boto:Params: {} 2022-09-13T07:35:12.436 DEBUG:boto:Token: None 2022-09-13T07:35:12.437 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.437 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:57I66G21fnct3JBqWQuKr6sVOsk= 2022-09-13T07:35:12.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:57I66G21fnct3JBqWQuKr6sVOsk=', 'Content-Length': '0'} 2022-09-13T07:35:12.441 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000075d588bdd4b48208-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.442 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:12.443 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:12.443 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.444 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.444 DEBUG:boto:Method: GET 2022-09-13T07:35:12.445 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.445 DEBUG:boto:Data: 2022-09-13T07:35:12.446 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.447 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.447 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.448 DEBUG:boto:Params: {} 2022-09-13T07:35:12.448 DEBUG:boto:Token: None 2022-09-13T07:35:12.449 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.450 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:57I66G21fnct3JBqWQuKr6sVOsk= 2022-09-13T07:35:12.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:57I66G21fnct3JBqWQuKr6sVOsk=', 'Content-Length': '0'} 2022-09-13T07:35:12.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bfe2cf0507827a1b-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.454 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.455 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.456 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:12.456 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:12.457 DEBUG:boto:Method: GET 2022-09-13T07:35:12.457 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:35:12.458 DEBUG:boto:Data: 2022-09-13T07:35:12.459 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.460 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.460 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.461 DEBUG:boto:Params: {} 2022-09-13T07:35:12.461 DEBUG:boto:Token: None 2022-09-13T07:35:12.462 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.462 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a0c021b8c8e00ff-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.507 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.508 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.509 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.509 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:12.510 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:12.511 DEBUG:boto:Method: GET 2022-09-13T07:35:12.512 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:35:12.513 DEBUG:boto:Data: 2022-09-13T07:35:12.514 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.514 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.515 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.515 DEBUG:boto:Params: {} 2022-09-13T07:35:12.516 DEBUG:boto:Token: None 2022-09-13T07:35:12.517 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.517 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004debab011ea247cf-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.522 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.523 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:12.523 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:12.524 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.524 DEBUG:boto:Path: /yxvvuh-18/dummy 2022-09-13T07:35:12.525 DEBUG:boto:Data: 2022-09-13T07:35:12.525 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.526 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.526 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.527 DEBUG:boto:Params: {} 2022-09-13T07:35:12.528 DEBUG:boto:Token: None 2022-09-13T07:35:12.528 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/dummy 2022-09-13T07:35:12.529 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PCRsi0BJnBgGK5rc6G07J4gLeiM= 2022-09-13T07:35:12.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PCRsi0BJnBgGK5rc6G07J4gLeiM=', 'Content-Length': '0'} 2022-09-13T07:35:12.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002173d45cf1275dd8-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.533 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:12.533 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:12.534 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.535 DEBUG:boto:Path: /yxvvuh-18/dummy 2022-09-13T07:35:12.535 DEBUG:boto:Data: 2022-09-13T07:35:12.536 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.536 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.537 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.537 DEBUG:boto:Params: {} 2022-09-13T07:35:12.538 DEBUG:boto:Token: None 2022-09-13T07:35:12.538 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/dummy 2022-09-13T07:35:12.539 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PCRsi0BJnBgGK5rc6G07J4gLeiM= 2022-09-13T07:35:12.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PCRsi0BJnBgGK5rc6G07J4gLeiM=', 'Content-Length': '0'} 2022-09-13T07:35:12.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fcc89a2c38e52bca-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:12.544 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:12.545 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:12.545 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.546 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.546 DEBUG:boto:Method: GET 2022-09-13T07:35:12.547 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.547 DEBUG:boto:Data: 2022-09-13T07:35:12.548 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.549 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.549 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.550 DEBUG:boto:Params: {} 2022-09-13T07:35:12.551 DEBUG:boto:Token: None 2022-09-13T07:35:12.551 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.552 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:57I66G21fnct3JBqWQuKr6sVOsk= 2022-09-13T07:35:12.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:57I66G21fnct3JBqWQuKr6sVOsk=', 'Content-Length': '0'} 2022-09-13T07:35:12.555 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d4223931d52c8889-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.556 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:12.556 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:12.557 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.558 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.558 DEBUG:boto:Method: GET 2022-09-13T07:35:12.559 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.559 DEBUG:boto:Data: 2022-09-13T07:35:12.560 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.561 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.561 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.562 DEBUG:boto:Params: {} 2022-09-13T07:35:12.562 DEBUG:boto:Token: None 2022-09-13T07:35:12.563 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:12.564 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:57I66G21fnct3JBqWQuKr6sVOsk= 2022-09-13T07:35:12.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:57I66G21fnct3JBqWQuKr6sVOsk=', 'Content-Length': '0'} 2022-09-13T07:35:12.566 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bdb338e88bf07e20-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.567 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.568 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.568 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:12.569 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:12.569 DEBUG:boto:Method: GET 2022-09-13T07:35:12.570 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:35:12.571 DEBUG:boto:Data: 2022-09-13T07:35:12.571 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.572 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.572 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.573 DEBUG:boto:Params: {} 2022-09-13T07:35:12.573 DEBUG:boto:Token: None 2022-09-13T07:35:12.574 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.575 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8f4269ae65744b5-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.578 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.579 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.580 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.580 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:12.581 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:12.581 DEBUG:boto:Method: GET 2022-09-13T07:35:12.582 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:35:12.582 DEBUG:boto:Data: 2022-09-13T07:35:12.583 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.583 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.584 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.584 DEBUG:boto:Params: {} 2022-09-13T07:35:12.585 DEBUG:boto:Token: None 2022-09-13T07:35:12.586 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.586 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d42899531a353519-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.590 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:12.592 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.593 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.593 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.594 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.594 DEBUG:boto:Method: GET 2022-09-13T07:35:12.595 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.595 DEBUG:boto:Data: 2022-09-13T07:35:12.596 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.596 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.597 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.598 DEBUG:boto:Params: {} 2022-09-13T07:35:12.598 DEBUG:boto:Token: None 2022-09-13T07:35:12.599 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.600 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tHUBXQC/eR22AdJch0qgYQYHXR0= 2022-09-13T07:35:12.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tHUBXQC/eR22AdJch0qgYQYHXR0=', 'Content-Length': '0'} 2022-09-13T07:35:12.603 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8a2c52489f16437-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.646 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.647 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.647 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.648 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.648 DEBUG:boto:Method: GET 2022-09-13T07:35:12.649 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.649 DEBUG:boto:Data: 2022-09-13T07:35:12.650 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.650 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.651 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.651 DEBUG:boto:Params: {} 2022-09-13T07:35:12.652 DEBUG:boto:Token: None 2022-09-13T07:35:12.653 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.653 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tHUBXQC/eR22AdJch0qgYQYHXR0= 2022-09-13T07:35:12.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tHUBXQC/eR22AdJch0qgYQYHXR0=', 'Content-Length': '0'} 2022-09-13T07:35:12.656 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4fc0ecaac709c74-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.657 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.658 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.658 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.658 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.659 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.659 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.660 DEBUG:boto:Data: 2022-09-13T07:35:12.660 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.661 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.661 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.662 DEBUG:boto:Params: {} 2022-09-13T07:35:12.662 DEBUG:boto:Token: None 2022-09-13T07:35:12.663 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.663 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUvsdJC9hdyo3OffmKYuY3SLa6c= 2022-09-13T07:35:12.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUvsdJC9hdyo3OffmKYuY3SLa6c=', 'Content-Length': '0'} 2022-09-13T07:35:12.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e90461af9944cc61-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.667 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.668 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.669 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.669 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.669 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.670 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.670 DEBUG:boto:Data: 2022-09-13T07:35:12.671 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.671 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.672 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.672 DEBUG:boto:Params: {} 2022-09-13T07:35:12.673 DEBUG:boto:Token: None 2022-09-13T07:35:12.673 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.674 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUvsdJC9hdyo3OffmKYuY3SLa6c= 2022-09-13T07:35:12.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUvsdJC9hdyo3OffmKYuY3SLa6c=', 'Content-Length': '0'} 2022-09-13T07:35:12.677 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4ea3d30890837de-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:12.678 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.679 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.679 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.680 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.680 DEBUG:boto:Method: GET 2022-09-13T07:35:12.681 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.681 DEBUG:boto:Data: 2022-09-13T07:35:12.682 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.682 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.683 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.683 DEBUG:boto:Params: {} 2022-09-13T07:35:12.684 DEBUG:boto:Token: None 2022-09-13T07:35:12.685 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.685 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tHUBXQC/eR22AdJch0qgYQYHXR0= 2022-09-13T07:35:12.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tHUBXQC/eR22AdJch0qgYQYHXR0=', 'Content-Length': '0'} 2022-09-13T07:35:12.688 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad8aa4b09c5cabb7-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.689 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.690 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.690 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.691 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.691 DEBUG:boto:Method: GET 2022-09-13T07:35:12.692 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.692 DEBUG:boto:Data: 2022-09-13T07:35:12.693 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.693 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.694 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.694 DEBUG:boto:Params: {} 2022-09-13T07:35:12.695 DEBUG:boto:Token: None 2022-09-13T07:35:12.695 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.696 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tHUBXQC/eR22AdJch0qgYQYHXR0= 2022-09-13T07:35:12.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tHUBXQC/eR22AdJch0qgYQYHXR0=', 'Content-Length': '0'} 2022-09-13T07:35:12.699 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000acaaf551e4f69e01-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.700 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.700 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.701 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:12.701 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:12.702 DEBUG:boto:Method: GET 2022-09-13T07:35:12.702 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:12.703 DEBUG:boto:Data: 2022-09-13T07:35:12.703 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.704 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.704 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.705 DEBUG:boto:Params: {} 2022-09-13T07:35:12.705 DEBUG:boto:Token: None 2022-09-13T07:35:12.706 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.707 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004459b9cc2c0465ac-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.711 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.712 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.712 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.713 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:12.713 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:12.714 DEBUG:boto:Method: GET 2022-09-13T07:35:12.714 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:12.715 DEBUG:boto:Data: 2022-09-13T07:35:12.715 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.716 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.716 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.717 DEBUG:boto:Params: {} 2022-09-13T07:35:12.717 DEBUG:boto:Token: None 2022-09-13T07:35:12.718 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.719 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f44588bc31200a9-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.766 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.768 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.768 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.769 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.770 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.771 DEBUG:boto:Data: 2022-09-13T07:35:12.772 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.773 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.774 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.774 DEBUG:boto:Params: {} 2022-09-13T07:35:12.775 DEBUG:boto:Token: None 2022-09-13T07:35:12.776 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.777 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mmLDkWEl76J9ZvBfRaetkvBkbug= 2022-09-13T07:35:12.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mmLDkWEl76J9ZvBfRaetkvBkbug=', 'Content-Length': '0'} 2022-09-13T07:35:12.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e16e66092823d2f8-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.781 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.781 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.782 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.783 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.783 DEBUG:boto:Data: 2022-09-13T07:35:12.784 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.784 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.785 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.785 DEBUG:boto:Params: {} 2022-09-13T07:35:12.786 DEBUG:boto:Token: None 2022-09-13T07:35:12.786 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.787 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mmLDkWEl76J9ZvBfRaetkvBkbug= 2022-09-13T07:35:12.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mmLDkWEl76J9ZvBfRaetkvBkbug=', 'Content-Length': '0'} 2022-09-13T07:35:12.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010967fd7ceeccf1a-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:12.793 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.793 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.794 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.795 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.795 DEBUG:boto:Method: GET 2022-09-13T07:35:12.796 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.796 DEBUG:boto:Data: 2022-09-13T07:35:12.797 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.797 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.798 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.799 DEBUG:boto:Params: {} 2022-09-13T07:35:12.800 DEBUG:boto:Token: None 2022-09-13T07:35:12.800 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.801 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tHUBXQC/eR22AdJch0qgYQYHXR0= 2022-09-13T07:35:12.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tHUBXQC/eR22AdJch0qgYQYHXR0=', 'Content-Length': '0'} 2022-09-13T07:35:12.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5d26b537bc22c73-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.805 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.805 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:12.806 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.807 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.807 DEBUG:boto:Method: GET 2022-09-13T07:35:12.808 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.809 DEBUG:boto:Data: 2022-09-13T07:35:12.809 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.810 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.810 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.811 DEBUG:boto:Params: {} 2022-09-13T07:35:12.811 DEBUG:boto:Token: None 2022-09-13T07:35:12.812 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:12.813 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tHUBXQC/eR22AdJch0qgYQYHXR0= 2022-09-13T07:35:12.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tHUBXQC/eR22AdJch0qgYQYHXR0=', 'Content-Length': '0'} 2022-09-13T07:35:12.815 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a71d5fb25678780e-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.816 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.817 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.817 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:12.818 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:12.819 DEBUG:boto:Method: GET 2022-09-13T07:35:12.820 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:12.820 DEBUG:boto:Data: 2022-09-13T07:35:12.821 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.822 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.823 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.824 DEBUG:boto:Params: {} 2022-09-13T07:35:12.825 DEBUG:boto:Token: None 2022-09-13T07:35:12.826 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.826 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf1b6d72e6f2e12b-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.830 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.831 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.832 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.832 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:12.833 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:12.833 DEBUG:boto:Method: GET 2022-09-13T07:35:12.834 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:12.834 DEBUG:boto:Data: 2022-09-13T07:35:12.835 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.835 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.836 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.836 DEBUG:boto:Params: {} 2022-09-13T07:35:12.837 DEBUG:boto:Token: None 2022-09-13T07:35:12.837 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.838 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d91d62eccc214eb3-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.842 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:12.844 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.844 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.845 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.845 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.846 DEBUG:boto:Method: GET 2022-09-13T07:35:12.846 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.847 DEBUG:boto:Data: 2022-09-13T07:35:12.847 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.848 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.848 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.849 DEBUG:boto:Params: {} 2022-09-13T07:35:12.850 DEBUG:boto:Token: None 2022-09-13T07:35:12.850 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.851 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2qlgV9MS/6uQ+2NnEjTTwPsYHG4= 2022-09-13T07:35:12.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2qlgV9MS/6uQ+2NnEjTTwPsYHG4=', 'Content-Length': '0'} 2022-09-13T07:35:12.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b4411d5f6cf708a-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.854 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.855 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.855 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.856 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.856 DEBUG:boto:Method: GET 2022-09-13T07:35:12.857 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.857 DEBUG:boto:Data: 2022-09-13T07:35:12.858 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.858 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.859 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.859 DEBUG:boto:Params: {} 2022-09-13T07:35:12.860 DEBUG:boto:Token: None 2022-09-13T07:35:12.861 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.861 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2qlgV9MS/6uQ+2NnEjTTwPsYHG4= 2022-09-13T07:35:12.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2qlgV9MS/6uQ+2NnEjTTwPsYHG4=', 'Content-Length': '0'} 2022-09-13T07:35:12.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fedb2e4bd4302b9-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.865 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.865 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.866 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.866 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.867 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.867 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.868 DEBUG:boto:Data: 2022-09-13T07:35:12.868 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.868 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.869 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.869 DEBUG:boto:Params: {} 2022-09-13T07:35:12.870 DEBUG:boto:Token: None 2022-09-13T07:35:12.871 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.871 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2T12bOiyKwAnKYUXG/4wqE6vSts= 2022-09-13T07:35:12.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2T12bOiyKwAnKYUXG/4wqE6vSts=', 'Content-Length': '0'} 2022-09-13T07:35:12.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ee1b52cecf4c755-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.875 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.876 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.877 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.878 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.879 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.880 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.881 DEBUG:boto:Data: 2022-09-13T07:35:12.882 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.882 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.883 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.884 DEBUG:boto:Params: {} 2022-09-13T07:35:12.885 DEBUG:boto:Token: None 2022-09-13T07:35:12.886 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.887 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2T12bOiyKwAnKYUXG/4wqE6vSts= 2022-09-13T07:35:12.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2T12bOiyKwAnKYUXG/4wqE6vSts=', 'Content-Length': '0'} 2022-09-13T07:35:12.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001efcbbf4c93d1541-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:12.891 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.891 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.892 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.892 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.893 DEBUG:boto:Method: GET 2022-09-13T07:35:12.893 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.894 DEBUG:boto:Data: 2022-09-13T07:35:12.894 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.895 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.895 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.896 DEBUG:boto:Params: {} 2022-09-13T07:35:12.896 DEBUG:boto:Token: None 2022-09-13T07:35:12.897 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.898 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2qlgV9MS/6uQ+2NnEjTTwPsYHG4= 2022-09-13T07:35:12.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2qlgV9MS/6uQ+2NnEjTTwPsYHG4=', 'Content-Length': '0'} 2022-09-13T07:35:12.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f4fd66da7cf0a09-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.901 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.902 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.902 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.903 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.903 DEBUG:boto:Method: GET 2022-09-13T07:35:12.904 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.904 DEBUG:boto:Data: 2022-09-13T07:35:12.905 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.905 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.906 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.906 DEBUG:boto:Params: {} 2022-09-13T07:35:12.907 DEBUG:boto:Token: None 2022-09-13T07:35:12.908 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:12.908 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2qlgV9MS/6uQ+2NnEjTTwPsYHG4= 2022-09-13T07:35:12.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2qlgV9MS/6uQ+2NnEjTTwPsYHG4=', 'Content-Length': '0'} 2022-09-13T07:35:12.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041e06dfd63298c4c-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.912 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.912 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.913 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:12.913 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:12.914 DEBUG:boto:Method: GET 2022-09-13T07:35:12.914 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:12.915 DEBUG:boto:Data: 2022-09-13T07:35:12.915 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.916 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.916 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.917 DEBUG:boto:Params: {} 2022-09-13T07:35:12.917 DEBUG:boto:Token: None 2022-09-13T07:35:12.918 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.919 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b773b6f23e13c2f6-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.923 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.923 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:12.924 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:12.925 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:12.926 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:12.926 DEBUG:boto:Method: GET 2022-09-13T07:35:12.927 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:12.928 DEBUG:boto:Data: 2022-09-13T07:35:12.929 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.930 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.931 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.932 DEBUG:boto:Params: {} 2022-09-13T07:35:12.933 DEBUG:boto:Token: None 2022-09-13T07:35:12.934 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/ 2022-09-13T07:35:12.935 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc= 2022-09-13T07:35:12.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zZBtdu1MGYmgET7/eCufB4uyiTc=', 'Content-Length': '0'} 2022-09-13T07:35:12.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097c539134b39e15b-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.982 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:12.983 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.984 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.985 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.985 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.986 DEBUG:boto:Data: 2022-09-13T07:35:12.986 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.987 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:12.987 DEBUG:boto:Port: 8001 2022-09-13T07:35:12.988 DEBUG:boto:Params: {} 2022-09-13T07:35:12.988 DEBUG:boto:Token: None 2022-09-13T07:35:12.989 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:12 GMT /yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.990 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nAyMp136eeK9WHCOGyOmztJiAwY= 2022-09-13T07:35:12.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nAyMp136eeK9WHCOGyOmztJiAwY=', 'Content-Length': '0'} 2022-09-13T07:35:12.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062f9576233cc1617-00632032b0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:12.993 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.994 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.995 DEBUG:boto:Method: HEAD 2022-09-13T07:35:12.995 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2 2022-09-13T07:35:12.996 DEBUG:boto:Data: 2022-09-13T07:35:12.996 DEBUG:boto:Headers: {} 2022-09-13T07:35:12.997 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:12.997 DEBUG:boto:Port: 8000 2022-09-13T07:35:12.998 DEBUG:boto:Params: {} 2022-09-13T07:35:12.999 DEBUG:boto:Token: None 2022-09-13T07:35:13.000 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone2 2022-09-13T07:35:13.001 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3Y1eci9ZXRjZe0lcVTzkaFOx+LY= 2022-09-13T07:35:13.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3Y1eci9ZXRjZe0lcVTzkaFOx+LY=', 'Content-Length': '0'} 2022-09-13T07:35:13.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006462aadd01f62a18-00632032b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:13.005 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:13.006 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:13.007 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.007 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.008 DEBUG:boto:Method: GET 2022-09-13T07:35:13.008 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.009 DEBUG:boto:Data: 2022-09-13T07:35:13.009 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.009 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.011 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.011 DEBUG:boto:Params: {} 2022-09-13T07:35:13.012 DEBUG:boto:Token: None 2022-09-13T07:35:13.012 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.013 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Z0ULyuT9TkxObnK/Nevlud2c8VU= 2022-09-13T07:35:13.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Z0ULyuT9TkxObnK/Nevlud2c8VU=', 'Content-Length': '0'} 2022-09-13T07:35:13.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077792ac589eb3061-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.017 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:13.017 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:13.018 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.018 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.019 DEBUG:boto:Method: GET 2022-09-13T07:35:13.019 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.020 DEBUG:boto:Data: 2022-09-13T07:35:13.020 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.021 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:13.021 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.022 DEBUG:boto:Params: {} 2022-09-13T07:35:13.022 DEBUG:boto:Token: None 2022-09-13T07:35:13.023 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.023 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Z0ULyuT9TkxObnK/Nevlud2c8VU= 2022-09-13T07:35:13.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Z0ULyuT9TkxObnK/Nevlud2c8VU=', 'Content-Length': '0'} 2022-09-13T07:35:13.026 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc4eae5131f5ecff-00632032b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.027 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.028 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.028 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:13.029 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:13.029 DEBUG:boto:Method: GET 2022-09-13T07:35:13.030 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:13.030 DEBUG:boto:Data: 2022-09-13T07:35:13.031 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.031 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.032 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.033 DEBUG:boto:Params: {} 2022-09-13T07:35:13.033 DEBUG:boto:Token: None 2022-09-13T07:35:13.034 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.035 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c= 2022-09-13T07:35:13.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c=', 'Content-Length': '0'} 2022-09-13T07:35:13.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088e8a3519ed44208-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.039 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.040 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.040 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.041 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:13.041 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:13.042 DEBUG:boto:Method: GET 2022-09-13T07:35:13.042 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:13.043 DEBUG:boto:Data: 2022-09-13T07:35:13.043 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.044 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:13.045 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.045 DEBUG:boto:Params: {} 2022-09-13T07:35:13.046 DEBUG:boto:Token: None 2022-09-13T07:35:13.046 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.047 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c= 2022-09-13T07:35:13.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c=', 'Content-Length': '0'} 2022-09-13T07:35:13.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d285597217549961-00632032b1-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.051 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:13.052 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.053 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.053 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.054 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.054 DEBUG:boto:Method: GET 2022-09-13T07:35:13.055 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.055 DEBUG:boto:Data: 2022-09-13T07:35:13.056 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.056 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.057 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.057 DEBUG:boto:Params: {} 2022-09-13T07:35:13.058 DEBUG:boto:Token: None 2022-09-13T07:35:13.059 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.060 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:srn6hSiSZbzUs0ZV8sd4WeJIMec= 2022-09-13T07:35:13.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:srn6hSiSZbzUs0ZV8sd4WeJIMec=', 'Content-Length': '0'} 2022-09-13T07:35:13.063 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0b6f28f1303fc60-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.106 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.107 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.107 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.108 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.108 DEBUG:boto:Method: GET 2022-09-13T07:35:13.109 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.109 DEBUG:boto:Data: 2022-09-13T07:35:13.110 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.111 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:13.112 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.113 DEBUG:boto:Params: {} 2022-09-13T07:35:13.114 DEBUG:boto:Token: None 2022-09-13T07:35:13.115 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.116 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:srn6hSiSZbzUs0ZV8sd4WeJIMec= 2022-09-13T07:35:13.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:srn6hSiSZbzUs0ZV8sd4WeJIMec=', 'Content-Length': '0'} 2022-09-13T07:35:13.120 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d121620688d31061-00632032b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.121 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.122 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.123 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.124 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.125 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.125 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.126 DEBUG:boto:Data: 2022-09-13T07:35:13.127 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.128 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.129 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.129 DEBUG:boto:Params: {} 2022-09-13T07:35:13.130 DEBUG:boto:Token: None 2022-09-13T07:35:13.131 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.132 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GLa5KTAbV9ycaNYIdP+Jn0aplPE= 2022-09-13T07:35:13.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GLa5KTAbV9ycaNYIdP+Jn0aplPE=', 'Content-Length': '0'} 2022-09-13T07:35:13.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000427ea37686422e8e-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.137 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.138 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.138 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.139 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.140 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.141 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.141 DEBUG:boto:Data: 2022-09-13T07:35:13.142 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.143 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:13.144 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.145 DEBUG:boto:Params: {} 2022-09-13T07:35:13.146 DEBUG:boto:Token: None 2022-09-13T07:35:13.146 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.147 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GLa5KTAbV9ycaNYIdP+Jn0aplPE= 2022-09-13T07:35:13.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GLa5KTAbV9ycaNYIdP+Jn0aplPE=', 'Content-Length': '0'} 2022-09-13T07:35:13.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8a94bff02d63f50-00632032b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:13.152 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.153 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.153 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.154 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.155 DEBUG:boto:Method: GET 2022-09-13T07:35:13.156 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.157 DEBUG:boto:Data: 2022-09-13T07:35:13.158 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.159 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.160 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.160 DEBUG:boto:Params: {} 2022-09-13T07:35:13.161 DEBUG:boto:Token: None 2022-09-13T07:35:13.162 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.163 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:srn6hSiSZbzUs0ZV8sd4WeJIMec= 2022-09-13T07:35:13.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:srn6hSiSZbzUs0ZV8sd4WeJIMec=', 'Content-Length': '0'} 2022-09-13T07:35:13.167 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a4edd63f6178e22-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.168 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.169 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.170 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.171 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.172 DEBUG:boto:Method: GET 2022-09-13T07:35:13.172 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.173 DEBUG:boto:Data: 2022-09-13T07:35:13.174 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.175 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:13.176 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.177 DEBUG:boto:Params: {} 2022-09-13T07:35:13.178 DEBUG:boto:Token: None 2022-09-13T07:35:13.179 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.193 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:srn6hSiSZbzUs0ZV8sd4WeJIMec= 2022-09-13T07:35:13.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:srn6hSiSZbzUs0ZV8sd4WeJIMec=', 'Content-Length': '0'} 2022-09-13T07:35:13.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000481a147e3007b6f9-00632032b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.197 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.198 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.198 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:13.199 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:13.200 DEBUG:boto:Method: GET 2022-09-13T07:35:13.200 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:13.201 DEBUG:boto:Data: 2022-09-13T07:35:13.201 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.202 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.203 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.203 DEBUG:boto:Params: {} 2022-09-13T07:35:13.204 DEBUG:boto:Token: None 2022-09-13T07:35:13.205 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.205 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c= 2022-09-13T07:35:13.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c=', 'Content-Length': '0'} 2022-09-13T07:35:13.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cfc0e4edc2ff5cf-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.210 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.211 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.211 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.212 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:13.212 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:13.213 DEBUG:boto:Method: GET 2022-09-13T07:35:13.214 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:13.214 DEBUG:boto:Data: 2022-09-13T07:35:13.215 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.215 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:13.216 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.216 DEBUG:boto:Params: {} 2022-09-13T07:35:13.217 DEBUG:boto:Token: None 2022-09-13T07:35:13.218 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.218 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c= 2022-09-13T07:35:13.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c=', 'Content-Length': '0'} 2022-09-13T07:35:13.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014cf7b13aff526c7-00632032b1-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.223 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.223 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.224 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.224 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.225 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.225 DEBUG:boto:Data: 2022-09-13T07:35:13.226 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.227 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.227 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.227 DEBUG:boto:Params: {} 2022-09-13T07:35:13.228 DEBUG:boto:Token: None 2022-09-13T07:35:13.229 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.230 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qesXRfeXQnViVGX71JRhpbZCKfI= 2022-09-13T07:35:13.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qesXRfeXQnViVGX71JRhpbZCKfI=', 'Content-Length': '0'} 2022-09-13T07:35:13.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c329c9b34c4e3f6a-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.233 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.234 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.235 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.235 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.236 DEBUG:boto:Data: 2022-09-13T07:35:13.236 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.237 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:13.238 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.238 DEBUG:boto:Params: {} 2022-09-13T07:35:13.239 DEBUG:boto:Token: None 2022-09-13T07:35:13.239 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.240 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qesXRfeXQnViVGX71JRhpbZCKfI= 2022-09-13T07:35:13.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qesXRfeXQnViVGX71JRhpbZCKfI=', 'Content-Length': '0'} 2022-09-13T07:35:13.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000988bb28514be1175-00632032b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:13.244 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.245 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.246 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.247 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.247 DEBUG:boto:Method: GET 2022-09-13T07:35:13.248 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.248 DEBUG:boto:Data: 2022-09-13T07:35:13.249 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.249 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.250 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.251 DEBUG:boto:Params: {} 2022-09-13T07:35:13.251 DEBUG:boto:Token: None 2022-09-13T07:35:13.252 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.253 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:srn6hSiSZbzUs0ZV8sd4WeJIMec= 2022-09-13T07:35:13.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:srn6hSiSZbzUs0ZV8sd4WeJIMec=', 'Content-Length': '0'} 2022-09-13T07:35:13.255 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab90d8f8b6e89e80-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.256 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.257 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:13.257 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.258 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.258 DEBUG:boto:Method: GET 2022-09-13T07:35:13.259 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.259 DEBUG:boto:Data: 2022-09-13T07:35:13.260 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.260 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:13.261 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.262 DEBUG:boto:Params: {} 2022-09-13T07:35:13.262 DEBUG:boto:Token: None 2022-09-13T07:35:13.263 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:13.263 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:srn6hSiSZbzUs0ZV8sd4WeJIMec= 2022-09-13T07:35:13.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:srn6hSiSZbzUs0ZV8sd4WeJIMec=', 'Content-Length': '0'} 2022-09-13T07:35:13.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003eaa3d2dba84fcdf-00632032b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.310 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.311 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.311 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:13.312 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:13.312 DEBUG:boto:Method: GET 2022-09-13T07:35:13.313 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:13.313 DEBUG:boto:Data: 2022-09-13T07:35:13.314 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.315 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.315 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.316 DEBUG:boto:Params: {} 2022-09-13T07:35:13.316 DEBUG:boto:Token: None 2022-09-13T07:35:13.317 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.317 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c= 2022-09-13T07:35:13.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c=', 'Content-Length': '0'} 2022-09-13T07:35:13.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042deb1d3c95fecd2-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.321 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.322 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.323 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.323 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:13.324 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:13.324 DEBUG:boto:Method: GET 2022-09-13T07:35:13.325 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:13.325 DEBUG:boto:Data: 2022-09-13T07:35:13.326 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.326 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:13.327 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.327 DEBUG:boto:Params: {} 2022-09-13T07:35:13.328 DEBUG:boto:Token: None 2022-09-13T07:35:13.329 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.329 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c= 2022-09-13T07:35:13.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c=', 'Content-Length': '0'} 2022-09-13T07:35:13.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045e9f10bebfa1fc7-00632032b1-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.333 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.334 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-18 zones={test-zone3, test-zone1} 2022-09-13T07:35:13.335 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-18 zones={test-zone3, test-zone2} 2022-09-13T07:35:13.335 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.336 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.336 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.337 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:35:13.337 DEBUG:boto:Data: 2022-09-13T07:35:13.338 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.338 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.339 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.339 DEBUG:boto:Params: {} 2022-09-13T07:35:13.340 DEBUG:boto:Token: None 2022-09-13T07:35:13.340 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.341 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:agoAk/io1QHe1/NnIES7H7A/1a0= 2022-09-13T07:35:13.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:agoAk/io1QHe1/NnIES7H7A/1a0=', 'Content-Length': '0'} 2022-09-13T07:35:13.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073335da415c5bbd0-00632032b1-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.345 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.345 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.346 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.346 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:35:13.347 DEBUG:boto:Data: 2022-09-13T07:35:13.347 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.348 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.349 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.349 DEBUG:boto:Params: {} 2022-09-13T07:35:13.350 DEBUG:boto:Token: None 2022-09-13T07:35:13.351 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.351 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:agoAk/io1QHe1/NnIES7H7A/1a0= 2022-09-13T07:35:13.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:agoAk/io1QHe1/NnIES7H7A/1a0=', 'Content-Length': '0'} 2022-09-13T07:35:13.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045b2c9f1a3064265-00632032b1-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.355 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:35:13.356 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.357 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.358 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:13.359 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:13.360 DEBUG:boto:Method: GET 2022-09-13T07:35:13.361 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:13.362 DEBUG:boto:Data: 2022-09-13T07:35:13.363 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.363 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.364 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.365 DEBUG:boto:Params: {} 2022-09-13T07:35:13.366 DEBUG:boto:Token: None 2022-09-13T07:35:13.367 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/?versions 2022-09-13T07:35:13.368 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nO4yh09NgVmg6plIe1+UwM9EmUc= 2022-09-13T07:35:13.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nO4yh09NgVmg6plIe1+UwM9EmUc=', 'Content-Length': '0'} 2022-09-13T07:35:13.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000950eae6f19839d85-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.374 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.375 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:35:13.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:35:13.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:35:13.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:35:13.379 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:35:13.380 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.381 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.382 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:13.383 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:13.383 DEBUG:boto:Method: GET 2022-09-13T07:35:13.384 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:13.385 DEBUG:boto:Data: 2022-09-13T07:35:13.386 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.387 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.388 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.389 DEBUG:boto:Params: {} 2022-09-13T07:35:13.390 DEBUG:boto:Token: None 2022-09-13T07:35:13.391 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/?versions 2022-09-13T07:35:13.392 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nO4yh09NgVmg6plIe1+UwM9EmUc= 2022-09-13T07:35:13.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nO4yh09NgVmg6plIe1+UwM9EmUc=', 'Content-Length': '0'} 2022-09-13T07:35:13.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021d3d6ed2e380d80-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.397 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.399 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:35:13.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:35:13.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:35:13.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:35:13.402 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.403 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.404 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:13.405 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:13.406 DEBUG:boto:Method: GET 2022-09-13T07:35:13.407 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:13.407 DEBUG:boto:Data: 2022-09-13T07:35:13.408 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.409 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.410 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.411 DEBUG:boto:Params: {} 2022-09-13T07:35:13.412 DEBUG:boto:Token: None 2022-09-13T07:35:13.413 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/?versions 2022-09-13T07:35:13.414 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nO4yh09NgVmg6plIe1+UwM9EmUc= 2022-09-13T07:35:13.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nO4yh09NgVmg6plIe1+UwM9EmUc=', 'Content-Length': '0'} 2022-09-13T07:35:13.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008503558b27bf2f28-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.420 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.421 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.421 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.422 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:13.422 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:13.423 DEBUG:boto:Method: GET 2022-09-13T07:35:13.423 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:35:13.424 DEBUG:boto:Data: 2022-09-13T07:35:13.424 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.425 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.425 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.426 DEBUG:boto:Params: {} 2022-09-13T07:35:13.426 DEBUG:boto:Token: None 2022-09-13T07:35:13.427 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/?versions 2022-09-13T07:35:13.428 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nO4yh09NgVmg6plIe1+UwM9EmUc= 2022-09-13T07:35:13.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nO4yh09NgVmg6plIe1+UwM9EmUc=', 'Content-Length': '0'} 2022-09-13T07:35:13.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfacebb2249bef05-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.432 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:13.435 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:13.436 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:13.436 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.437 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.438 DEBUG:boto:Method: GET 2022-09-13T07:35:13.439 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.441 DEBUG:boto:Data: 2022-09-13T07:35:13.442 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.443 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.444 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.445 DEBUG:boto:Params: {} 2022-09-13T07:35:13.446 DEBUG:boto:Token: None 2022-09-13T07:35:13.447 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.448 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gqT4/XPdpPywAcGlbzEmaTy5yZE= 2022-09-13T07:35:13.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gqT4/XPdpPywAcGlbzEmaTy5yZE=', 'Content-Length': '0'} 2022-09-13T07:35:13.452 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008a4a611c89340121-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.453 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:13.454 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:13.455 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.456 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.457 DEBUG:boto:Method: GET 2022-09-13T07:35:13.457 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.458 DEBUG:boto:Data: 2022-09-13T07:35:13.459 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.460 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.461 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.462 DEBUG:boto:Params: {} 2022-09-13T07:35:13.463 DEBUG:boto:Token: None 2022-09-13T07:35:13.465 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.466 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gqT4/XPdpPywAcGlbzEmaTy5yZE= 2022-09-13T07:35:13.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gqT4/XPdpPywAcGlbzEmaTy5yZE=', 'Content-Length': '0'} 2022-09-13T07:35:13.469 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e571f64723fb4302-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.470 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:13.471 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:13.473 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.474 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.475 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.476 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.476 DEBUG:boto:Data: 2022-09-13T07:35:13.477 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.478 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.479 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.480 DEBUG:boto:Params: {} 2022-09-13T07:35:13.481 DEBUG:boto:Token: None 2022-09-13T07:35:13.482 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.483 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uq7eFlCWcuL3mqQwBu7w9aIPO2I= 2022-09-13T07:35:13.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uq7eFlCWcuL3mqQwBu7w9aIPO2I=', 'Content-Length': '0'} 2022-09-13T07:35:13.487 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000777621e87b81897c-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.488 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:13.489 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:13.489 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.490 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.491 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.492 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.492 DEBUG:boto:Data: 2022-09-13T07:35:13.493 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.494 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.495 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.496 DEBUG:boto:Params: {} 2022-09-13T07:35:13.497 DEBUG:boto:Token: None 2022-09-13T07:35:13.498 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.499 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uq7eFlCWcuL3mqQwBu7w9aIPO2I= 2022-09-13T07:35:13.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uq7eFlCWcuL3mqQwBu7w9aIPO2I=', 'Content-Length': '0'} 2022-09-13T07:35:13.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016bcc44fab060819-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:13.505 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:13.505 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:13.506 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.507 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.507 DEBUG:boto:Method: GET 2022-09-13T07:35:13.508 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.508 DEBUG:boto:Data: 2022-09-13T07:35:13.509 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.509 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.510 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.510 DEBUG:boto:Params: {} 2022-09-13T07:35:13.511 DEBUG:boto:Token: None 2022-09-13T07:35:13.512 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.512 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gqT4/XPdpPywAcGlbzEmaTy5yZE= 2022-09-13T07:35:13.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gqT4/XPdpPywAcGlbzEmaTy5yZE=', 'Content-Length': '0'} 2022-09-13T07:35:13.516 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060ea95a08e620d8c-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.516 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:13.517 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:13.518 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.518 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.519 DEBUG:boto:Method: GET 2022-09-13T07:35:13.519 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.520 DEBUG:boto:Data: 2022-09-13T07:35:13.520 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.521 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.521 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.522 DEBUG:boto:Params: {} 2022-09-13T07:35:13.523 DEBUG:boto:Token: None 2022-09-13T07:35:13.523 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.524 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gqT4/XPdpPywAcGlbzEmaTy5yZE= 2022-09-13T07:35:13.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gqT4/XPdpPywAcGlbzEmaTy5yZE=', 'Content-Length': '0'} 2022-09-13T07:35:13.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f96bbf653c1a15d1-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.528 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.529 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.530 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:13.531 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:13.532 DEBUG:boto:Method: GET 2022-09-13T07:35:13.533 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:35:13.533 DEBUG:boto:Data: 2022-09-13T07:35:13.534 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.535 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.536 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.537 DEBUG:boto:Params: {} 2022-09-13T07:35:13.538 DEBUG:boto:Token: None 2022-09-13T07:35:13.540 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.540 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c= 2022-09-13T07:35:13.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c=', 'Content-Length': '0'} 2022-09-13T07:35:13.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003df121374bb3c360-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.586 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.587 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.588 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.588 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:13.589 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:13.589 DEBUG:boto:Method: GET 2022-09-13T07:35:13.590 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:35:13.590 DEBUG:boto:Data: 2022-09-13T07:35:13.591 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.591 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.592 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.592 DEBUG:boto:Params: {} 2022-09-13T07:35:13.593 DEBUG:boto:Token: None 2022-09-13T07:35:13.593 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.594 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c= 2022-09-13T07:35:13.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c=', 'Content-Length': '0'} 2022-09-13T07:35:13.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc81b39cfc6aa4a4-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.597 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.598 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:13.599 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:13.599 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.600 DEBUG:boto:Path: /yxvvuh-18/dummy 2022-09-13T07:35:13.600 DEBUG:boto:Data: 2022-09-13T07:35:13.601 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.601 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.601 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.602 DEBUG:boto:Params: {} 2022-09-13T07:35:13.603 DEBUG:boto:Token: None 2022-09-13T07:35:13.603 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/dummy 2022-09-13T07:35:13.604 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rrAUraQZk0lg6eYWV+k0Puhg4yk= 2022-09-13T07:35:13.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rrAUraQZk0lg6eYWV+k0Puhg4yk=', 'Content-Length': '0'} 2022-09-13T07:35:13.607 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000823623f0d80cc61f-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.607 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:13.608 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:13.608 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.609 DEBUG:boto:Path: /yxvvuh-18/dummy 2022-09-13T07:35:13.609 DEBUG:boto:Data: 2022-09-13T07:35:13.610 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.610 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.611 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.611 DEBUG:boto:Params: {} 2022-09-13T07:35:13.612 DEBUG:boto:Token: None 2022-09-13T07:35:13.612 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/dummy 2022-09-13T07:35:13.613 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rrAUraQZk0lg6eYWV+k0Puhg4yk= 2022-09-13T07:35:13.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rrAUraQZk0lg6eYWV+k0Puhg4yk=', 'Content-Length': '0'} 2022-09-13T07:35:13.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000093e696a83fc22975-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:13.619 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:13.620 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:13.620 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.621 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.621 DEBUG:boto:Method: GET 2022-09-13T07:35:13.622 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.622 DEBUG:boto:Data: 2022-09-13T07:35:13.623 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.623 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.624 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.624 DEBUG:boto:Params: {} 2022-09-13T07:35:13.625 DEBUG:boto:Token: None 2022-09-13T07:35:13.625 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.626 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gqT4/XPdpPywAcGlbzEmaTy5yZE= 2022-09-13T07:35:13.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gqT4/XPdpPywAcGlbzEmaTy5yZE=', 'Content-Length': '0'} 2022-09-13T07:35:13.629 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a69eaa4d5b05595a-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.630 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:35:13.630 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:35:13.631 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.631 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.632 DEBUG:boto:Method: GET 2022-09-13T07:35:13.633 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.633 DEBUG:boto:Data: 2022-09-13T07:35:13.634 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.635 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.635 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.636 DEBUG:boto:Params: {} 2022-09-13T07:35:13.637 DEBUG:boto:Token: None 2022-09-13T07:35:13.638 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:35:13.639 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gqT4/XPdpPywAcGlbzEmaTy5yZE= 2022-09-13T07:35:13.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gqT4/XPdpPywAcGlbzEmaTy5yZE=', 'Content-Length': '0'} 2022-09-13T07:35:13.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000067ba71ebe43264dc-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.643 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.643 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.644 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:13.644 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:13.645 DEBUG:boto:Method: GET 2022-09-13T07:35:13.647 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:35:13.648 DEBUG:boto:Data: 2022-09-13T07:35:13.649 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.650 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.651 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.654 DEBUG:boto:Params: {} 2022-09-13T07:35:13.655 DEBUG:boto:Token: None 2022-09-13T07:35:13.656 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.658 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c= 2022-09-13T07:35:13.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c=', 'Content-Length': '0'} 2022-09-13T07:35:13.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063689ab14366f85f-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.664 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.666 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.667 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.668 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:13.669 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:35:13.670 DEBUG:boto:Method: GET 2022-09-13T07:35:13.671 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:35:13.672 DEBUG:boto:Data: 2022-09-13T07:35:13.673 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.674 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.675 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.676 DEBUG:boto:Params: {} 2022-09-13T07:35:13.677 DEBUG:boto:Token: None 2022-09-13T07:35:13.678 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.679 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c= 2022-09-13T07:35:13.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c=', 'Content-Length': '0'} 2022-09-13T07:35:13.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b753f240f33e1306-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.684 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:13.685 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.686 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.686 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.687 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.687 DEBUG:boto:Method: GET 2022-09-13T07:35:13.688 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.688 DEBUG:boto:Data: 2022-09-13T07:35:13.689 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.689 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.690 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.691 DEBUG:boto:Params: {} 2022-09-13T07:35:13.691 DEBUG:boto:Token: None 2022-09-13T07:35:13.692 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.693 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wjtAPo1mnKeM/YOGQgJV7gQ/lak= 2022-09-13T07:35:13.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wjtAPo1mnKeM/YOGQgJV7gQ/lak=', 'Content-Length': '0'} 2022-09-13T07:35:13.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ec1d5b7b46560cc-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.738 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.739 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.740 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.741 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.742 DEBUG:boto:Method: GET 2022-09-13T07:35:13.743 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.744 DEBUG:boto:Data: 2022-09-13T07:35:13.745 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.746 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.747 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.748 DEBUG:boto:Params: {} 2022-09-13T07:35:13.749 DEBUG:boto:Token: None 2022-09-13T07:35:13.750 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.751 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wjtAPo1mnKeM/YOGQgJV7gQ/lak= 2022-09-13T07:35:13.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wjtAPo1mnKeM/YOGQgJV7gQ/lak=', 'Content-Length': '0'} 2022-09-13T07:35:13.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9bed03e15bb5bc2-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.758 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.758 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.760 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.761 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.761 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.762 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.763 DEBUG:boto:Data: 2022-09-13T07:35:13.764 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.765 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.765 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.766 DEBUG:boto:Params: {} 2022-09-13T07:35:13.766 DEBUG:boto:Token: None 2022-09-13T07:35:13.767 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.767 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SimlHR2gZsB3ige4GVyiAtBnyYA= 2022-09-13T07:35:13.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SimlHR2gZsB3ige4GVyiAtBnyYA=', 'Content-Length': '0'} 2022-09-13T07:35:13.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000468e1e729eaab390-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.771 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.772 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.773 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.774 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.775 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.776 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.777 DEBUG:boto:Data: 2022-09-13T07:35:13.777 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.778 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.779 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.780 DEBUG:boto:Params: {} 2022-09-13T07:35:13.781 DEBUG:boto:Token: None 2022-09-13T07:35:13.783 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.784 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SimlHR2gZsB3ige4GVyiAtBnyYA= 2022-09-13T07:35:13.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SimlHR2gZsB3ige4GVyiAtBnyYA=', 'Content-Length': '0'} 2022-09-13T07:35:13.789 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cf1bbe11106199e-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:13.791 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.792 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.792 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.793 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.794 DEBUG:boto:Method: GET 2022-09-13T07:35:13.795 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.796 DEBUG:boto:Data: 2022-09-13T07:35:13.796 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.797 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.798 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.799 DEBUG:boto:Params: {} 2022-09-13T07:35:13.800 DEBUG:boto:Token: None 2022-09-13T07:35:13.801 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.802 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wjtAPo1mnKeM/YOGQgJV7gQ/lak= 2022-09-13T07:35:13.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wjtAPo1mnKeM/YOGQgJV7gQ/lak=', 'Content-Length': '0'} 2022-09-13T07:35:13.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006120e307e7b02289-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.806 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.806 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.807 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.807 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.808 DEBUG:boto:Method: GET 2022-09-13T07:35:13.808 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.809 DEBUG:boto:Data: 2022-09-13T07:35:13.809 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.810 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.810 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.811 DEBUG:boto:Params: {} 2022-09-13T07:35:13.811 DEBUG:boto:Token: None 2022-09-13T07:35:13.812 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.813 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wjtAPo1mnKeM/YOGQgJV7gQ/lak= 2022-09-13T07:35:13.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wjtAPo1mnKeM/YOGQgJV7gQ/lak=', 'Content-Length': '0'} 2022-09-13T07:35:13.816 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f54b111cd1789720-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.816 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.817 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.817 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:13.818 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:13.819 DEBUG:boto:Method: GET 2022-09-13T07:35:13.819 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:13.820 DEBUG:boto:Data: 2022-09-13T07:35:13.820 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.820 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.821 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.821 DEBUG:boto:Params: {} 2022-09-13T07:35:13.822 DEBUG:boto:Token: None 2022-09-13T07:35:13.823 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.823 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c= 2022-09-13T07:35:13.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c=', 'Content-Length': '0'} 2022-09-13T07:35:13.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000236684c33192d0f9-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.827 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.828 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.829 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.830 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:13.831 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:13.832 DEBUG:boto:Method: GET 2022-09-13T07:35:13.833 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:13.834 DEBUG:boto:Data: 2022-09-13T07:35:13.835 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.836 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.837 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.838 DEBUG:boto:Params: {} 2022-09-13T07:35:13.839 DEBUG:boto:Token: None 2022-09-13T07:35:13.840 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.840 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c= 2022-09-13T07:35:13.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c=', 'Content-Length': '0'} 2022-09-13T07:35:13.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007fe83088c9f83ce-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.844 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.845 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.845 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.846 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.847 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.847 DEBUG:boto:Data: 2022-09-13T07:35:13.848 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.848 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.849 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.849 DEBUG:boto:Params: {} 2022-09-13T07:35:13.850 DEBUG:boto:Token: None 2022-09-13T07:35:13.850 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.851 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:26NVOIdsHmXReaSRxpkj8RQr5HM= 2022-09-13T07:35:13.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:26NVOIdsHmXReaSRxpkj8RQr5HM=', 'Content-Length': '0'} 2022-09-13T07:35:13.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa2ec9f9c2e52306-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.855 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.856 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.856 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.857 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.857 DEBUG:boto:Data: 2022-09-13T07:35:13.858 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.858 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.859 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.859 DEBUG:boto:Params: {} 2022-09-13T07:35:13.860 DEBUG:boto:Token: None 2022-09-13T07:35:13.860 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.861 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:26NVOIdsHmXReaSRxpkj8RQr5HM= 2022-09-13T07:35:13.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:26NVOIdsHmXReaSRxpkj8RQr5HM=', 'Content-Length': '0'} 2022-09-13T07:35:13.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002db0bcac25a33fe1-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:13.865 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.865 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.866 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.866 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.867 DEBUG:boto:Method: GET 2022-09-13T07:35:13.867 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.868 DEBUG:boto:Data: 2022-09-13T07:35:13.868 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.869 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.869 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.870 DEBUG:boto:Params: {} 2022-09-13T07:35:13.870 DEBUG:boto:Token: None 2022-09-13T07:35:13.871 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.872 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wjtAPo1mnKeM/YOGQgJV7gQ/lak= 2022-09-13T07:35:13.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wjtAPo1mnKeM/YOGQgJV7gQ/lak=', 'Content-Length': '0'} 2022-09-13T07:35:13.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4283d74eb180ced-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.918 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.919 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:13.920 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.920 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.921 DEBUG:boto:Method: GET 2022-09-13T07:35:13.921 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.922 DEBUG:boto:Data: 2022-09-13T07:35:13.923 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.924 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.925 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.926 DEBUG:boto:Params: {} 2022-09-13T07:35:13.926 DEBUG:boto:Token: None 2022-09-13T07:35:13.927 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:35:13.927 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wjtAPo1mnKeM/YOGQgJV7gQ/lak= 2022-09-13T07:35:13.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wjtAPo1mnKeM/YOGQgJV7gQ/lak=', 'Content-Length': '0'} 2022-09-13T07:35:13.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce6c58d69ccad5ba-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.931 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.932 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.932 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:13.933 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:13.933 DEBUG:boto:Method: GET 2022-09-13T07:35:13.934 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:13.934 DEBUG:boto:Data: 2022-09-13T07:35:13.935 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.935 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.936 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.936 DEBUG:boto:Params: {} 2022-09-13T07:35:13.937 DEBUG:boto:Token: None 2022-09-13T07:35:13.937 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.938 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c= 2022-09-13T07:35:13.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c=', 'Content-Length': '0'} 2022-09-13T07:35:13.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0966c810504229c-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.942 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.943 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:13.943 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:13.944 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:13.944 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:13.945 DEBUG:boto:Method: GET 2022-09-13T07:35:13.945 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:35:13.946 DEBUG:boto:Data: 2022-09-13T07:35:13.946 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.947 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.947 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.948 DEBUG:boto:Params: {} 2022-09-13T07:35:13.948 DEBUG:boto:Token: None 2022-09-13T07:35:13.949 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/ 2022-09-13T07:35:13.949 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c= 2022-09-13T07:35:13.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MGzJ4L2plTV8rnKSRw1hTSVrP5c=', 'Content-Length': '0'} 2022-09-13T07:35:13.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcb0f92edaae8772-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.953 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:13.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:13.955 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:13.955 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:13.956 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.956 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.957 DEBUG:boto:Method: GET 2022-09-13T07:35:13.957 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.958 DEBUG:boto:Data: 2022-09-13T07:35:13.958 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.959 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.960 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.960 DEBUG:boto:Params: {} 2022-09-13T07:35:13.961 DEBUG:boto:Token: None 2022-09-13T07:35:13.961 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.962 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Z0ULyuT9TkxObnK/Nevlud2c8VU= 2022-09-13T07:35:13.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Z0ULyuT9TkxObnK/Nevlud2c8VU=', 'Content-Length': '0'} 2022-09-13T07:35:13.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006a4176c636b51ad-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.966 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:13.967 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:13.967 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.968 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.968 DEBUG:boto:Method: GET 2022-09-13T07:35:13.969 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.969 DEBUG:boto:Data: 2022-09-13T07:35:13.970 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.970 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.971 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.971 DEBUG:boto:Params: {} 2022-09-13T07:35:13.972 DEBUG:boto:Token: None 2022-09-13T07:35:13.972 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.973 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Z0ULyuT9TkxObnK/Nevlud2c8VU= 2022-09-13T07:35:13.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Z0ULyuT9TkxObnK/Nevlud2c8VU=', 'Content-Length': '0'} 2022-09-13T07:35:13.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080a01dde1d938ba6-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.977 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:13.977 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:13.978 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.978 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.979 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.979 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.979 DEBUG:boto:Data: 2022-09-13T07:35:13.980 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.980 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:13.981 DEBUG:boto:Port: 8001 2022-09-13T07:35:13.981 DEBUG:boto:Params: {} 2022-09-13T07:35:13.982 DEBUG:boto:Token: None 2022-09-13T07:35:13.982 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.983 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:k3MLTxILJS67dmT0EoMn3l5iWZ4= 2022-09-13T07:35:13.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:k3MLTxILJS67dmT0EoMn3l5iWZ4=', 'Content-Length': '0'} 2022-09-13T07:35:13.986 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000496dbf5662a3e46b-00632032b1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.986 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:13.987 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:13.987 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.988 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.989 DEBUG:boto:Method: HEAD 2022-09-13T07:35:13.989 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.989 DEBUG:boto:Data: 2022-09-13T07:35:13.990 DEBUG:boto:Headers: {} 2022-09-13T07:35:13.990 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:13.991 DEBUG:boto:Port: 8000 2022-09-13T07:35:13.991 DEBUG:boto:Params: {} 2022-09-13T07:35:13.992 DEBUG:boto:Token: None 2022-09-13T07:35:13.992 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:13 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.993 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:k3MLTxILJS67dmT0EoMn3l5iWZ4= 2022-09-13T07:35:13.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:13 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:k3MLTxILJS67dmT0EoMn3l5iWZ4=', 'Content-Length': '0'} 2022-09-13T07:35:13.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be91b2ff24b94d05-00632032b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:13.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:13.997 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:13.997 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:13.998 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.999 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:13.999 DEBUG:boto:Method: GET 2022-09-13T07:35:14.000 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:14.000 DEBUG:boto:Data: 2022-09-13T07:35:14.000 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.001 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.002 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.002 DEBUG:boto:Params: {} 2022-09-13T07:35:14.003 DEBUG:boto:Token: None 2022-09-13T07:35:14.003 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:14.004 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/TKTnY7vAXEqao+AAGPScdAi6j4= 2022-09-13T07:35:14.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/TKTnY7vAXEqao+AAGPScdAi6j4=', 'Content-Length': '0'} 2022-09-13T07:35:14.007 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f40acba791596160-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.008 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.008 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.009 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:14.009 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:14.010 DEBUG:boto:Method: GET 2022-09-13T07:35:14.011 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:14.011 DEBUG:boto:Data: 2022-09-13T07:35:14.012 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.012 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.013 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.013 DEBUG:boto:Params: {} 2022-09-13T07:35:14.014 DEBUG:boto:Token: None 2022-09-13T07:35:14.014 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:14.015 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/TKTnY7vAXEqao+AAGPScdAi6j4= 2022-09-13T07:35:14.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/TKTnY7vAXEqao+AAGPScdAi6j4=', 'Content-Length': '0'} 2022-09-13T07:35:14.018 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eaf4fa28c6d6cc58-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.019 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:14.019 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:14.020 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:14.020 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:14.021 DEBUG:boto:Method: GET 2022-09-13T07:35:14.021 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:14.022 DEBUG:boto:Data: 2022-09-13T07:35:14.022 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.023 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.023 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.024 DEBUG:boto:Params: {} 2022-09-13T07:35:14.024 DEBUG:boto:Token: None 2022-09-13T07:35:14.025 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/ 2022-09-13T07:35:14.025 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0= 2022-09-13T07:35:14.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0=', 'Content-Length': '0'} 2022-09-13T07:35:14.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078c61917481c0b6b-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.029 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.030 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:14.030 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:14.031 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:14.032 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:14.032 DEBUG:boto:Method: GET 2022-09-13T07:35:14.033 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:14.033 DEBUG:boto:Data: 2022-09-13T07:35:14.034 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.034 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.035 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.035 DEBUG:boto:Params: {} 2022-09-13T07:35:14.036 DEBUG:boto:Token: None 2022-09-13T07:35:14.036 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/ 2022-09-13T07:35:14.037 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0= 2022-09-13T07:35:14.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0=', 'Content-Length': '0'} 2022-09-13T07:35:14.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c247d2396959473e-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.041 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.042 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.042 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.043 DEBUG:boto:Method: HEAD 2022-09-13T07:35:14.043 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.044 DEBUG:boto:Data: 2022-09-13T07:35:14.044 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.045 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.045 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.046 DEBUG:boto:Params: {} 2022-09-13T07:35:14.046 DEBUG:boto:Token: None 2022-09-13T07:35:14.047 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.048 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qXZIDHWOv4iYErksdfL2kTUYpfE= 2022-09-13T07:35:14.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qXZIDHWOv4iYErksdfL2kTUYpfE=', 'Content-Length': '0'} 2022-09-13T07:35:14.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f034ee0c852a38f-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.051 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.052 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.052 DEBUG:boto:Method: HEAD 2022-09-13T07:35:14.053 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.053 DEBUG:boto:Data: 2022-09-13T07:35:14.054 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.054 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.055 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.055 DEBUG:boto:Params: {} 2022-09-13T07:35:14.056 DEBUG:boto:Token: None 2022-09-13T07:35:14.057 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.057 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qXZIDHWOv4iYErksdfL2kTUYpfE= 2022-09-13T07:35:14.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qXZIDHWOv4iYErksdfL2kTUYpfE=', 'Content-Length': '0'} 2022-09-13T07:35:14.060 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000187b795cc2cc06a5-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:14.061 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.062 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.063 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:14.064 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:14.064 DEBUG:boto:Method: GET 2022-09-13T07:35:14.065 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:14.065 DEBUG:boto:Data: 2022-09-13T07:35:14.066 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.066 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.067 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.067 DEBUG:boto:Params: {} 2022-09-13T07:35:14.068 DEBUG:boto:Token: None 2022-09-13T07:35:14.069 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:14.069 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/TKTnY7vAXEqao+AAGPScdAi6j4= 2022-09-13T07:35:14.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/TKTnY7vAXEqao+AAGPScdAi6j4=', 'Content-Length': '0'} 2022-09-13T07:35:14.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000300e61a74021a582-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.114 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.115 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.116 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:14.116 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:14.117 DEBUG:boto:Method: GET 2022-09-13T07:35:14.117 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:14.118 DEBUG:boto:Data: 2022-09-13T07:35:14.118 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.119 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.120 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.121 DEBUG:boto:Params: {} 2022-09-13T07:35:14.121 DEBUG:boto:Token: None 2022-09-13T07:35:14.122 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:35:14.123 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/TKTnY7vAXEqao+AAGPScdAi6j4= 2022-09-13T07:35:14.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/TKTnY7vAXEqao+AAGPScdAi6j4=', 'Content-Length': '0'} 2022-09-13T07:35:14.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b2f09cc4d9c8616-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.127 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:14.127 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:14.128 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:14.128 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:14.129 DEBUG:boto:Method: GET 2022-09-13T07:35:14.129 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:14.130 DEBUG:boto:Data: 2022-09-13T07:35:14.131 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.131 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.132 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.132 DEBUG:boto:Params: {} 2022-09-13T07:35:14.133 DEBUG:boto:Token: None 2022-09-13T07:35:14.134 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/ 2022-09-13T07:35:14.134 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0= 2022-09-13T07:35:14.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0=', 'Content-Length': '0'} 2022-09-13T07:35:14.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088d0c2905fea8726-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.138 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.139 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:14.140 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:14.140 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:14.141 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:14.141 DEBUG:boto:Method: GET 2022-09-13T07:35:14.142 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:35:14.143 DEBUG:boto:Data: 2022-09-13T07:35:14.143 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.144 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.144 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.145 DEBUG:boto:Params: {} 2022-09-13T07:35:14.145 DEBUG:boto:Token: None 2022-09-13T07:35:14.146 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/ 2022-09-13T07:35:14.147 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0= 2022-09-13T07:35:14.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0=', 'Content-Length': '0'} 2022-09-13T07:35:14.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e3ebe40d97fa90e-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.150 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:14.152 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.153 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.153 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.154 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.154 DEBUG:boto:Method: GET 2022-09-13T07:35:14.155 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.155 DEBUG:boto:Data: 2022-09-13T07:35:14.156 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.156 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.157 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.157 DEBUG:boto:Params: {} 2022-09-13T07:35:14.158 DEBUG:boto:Token: None 2022-09-13T07:35:14.159 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.159 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:goJccBKLpnUUzaK4+3jOTS7Gu94= 2022-09-13T07:35:14.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:goJccBKLpnUUzaK4+3jOTS7Gu94=', 'Content-Length': '0'} 2022-09-13T07:35:14.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032288c053f6000d6-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.163 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.164 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.164 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.165 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.165 DEBUG:boto:Method: GET 2022-09-13T07:35:14.166 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.166 DEBUG:boto:Data: 2022-09-13T07:35:14.167 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.167 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.168 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.168 DEBUG:boto:Params: {} 2022-09-13T07:35:14.169 DEBUG:boto:Token: None 2022-09-13T07:35:14.170 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.170 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:goJccBKLpnUUzaK4+3jOTS7Gu94= 2022-09-13T07:35:14.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:goJccBKLpnUUzaK4+3jOTS7Gu94=', 'Content-Length': '0'} 2022-09-13T07:35:14.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbe49596c8a25091-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.214 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.215 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.215 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.216 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.216 DEBUG:boto:Method: HEAD 2022-09-13T07:35:14.217 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.217 DEBUG:boto:Data: 2022-09-13T07:35:14.218 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.218 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.219 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.219 DEBUG:boto:Params: {} 2022-09-13T07:35:14.220 DEBUG:boto:Token: None 2022-09-13T07:35:14.220 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.221 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0nll7YEmJKMtLTsARti9QxzI7yA= 2022-09-13T07:35:14.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0nll7YEmJKMtLTsARti9QxzI7yA=', 'Content-Length': '0'} 2022-09-13T07:35:14.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b4f467775fbefe4-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.225 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.226 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.226 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.227 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.227 DEBUG:boto:Method: HEAD 2022-09-13T07:35:14.228 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.228 DEBUG:boto:Data: 2022-09-13T07:35:14.229 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.229 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.230 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.230 DEBUG:boto:Params: {} 2022-09-13T07:35:14.231 DEBUG:boto:Token: None 2022-09-13T07:35:14.231 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.232 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0nll7YEmJKMtLTsARti9QxzI7yA= 2022-09-13T07:35:14.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0nll7YEmJKMtLTsARti9QxzI7yA=', 'Content-Length': '0'} 2022-09-13T07:35:14.235 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e72741e7e82e5d9-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:14.236 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.237 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.237 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.238 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.238 DEBUG:boto:Method: GET 2022-09-13T07:35:14.239 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.239 DEBUG:boto:Data: 2022-09-13T07:35:14.240 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.240 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.241 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.242 DEBUG:boto:Params: {} 2022-09-13T07:35:14.242 DEBUG:boto:Token: None 2022-09-13T07:35:14.243 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.243 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:goJccBKLpnUUzaK4+3jOTS7Gu94= 2022-09-13T07:35:14.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:goJccBKLpnUUzaK4+3jOTS7Gu94=', 'Content-Length': '0'} 2022-09-13T07:35:14.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074302230376a72f1-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.247 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.247 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.248 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.248 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.249 DEBUG:boto:Method: GET 2022-09-13T07:35:14.249 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.250 DEBUG:boto:Data: 2022-09-13T07:35:14.250 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.251 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.251 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.252 DEBUG:boto:Params: {} 2022-09-13T07:35:14.253 DEBUG:boto:Token: None 2022-09-13T07:35:14.253 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.254 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:goJccBKLpnUUzaK4+3jOTS7Gu94= 2022-09-13T07:35:14.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:goJccBKLpnUUzaK4+3jOTS7Gu94=', 'Content-Length': '0'} 2022-09-13T07:35:14.257 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007894046299096fe5-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.258 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:14.259 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:14.260 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:14.260 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:14.261 DEBUG:boto:Method: GET 2022-09-13T07:35:14.261 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:14.262 DEBUG:boto:Data: 2022-09-13T07:35:14.262 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.263 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.263 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.264 DEBUG:boto:Params: {} 2022-09-13T07:35:14.264 DEBUG:boto:Token: None 2022-09-13T07:35:14.265 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/ 2022-09-13T07:35:14.266 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0= 2022-09-13T07:35:14.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0=', 'Content-Length': '0'} 2022-09-13T07:35:14.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002249d92bb37aaf8a-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.270 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.271 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:14.271 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:14.272 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:14.272 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:14.273 DEBUG:boto:Method: GET 2022-09-13T07:35:14.273 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:14.274 DEBUG:boto:Data: 2022-09-13T07:35:14.274 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.275 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.275 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.276 DEBUG:boto:Params: {} 2022-09-13T07:35:14.276 DEBUG:boto:Token: None 2022-09-13T07:35:14.277 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/ 2022-09-13T07:35:14.278 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0= 2022-09-13T07:35:14.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0=', 'Content-Length': '0'} 2022-09-13T07:35:14.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c447b769ef999c76-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.282 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.282 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.283 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.283 DEBUG:boto:Method: HEAD 2022-09-13T07:35:14.284 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.284 DEBUG:boto:Data: 2022-09-13T07:35:14.285 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.285 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.286 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.286 DEBUG:boto:Params: {} 2022-09-13T07:35:14.287 DEBUG:boto:Token: None 2022-09-13T07:35:14.288 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.288 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OOMjJ8PGGzo2JLo3h0ge3a/e0lA= 2022-09-13T07:35:14.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OOMjJ8PGGzo2JLo3h0ge3a/e0lA=', 'Content-Length': '0'} 2022-09-13T07:35:14.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000816d9ff4028c68d5-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.292 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.293 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.293 DEBUG:boto:Method: HEAD 2022-09-13T07:35:14.294 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.295 DEBUG:boto:Data: 2022-09-13T07:35:14.295 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.296 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.296 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.297 DEBUG:boto:Params: {} 2022-09-13T07:35:14.297 DEBUG:boto:Token: None 2022-09-13T07:35:14.298 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.298 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OOMjJ8PGGzo2JLo3h0ge3a/e0lA= 2022-09-13T07:35:14.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OOMjJ8PGGzo2JLo3h0ge3a/e0lA=', 'Content-Length': '0'} 2022-09-13T07:35:14.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d602dca3d3bf1f6a-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:14.302 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.303 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.304 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.304 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.305 DEBUG:boto:Method: GET 2022-09-13T07:35:14.305 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.306 DEBUG:boto:Data: 2022-09-13T07:35:14.306 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.307 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.307 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.308 DEBUG:boto:Params: {} 2022-09-13T07:35:14.308 DEBUG:boto:Token: None 2022-09-13T07:35:14.309 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.310 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:goJccBKLpnUUzaK4+3jOTS7Gu94= 2022-09-13T07:35:14.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:goJccBKLpnUUzaK4+3jOTS7Gu94=', 'Content-Length': '0'} 2022-09-13T07:35:14.312 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000303007c381dbfdbc-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.313 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.314 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.314 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.315 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.315 DEBUG:boto:Method: GET 2022-09-13T07:35:14.316 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.316 DEBUG:boto:Data: 2022-09-13T07:35:14.317 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.317 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.318 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.318 DEBUG:boto:Params: {} 2022-09-13T07:35:14.319 DEBUG:boto:Token: None 2022-09-13T07:35:14.320 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:35:14.321 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:goJccBKLpnUUzaK4+3jOTS7Gu94= 2022-09-13T07:35:14.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:goJccBKLpnUUzaK4+3jOTS7Gu94=', 'Content-Length': '0'} 2022-09-13T07:35:14.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:32:07 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4f2297a993c634a-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.324 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:14.325 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:14.325 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:14.326 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:14.326 DEBUG:boto:Method: GET 2022-09-13T07:35:14.327 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:14.327 DEBUG:boto:Data: 2022-09-13T07:35:14.328 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.328 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.329 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.329 DEBUG:boto:Params: {} 2022-09-13T07:35:14.330 DEBUG:boto:Token: None 2022-09-13T07:35:14.331 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/ 2022-09-13T07:35:14.331 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0= 2022-09-13T07:35:14.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0=', 'Content-Length': '0'} 2022-09-13T07:35:14.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000715765157bfef537-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.335 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.336 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:14.336 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:14.337 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:14.337 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:14.338 DEBUG:boto:Method: GET 2022-09-13T07:35:14.339 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:35:14.339 DEBUG:boto:Data: 2022-09-13T07:35:14.340 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.340 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.341 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.341 DEBUG:boto:Params: {} 2022-09-13T07:35:14.342 DEBUG:boto:Token: None 2022-09-13T07:35:14.343 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/ 2022-09-13T07:35:14.343 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0= 2022-09-13T07:35:14.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Xwo5seEp+fl7EhI6GTKHZm+27x0=', 'Content-Length': '0'} 2022-09-13T07:35:14.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaf796e89dcd5543-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.347 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.348 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-18 zones={test-zone3, test-zone2} 2022-09-13T07:35:14.349 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:14.349 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:14.350 DEBUG:boto:Method: HEAD 2022-09-13T07:35:14.350 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:35:14.351 DEBUG:boto:Data: 2022-09-13T07:35:14.351 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.352 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:14.352 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.353 DEBUG:boto:Params: {} 2022-09-13T07:35:14.353 DEBUG:boto:Token: None 2022-09-13T07:35:14.354 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-16/ 2022-09-13T07:35:14.354 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bst+Gtv0svazMPA3yrWdJk0x6QU= 2022-09-13T07:35:14.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bst+Gtv0svazMPA3yrWdJk0x6QU=', 'Content-Length': '0'} 2022-09-13T07:35:14.358 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c01a79dd1409ba3-00632032b2-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.359 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:14.359 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:14.360 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-13T07:35:14.360 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-13T07:35:14.361 DEBUG:boto:Method: GET 2022-09-13T07:35:14.361 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-13T07:35:14.362 DEBUG:boto:Data: 2022-09-13T07:35:14.363 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.363 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:14.364 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.364 DEBUG:boto:Params: {} 2022-09-13T07:35:14.365 DEBUG:boto:Token: None 2022-09-13T07:35:14.365 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-16/?versions 2022-09-13T07:35:14.366 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5zSmJPpEr0d8XxldZ+2LN0B29sg= 2022-09-13T07:35:14.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5zSmJPpEr0d8XxldZ+2LN0B29sg=', 'Content-Length': '0'} 2022-09-13T07:35:14.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022eb96eca6554cfa-00632032b2-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.371 DEBUG:boto:b'yxvvuh-16obj-test-zone1999falseobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.372 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:14.372 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:14.373 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?acl 2022-09-13T07:35:14.373 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?acl 2022-09-13T07:35:14.374 DEBUG:boto:Method: PUT 2022-09-13T07:35:14.375 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?acl 2022-09-13T07:35:14.375 DEBUG:boto:Data: 2022-09-13T07:35:14.376 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-13T07:35:14.376 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:14.377 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.377 DEBUG:boto:Params: {} 2022-09-13T07:35:14.378 DEBUG:boto:Token: None 2022-09-13T07:35:14.378 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:35:14 GMT x-amz-acl:public-read /yxvvuh-16/obj-test-zone1?acl 2022-09-13T07:35:14.379 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7PSldqTrM/Ir5fjuKZGrclK+a+8= 2022-09-13T07:35:14.380 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7PSldqTrM/Ir5fjuKZGrclK+a+8=', 'Content-Length': '0'} 2022-09-13T07:35:14.391 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006682bc3108192a8e-00632032b2-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.392 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:14.392 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:14.393 DEBUG:boto:Method: HEAD 2022-09-13T07:35:14.393 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:35:14.394 DEBUG:boto:Data: 2022-09-13T07:35:14.394 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.395 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.395 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.396 DEBUG:boto:Params: {} 2022-09-13T07:35:14.396 DEBUG:boto:Token: None 2022-09-13T07:35:14.397 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-16/ 2022-09-13T07:35:14.397 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bst+Gtv0svazMPA3yrWdJk0x6QU= 2022-09-13T07:35:14.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bst+Gtv0svazMPA3yrWdJk0x6QU=', 'Content-Length': '0'} 2022-09-13T07:35:14.400 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9ae95fc233aae34-00632032b2-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.401 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:14.402 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:14.402 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-13T07:35:14.403 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-13T07:35:14.403 DEBUG:boto:Method: GET 2022-09-13T07:35:14.404 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-13T07:35:14.404 DEBUG:boto:Data: 2022-09-13T07:35:14.405 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.405 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.406 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.406 DEBUG:boto:Params: {} 2022-09-13T07:35:14.407 DEBUG:boto:Token: None 2022-09-13T07:35:14.407 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-16/?versions 2022-09-13T07:35:14.408 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5zSmJPpEr0d8XxldZ+2LN0B29sg= 2022-09-13T07:35:14.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5zSmJPpEr0d8XxldZ+2LN0B29sg=', 'Content-Length': '0'} 2022-09-13T07:35:14.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba026e6903367156-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.412 DEBUG:boto:b'yxvvuh-16obj-test-zone2999falseobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.413 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:14.413 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:14.414 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?acl 2022-09-13T07:35:14.414 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?acl 2022-09-13T07:35:14.415 DEBUG:boto:Method: PUT 2022-09-13T07:35:14.415 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?acl 2022-09-13T07:35:14.416 DEBUG:boto:Data: 2022-09-13T07:35:14.416 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-13T07:35:14.417 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.417 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.418 DEBUG:boto:Params: {} 2022-09-13T07:35:14.418 DEBUG:boto:Token: None 2022-09-13T07:35:14.419 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:35:14 GMT x-amz-acl:public-read /yxvvuh-16/obj-test-zone2?acl 2022-09-13T07:35:14.420 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cxmEY5NQKJwjd2jBQoMpQ3Yq/MI= 2022-09-13T07:35:14.420 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cxmEY5NQKJwjd2jBQoMpQ3Yq/MI=', 'Content-Length': '0'} 2022-09-13T07:35:14.427 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a6a249a9369d7df6-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.428 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:14.429 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:14.429 DEBUG:boto:Method: HEAD 2022-09-13T07:35:14.430 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:35:14.431 DEBUG:boto:Data: 2022-09-13T07:35:14.431 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.432 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.432 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.433 DEBUG:boto:Params: {} 2022-09-13T07:35:14.433 DEBUG:boto:Token: None 2022-09-13T07:35:14.434 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-16/ 2022-09-13T07:35:14.434 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bst+Gtv0svazMPA3yrWdJk0x6QU= 2022-09-13T07:35:14.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bst+Gtv0svazMPA3yrWdJk0x6QU=', 'Content-Length': '0'} 2022-09-13T07:35:14.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072e9e29ae6928feb-00632032b2-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.438 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:14.439 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:14.439 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-13T07:35:14.440 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-13T07:35:14.440 DEBUG:boto:Method: GET 2022-09-13T07:35:14.441 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-13T07:35:14.441 DEBUG:boto:Data: 2022-09-13T07:35:14.442 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.443 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.443 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.444 DEBUG:boto:Params: {} 2022-09-13T07:35:14.444 DEBUG:boto:Token: None 2022-09-13T07:35:14.445 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-16/?versions 2022-09-13T07:35:14.445 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5zSmJPpEr0d8XxldZ+2LN0B29sg= 2022-09-13T07:35:14.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5zSmJPpEr0d8XxldZ+2LN0B29sg=', 'Content-Length': '0'} 2022-09-13T07:35:14.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee47fbe5b6b90763-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.449 DEBUG:boto:b'yxvvuh-16obj-test-zone3999falseobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.450 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:14.451 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:14.451 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?acl 2022-09-13T07:35:14.452 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?acl 2022-09-13T07:35:14.452 DEBUG:boto:Method: PUT 2022-09-13T07:35:14.453 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?acl 2022-09-13T07:35:14.453 DEBUG:boto:Data: 2022-09-13T07:35:14.454 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-13T07:35:14.454 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.455 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.455 DEBUG:boto:Params: {} 2022-09-13T07:35:14.456 DEBUG:boto:Token: None 2022-09-13T07:35:14.457 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:35:14 GMT x-amz-acl:public-read /yxvvuh-16/obj-test-zone3?acl 2022-09-13T07:35:14.457 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:18b25pq/dRlIFfYTkGxB5Be+XSo= 2022-09-13T07:35:14.458 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:18b25pq/dRlIFfYTkGxB5Be+XSo=', 'Content-Length': '0'} 2022-09-13T07:35:14.467 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000009f9277ee13bb397-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.468 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:35:14.469 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:35:14.469 DEBUG:boto:Method: HEAD 2022-09-13T07:35:14.470 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:35:14.470 DEBUG:boto:Data: 2022-09-13T07:35:14.471 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.471 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:14.472 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.474 DEBUG:boto:Params: {} 2022-09-13T07:35:14.475 DEBUG:boto:Token: None 2022-09-13T07:35:14.476 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-17/ 2022-09-13T07:35:14.477 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1Xa7bSeVi2nhxOb26TqPKDM8cgY= 2022-09-13T07:35:14.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1Xa7bSeVi2nhxOb26TqPKDM8cgY=', 'Content-Length': '0'} 2022-09-13T07:35:14.481 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc6dead8f6f66051-00632032b2-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.481 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:35:14.482 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:35:14.483 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-13T07:35:14.483 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-13T07:35:14.484 DEBUG:boto:Method: GET 2022-09-13T07:35:14.484 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-13T07:35:14.485 DEBUG:boto:Data: 2022-09-13T07:35:14.485 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.485 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:14.486 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.486 DEBUG:boto:Params: {} 2022-09-13T07:35:14.487 DEBUG:boto:Token: None 2022-09-13T07:35:14.488 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-17/?versions 2022-09-13T07:35:14.488 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:P1zTvd+i9RznNYM1N5Ppmgunn0o= 2022-09-13T07:35:14.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:P1zTvd+i9RznNYM1N5Ppmgunn0o=', 'Content-Length': '0'} 2022-09-13T07:35:14.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa2cd15c401aacfd-00632032b2-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.492 DEBUG:boto:b'yxvvuh-17obj-test-zone1999falseobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.493 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:35:14.493 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:35:14.494 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?acl 2022-09-13T07:35:14.494 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?acl 2022-09-13T07:35:14.495 DEBUG:boto:Method: PUT 2022-09-13T07:35:14.495 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?acl 2022-09-13T07:35:14.496 DEBUG:boto:Data: 2022-09-13T07:35:14.496 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-13T07:35:14.497 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:14.497 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.497 DEBUG:boto:Params: {} 2022-09-13T07:35:14.498 DEBUG:boto:Token: None 2022-09-13T07:35:14.499 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:35:14 GMT x-amz-acl:public-read /yxvvuh-17/obj-test-zone1?acl 2022-09-13T07:35:14.499 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hZIp9rfSCpUA+OBjCohDAX/gmDo= 2022-09-13T07:35:14.500 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hZIp9rfSCpUA+OBjCohDAX/gmDo=', 'Content-Length': '0'} 2022-09-13T07:35:14.507 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d9068c3795a85da4-00632032b2-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.508 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:35:14.508 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:35:14.509 DEBUG:boto:Method: HEAD 2022-09-13T07:35:14.509 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:35:14.510 DEBUG:boto:Data: 2022-09-13T07:35:14.510 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.511 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.511 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.512 DEBUG:boto:Params: {} 2022-09-13T07:35:14.512 DEBUG:boto:Token: None 2022-09-13T07:35:14.513 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-17/ 2022-09-13T07:35:14.513 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1Xa7bSeVi2nhxOb26TqPKDM8cgY= 2022-09-13T07:35:14.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1Xa7bSeVi2nhxOb26TqPKDM8cgY=', 'Content-Length': '0'} 2022-09-13T07:35:14.517 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be77d0e85e8c2d44-00632032b2-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.518 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:35:14.519 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:35:14.520 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-13T07:35:14.520 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-13T07:35:14.521 DEBUG:boto:Method: GET 2022-09-13T07:35:14.522 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-13T07:35:14.523 DEBUG:boto:Data: 2022-09-13T07:35:14.524 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.524 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.525 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.526 DEBUG:boto:Params: {} 2022-09-13T07:35:14.527 DEBUG:boto:Token: None 2022-09-13T07:35:14.528 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-17/?versions 2022-09-13T07:35:14.529 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:P1zTvd+i9RznNYM1N5Ppmgunn0o= 2022-09-13T07:35:14.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:P1zTvd+i9RznNYM1N5Ppmgunn0o=', 'Content-Length': '0'} 2022-09-13T07:35:14.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003faa51440ed24fe7-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.533 DEBUG:boto:b'yxvvuh-17obj-test-zone2999falseobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.534 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:35:14.535 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:35:14.535 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?acl 2022-09-13T07:35:14.536 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?acl 2022-09-13T07:35:14.536 DEBUG:boto:Method: PUT 2022-09-13T07:35:14.537 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?acl 2022-09-13T07:35:14.537 DEBUG:boto:Data: 2022-09-13T07:35:14.538 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-13T07:35:14.538 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.538 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.539 DEBUG:boto:Params: {} 2022-09-13T07:35:14.539 DEBUG:boto:Token: None 2022-09-13T07:35:14.540 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:35:14 GMT x-amz-acl:public-read /yxvvuh-17/obj-test-zone2?acl 2022-09-13T07:35:14.541 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yrMS0UH5mHrgqVkOkXHIFGpm/m0= 2022-09-13T07:35:14.541 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yrMS0UH5mHrgqVkOkXHIFGpm/m0=', 'Content-Length': '0'} 2022-09-13T07:35:14.549 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f66f43eb2beef37d-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.549 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:35:14.550 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:35:14.550 DEBUG:boto:Method: HEAD 2022-09-13T07:35:14.551 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:35:14.552 DEBUG:boto:Data: 2022-09-13T07:35:14.552 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.553 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.554 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.554 DEBUG:boto:Params: {} 2022-09-13T07:35:14.555 DEBUG:boto:Token: None 2022-09-13T07:35:14.555 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-17/ 2022-09-13T07:35:14.556 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1Xa7bSeVi2nhxOb26TqPKDM8cgY= 2022-09-13T07:35:14.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1Xa7bSeVi2nhxOb26TqPKDM8cgY=', 'Content-Length': '0'} 2022-09-13T07:35:14.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a167d3c7c96500ea-00632032b2-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.560 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:35:14.561 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:35:14.562 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-13T07:35:14.562 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-13T07:35:14.563 DEBUG:boto:Method: GET 2022-09-13T07:35:14.563 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-13T07:35:14.564 DEBUG:boto:Data: 2022-09-13T07:35:14.564 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.565 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.565 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.566 DEBUG:boto:Params: {} 2022-09-13T07:35:14.566 DEBUG:boto:Token: None 2022-09-13T07:35:14.567 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-17/?versions 2022-09-13T07:35:14.567 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:P1zTvd+i9RznNYM1N5Ppmgunn0o= 2022-09-13T07:35:14.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:P1zTvd+i9RznNYM1N5Ppmgunn0o=', 'Content-Length': '0'} 2022-09-13T07:35:14.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dbb0ebe309dcca7-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.614 DEBUG:boto:b'yxvvuh-17obj-test-zone3999falseobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.616 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:35:14.617 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:35:14.617 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?acl 2022-09-13T07:35:14.618 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?acl 2022-09-13T07:35:14.619 DEBUG:boto:Method: PUT 2022-09-13T07:35:14.620 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?acl 2022-09-13T07:35:14.621 DEBUG:boto:Data: 2022-09-13T07:35:14.622 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-13T07:35:14.623 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.624 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.624 DEBUG:boto:Params: {} 2022-09-13T07:35:14.625 DEBUG:boto:Token: None 2022-09-13T07:35:14.626 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:35:14 GMT x-amz-acl:public-read /yxvvuh-17/obj-test-zone3?acl 2022-09-13T07:35:14.627 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CKeK7C/9DtxghlKHqvaMnmctad4= 2022-09-13T07:35:14.628 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CKeK7C/9DtxghlKHqvaMnmctad4=', 'Content-Length': '0'} 2022-09-13T07:35:14.636 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000add598c70ceebe05-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.637 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:14.637 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:14.638 DEBUG:boto:Method: HEAD 2022-09-13T07:35:14.638 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:35:14.639 DEBUG:boto:Data: 2022-09-13T07:35:14.639 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.640 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:14.640 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.641 DEBUG:boto:Params: {} 2022-09-13T07:35:14.641 DEBUG:boto:Token: None 2022-09-13T07:35:14.642 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/ 2022-09-13T07:35:14.642 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MxsB/adXxd6Ti5YWmQBY9kapNeo= 2022-09-13T07:35:14.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MxsB/adXxd6Ti5YWmQBY9kapNeo=', 'Content-Length': '0'} 2022-09-13T07:35:14.645 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa667c5553438432-00632032b2-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.646 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:14.647 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:14.647 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-13T07:35:14.648 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-13T07:35:14.648 DEBUG:boto:Method: GET 2022-09-13T07:35:14.649 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-13T07:35:14.649 DEBUG:boto:Data: 2022-09-13T07:35:14.650 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.650 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:14.651 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.651 DEBUG:boto:Params: {} 2022-09-13T07:35:14.652 DEBUG:boto:Token: None 2022-09-13T07:35:14.652 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/?versions 2022-09-13T07:35:14.653 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gv2gYurpMop8LowxR/r2FQd1cwA= 2022-09-13T07:35:14.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gv2gYurpMop8LowxR/r2FQd1cwA=', 'Content-Length': '0'} 2022-09-13T07:35:14.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a8a1d5758f09eaa-00632032b2-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.657 DEBUG:boto:b'yxvvuh-18obj-test-zone1999falseobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.659 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:14.660 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:35:14.661 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?acl 2022-09-13T07:35:14.661 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?acl 2022-09-13T07:35:14.662 DEBUG:boto:Method: PUT 2022-09-13T07:35:14.663 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?acl 2022-09-13T07:35:14.664 DEBUG:boto:Data: 2022-09-13T07:35:14.665 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-13T07:35:14.666 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:14.667 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.667 DEBUG:boto:Params: {} 2022-09-13T07:35:14.668 DEBUG:boto:Token: None 2022-09-13T07:35:14.669 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:35:14 GMT x-amz-acl:public-read /yxvvuh-18/obj-test-zone1?acl 2022-09-13T07:35:14.669 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gUMxjndX082B2npGr4kocrSv4C4= 2022-09-13T07:35:14.670 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gUMxjndX082B2npGr4kocrSv4C4=', 'Content-Length': '0'} 2022-09-13T07:35:14.678 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000075fe44afb8a7ac26-00632032b2-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.679 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:14.680 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:14.681 DEBUG:boto:Method: HEAD 2022-09-13T07:35:14.682 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:35:14.682 DEBUG:boto:Data: 2022-09-13T07:35:14.683 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.684 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.685 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.685 DEBUG:boto:Params: {} 2022-09-13T07:35:14.686 DEBUG:boto:Token: None 2022-09-13T07:35:14.687 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/ 2022-09-13T07:35:14.688 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MxsB/adXxd6Ti5YWmQBY9kapNeo= 2022-09-13T07:35:14.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MxsB/adXxd6Ti5YWmQBY9kapNeo=', 'Content-Length': '0'} 2022-09-13T07:35:14.692 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3a5a7ddb2ccec3a-00632032b2-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.693 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:14.694 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:14.695 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-13T07:35:14.695 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-13T07:35:14.696 DEBUG:boto:Method: GET 2022-09-13T07:35:14.697 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-13T07:35:14.698 DEBUG:boto:Data: 2022-09-13T07:35:14.699 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.699 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.700 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.701 DEBUG:boto:Params: {} 2022-09-13T07:35:14.702 DEBUG:boto:Token: None 2022-09-13T07:35:14.703 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/?versions 2022-09-13T07:35:14.704 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gv2gYurpMop8LowxR/r2FQd1cwA= 2022-09-13T07:35:14.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gv2gYurpMop8LowxR/r2FQd1cwA=', 'Content-Length': '0'} 2022-09-13T07:35:14.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002d12f775bc1e1ef-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.709 DEBUG:boto:b'yxvvuh-18obj-test-zone2999falseobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.710 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.711 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:35:14.711 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?acl 2022-09-13T07:35:14.712 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?acl 2022-09-13T07:35:14.712 DEBUG:boto:Method: PUT 2022-09-13T07:35:14.713 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?acl 2022-09-13T07:35:14.713 DEBUG:boto:Data: 2022-09-13T07:35:14.713 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-13T07:35:14.714 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:14.714 DEBUG:boto:Port: 8000 2022-09-13T07:35:14.715 DEBUG:boto:Params: {} 2022-09-13T07:35:14.715 DEBUG:boto:Token: None 2022-09-13T07:35:14.716 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:35:14 GMT x-amz-acl:public-read /yxvvuh-18/obj-test-zone2?acl 2022-09-13T07:35:14.716 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9yvqLuJ3zamxi131y0V7OC+0In0= 2022-09-13T07:35:14.717 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9yvqLuJ3zamxi131y0V7OC+0In0=', 'Content-Length': '0'} 2022-09-13T07:35:14.724 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000731b418db86e5b5c-00632032b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.725 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:14.726 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:14.726 DEBUG:boto:Method: HEAD 2022-09-13T07:35:14.727 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:35:14.727 DEBUG:boto:Data: 2022-09-13T07:35:14.728 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.729 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.729 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.730 DEBUG:boto:Params: {} 2022-09-13T07:35:14.731 DEBUG:boto:Token: None 2022-09-13T07:35:14.732 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/ 2022-09-13T07:35:14.733 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MxsB/adXxd6Ti5YWmQBY9kapNeo= 2022-09-13T07:35:14.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MxsB/adXxd6Ti5YWmQBY9kapNeo=', 'Content-Length': '0'} 2022-09-13T07:35:14.737 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005129ad9e49e5a52a-00632032b2-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.738 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:35:14.739 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:35:14.740 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-13T07:35:14.741 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-13T07:35:14.741 DEBUG:boto:Method: GET 2022-09-13T07:35:14.742 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-13T07:35:14.743 DEBUG:boto:Data: 2022-09-13T07:35:14.744 DEBUG:boto:Headers: {} 2022-09-13T07:35:14.745 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.745 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.746 DEBUG:boto:Params: {} 2022-09-13T07:35:14.747 DEBUG:boto:Token: None 2022-09-13T07:35:14.748 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:14 GMT /yxvvuh-18/?versions 2022-09-13T07:35:14.749 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gv2gYurpMop8LowxR/r2FQd1cwA= 2022-09-13T07:35:14.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gv2gYurpMop8LowxR/r2FQd1cwA=', 'Content-Length': '0'} 2022-09-13T07:35:14.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000880ff4e1feeec66f-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.754 DEBUG:boto:b'yxvvuh-18obj-test-zone3999falseobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:14.755 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.756 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:35:14.757 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?acl 2022-09-13T07:35:14.757 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?acl 2022-09-13T07:35:14.758 DEBUG:boto:Method: PUT 2022-09-13T07:35:14.759 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?acl 2022-09-13T07:35:14.760 DEBUG:boto:Data: 2022-09-13T07:35:14.761 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-13T07:35:14.761 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:14.762 DEBUG:boto:Port: 8001 2022-09-13T07:35:14.763 DEBUG:boto:Params: {} 2022-09-13T07:35:14.764 DEBUG:boto:Token: None 2022-09-13T07:35:14.765 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:35:14 GMT x-amz-acl:public-read /yxvvuh-18/obj-test-zone3?acl 2022-09-13T07:35:14.766 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5qjyxNg1Yl4VZrB4JNE6/4lCfo4= 2022-09-13T07:35:14.767 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:14 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5qjyxNg1Yl4VZrB4JNE6/4lCfo4=', 'Content-Length': '0'} 2022-09-13T07:35:14.776 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000df605f4ce2a368df-00632032b2-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:14.777 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-16 2022-09-13T07:35:14.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:14.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:14.779 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:35:14.843 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:35:14.843 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:35:14.880 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:35:14.874+0000 7f0fc5d02600 1 waiting for incremental sync to catch up: 2022-09-13T07:35:14.880 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000010.31.5, , 00000000003.12.9, , 00000000007.36.9, , , , , ] 2022-09-13T07:35:14.881 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000011.34.5, , 00000000003.12.9, , 00000000007.36.9, , , , , ] 2022-09-13T07:35:19.880 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:35:19.875+0000 7f0fc5d02600 1 waiting for incremental sync to catch up: 2022-09-13T07:35:19.880 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000010.31.5, , 00000000003.12.9, , 00000000007.36.9, , , , , ] 2022-09-13T07:35:19.881 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000011.34.5, , 00000000003.12.9, , 00000000007.36.9, , , , , ] 2022-09-13T07:35:24.881 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:35:24.877+0000 7f0fc5d02600 1 waiting for incremental sync to catch up: 2022-09-13T07:35:24.882 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000010.31.5, , 00000000003.12.9, , 00000000007.36.9, , , , , ] 2022-09-13T07:35:24.882 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000011.34.5, , 00000000003.12.9, , 00000000007.36.9, , , , , ] 2022-09-13T07:35:29.883 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:35:29.878+0000 7f0fc5d02600 1 bucket sync caught up with source: 2022-09-13T07:35:29.883 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000011.34.5, , 00000000004.15.5, , 00000000008.42.5, , , , , ] 2022-09-13T07:35:29.884 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000011.34.5, , 00000000003.12.9, , 00000000007.36.9, , , , , ] 2022-09-13T07:35:29.884 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:35:29.878+0000 7f0fc5d02600 0 bucket checkpoint complete 2022-09-13T07:35:29.898 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-16 2022-09-13T07:35:29.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:29.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:29.901 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:35:29.967 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:35:29.968 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:35:30.019 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:30.015+0000 7f8da6e27600 1 waiting for incremental sync to catch up: 2022-09-13T07:35:30.019 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000010.31.5, , 00000000003.12.9, , 00000000007.36.9, , , , , ] 2022-09-13T07:35:30.019 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000011.34.5, , 00000000004.15.5, , 00000000008.42.5, , , , , ] 2022-09-13T07:35:35.021 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:35.016+0000 7f8da6e27600 1 bucket sync caught up with source: 2022-09-13T07:35:35.021 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000011.34.5, , 00000000004.15.5, , 00000000008.42.5, , , , , ] 2022-09-13T07:35:35.022 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000011.34.5, , 00000000004.15.5, , 00000000008.42.5, , , , , ] 2022-09-13T07:35:35.022 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:35.016+0000 7f8da6e27600 0 bucket checkpoint complete 2022-09-13T07:35:35.044 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-16 2022-09-13T07:35:35.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:35.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:35.047 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:35:35.112 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:35:35.113 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:35:35.167 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:35.163+0000 7f93b0aab600 1 waiting for incremental sync to catch up: 2022-09-13T07:35:35.168 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.51.9, , 00000000011.38.5, , 00000000008.54.5, , , , , ] 2022-09-13T07:35:35.168 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.54.5, , 00000000011.38.5, , 00000000008.54.5, , , , , ] 2022-09-13T07:35:40.169 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:40.165+0000 7f93b0aab600 1 waiting for incremental sync to catch up: 2022-09-13T07:35:40.170 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.51.9, , 00000000011.38.5, , 00000000008.54.5, , , , , ] 2022-09-13T07:35:40.170 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.54.5, , 00000000011.38.5, , 00000000008.54.5, , , , , ] 2022-09-13T07:35:45.174 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:45.167+0000 7f93b0aab600 1 waiting for incremental sync to catch up: 2022-09-13T07:35:45.175 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000003.51.9, , 00000000011.38.5, , 00000000008.54.5, , , , , ] 2022-09-13T07:35:45.175 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.54.5, , 00000000011.38.5, , 00000000008.54.5, , , , , ] 2022-09-13T07:35:50.172 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:50.169+0000 7f93b0aab600 1 bucket sync caught up with source: 2022-09-13T07:35:50.172 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.54.5, , 00000000011.38.5, , 00000000008.54.5, , , , , ] 2022-09-13T07:35:50.173 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.54.5, , 00000000011.38.5, , 00000000008.54.5, , , , , ] 2022-09-13T07:35:50.173 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:50.169+0000 7f93b0aab600 0 bucket checkpoint complete 2022-09-13T07:35:50.196 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-16 2022-09-13T07:35:50.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:50.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:50.198 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:35:50.261 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:35:50.262 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:35:50.312 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:50.309+0000 7f2c59804600 1 bucket sync caught up with source: 2022-09-13T07:35:50.312 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.54.5, , 00000000011.38.5, , 00000000008.54.5, , , , , ] 2022-09-13T07:35:50.313 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.54.5, , 00000000011.38.5, , 00000000008.54.5, , , , , ] 2022-09-13T07:35:50.313 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:50.309+0000 7f2c59804600 0 bucket checkpoint complete 2022-09-13T07:35:50.326 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-16 2022-09-13T07:35:50.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:50.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:50.327 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:35:50.385 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:35:50.386 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:35:50.441 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:50.438+0000 7f317a09b600 1 bucket sync caught up with source: 2022-09-13T07:35:50.441 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.58.5, , 00000000004.22.5, , 00000000015.75.5, , , , , ] 2022-09-13T07:35:50.441 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.58.5, , 00000000004.22.5, , 00000000015.75.5, , , , , ] 2022-09-13T07:35:50.442 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:50.438+0000 7f317a09b600 0 bucket checkpoint complete 2022-09-13T07:35:50.461 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-16 2022-09-13T07:35:50.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:50.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:50.463 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:35:50.526 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:35:50.527 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:35:50.569 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:35:50.565+0000 7f6340334600 1 bucket sync caught up with source: 2022-09-13T07:35:50.569 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000004.58.5, , 00000000004.22.5, , 00000000015.75.5, , , , , ] 2022-09-13T07:35:50.569 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000004.58.5, , 00000000004.22.5, , 00000000015.75.5, , , , , ] 2022-09-13T07:35:50.570 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:35:50.565+0000 7f6340334600 0 bucket checkpoint complete 2022-09-13T07:35:50.583 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-16 zones={test-zone2, test-zone1} 2022-09-13T07:35:50.584 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:50.584 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:50.585 DEBUG:boto:Method: HEAD 2022-09-13T07:35:50.586 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:35:50.586 DEBUG:boto:Data: 2022-09-13T07:35:50.587 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.587 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.588 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.589 DEBUG:boto:Params: {} 2022-09-13T07:35:50.590 DEBUG:boto:Token: None 2022-09-13T07:35:50.591 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/ 2022-09-13T07:35:50.592 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eaIdvKcPwoqDDd8Al6Xdc0CEC64= 2022-09-13T07:35:50.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eaIdvKcPwoqDDd8Al6Xdc0CEC64=', 'Content-Length': '0'} 2022-09-13T07:35:50.595 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6260a4afe4718e3-00632032d6-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.596 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:50.597 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:50.597 DEBUG:boto:Method: HEAD 2022-09-13T07:35:50.598 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:35:50.598 DEBUG:boto:Data: 2022-09-13T07:35:50.599 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.599 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.600 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.600 DEBUG:boto:Params: {} 2022-09-13T07:35:50.601 DEBUG:boto:Token: None 2022-09-13T07:35:50.601 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/ 2022-09-13T07:35:50.602 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eaIdvKcPwoqDDd8Al6Xdc0CEC64= 2022-09-13T07:35:50.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eaIdvKcPwoqDDd8Al6Xdc0CEC64=', 'Content-Length': '0'} 2022-09-13T07:35:50.605 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074783905aff04d72-00632032d6-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.606 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:35:50.606 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:50.607 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:50.607 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:50.608 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:50.608 DEBUG:boto:Method: GET 2022-09-13T07:35:50.609 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:50.609 DEBUG:boto:Data: 2022-09-13T07:35:50.610 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.610 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.611 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.611 DEBUG:boto:Params: {} 2022-09-13T07:35:50.612 DEBUG:boto:Token: None 2022-09-13T07:35:50.613 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/?versions 2022-09-13T07:35:50.613 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jHbttX3nOXy11IFTn7xuwPRHLJc= 2022-09-13T07:35:50.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jHbttX3nOXy11IFTn7xuwPRHLJc=', 'Content-Length': '0'} 2022-09-13T07:35:50.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b9eb68e134cc335-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.617 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:50.618 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:35:50.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:35:50.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:35:50.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:35:50.620 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:35:50.621 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:50.621 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:50.622 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:50.622 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:50.623 DEBUG:boto:Method: GET 2022-09-13T07:35:50.623 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:50.624 DEBUG:boto:Data: 2022-09-13T07:35:50.624 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.625 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.625 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.626 DEBUG:boto:Params: {} 2022-09-13T07:35:50.627 DEBUG:boto:Token: None 2022-09-13T07:35:50.627 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/?versions 2022-09-13T07:35:50.628 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jHbttX3nOXy11IFTn7xuwPRHLJc= 2022-09-13T07:35:50.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jHbttX3nOXy11IFTn7xuwPRHLJc=', 'Content-Length': '0'} 2022-09-13T07:35:50.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b5a4795babbe812-00632032d6-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.631 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:50.632 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:35:50.633 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:35:50.633 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:35:50.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:35:50.634 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:50.635 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:50.635 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:50.636 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:50.637 DEBUG:boto:Method: GET 2022-09-13T07:35:50.637 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:50.637 DEBUG:boto:Data: 2022-09-13T07:35:50.638 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.638 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.639 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.639 DEBUG:boto:Params: {} 2022-09-13T07:35:50.640 DEBUG:boto:Token: None 2022-09-13T07:35:50.641 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/?versions 2022-09-13T07:35:50.641 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jHbttX3nOXy11IFTn7xuwPRHLJc= 2022-09-13T07:35:50.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jHbttX3nOXy11IFTn7xuwPRHLJc=', 'Content-Length': '0'} 2022-09-13T07:35:50.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007150ec26aa3db31a-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.645 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:50.646 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:50.646 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:50.647 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:50.647 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:50.648 DEBUG:boto:Method: GET 2022-09-13T07:35:50.648 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:50.649 DEBUG:boto:Data: 2022-09-13T07:35:50.649 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.650 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.650 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.651 DEBUG:boto:Params: {} 2022-09-13T07:35:50.651 DEBUG:boto:Token: None 2022-09-13T07:35:50.652 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/?versions 2022-09-13T07:35:50.652 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jHbttX3nOXy11IFTn7xuwPRHLJc= 2022-09-13T07:35:50.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jHbttX3nOXy11IFTn7xuwPRHLJc=', 'Content-Length': '0'} 2022-09-13T07:35:50.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054f9a161cdbf2a16-00632032d6-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.656 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:50.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:50.658 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:50.658 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:50.659 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.659 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.660 DEBUG:boto:Method: GET 2022-09-13T07:35:50.660 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.661 DEBUG:boto:Data: 2022-09-13T07:35:50.661 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.662 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.662 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.663 DEBUG:boto:Params: {} 2022-09-13T07:35:50.663 DEBUG:boto:Token: None 2022-09-13T07:35:50.664 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.664 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TkQGLfF61t3oQpVKtF/eKInBgcc= 2022-09-13T07:35:50.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TkQGLfF61t3oQpVKtF/eKInBgcc=', 'Content-Length': '0'} 2022-09-13T07:35:50.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000072af002733abbe26-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.668 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:50.668 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:50.669 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.669 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.670 DEBUG:boto:Method: GET 2022-09-13T07:35:50.670 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.671 DEBUG:boto:Data: 2022-09-13T07:35:50.671 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.672 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.672 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.673 DEBUG:boto:Params: {} 2022-09-13T07:35:50.673 DEBUG:boto:Token: None 2022-09-13T07:35:50.674 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.675 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TkQGLfF61t3oQpVKtF/eKInBgcc= 2022-09-13T07:35:50.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TkQGLfF61t3oQpVKtF/eKInBgcc=', 'Content-Length': '0'} 2022-09-13T07:35:50.677 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000248c31934de53646-00632032d6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.678 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:50.678 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:50.679 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.679 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.680 DEBUG:boto:Method: HEAD 2022-09-13T07:35:50.680 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.681 DEBUG:boto:Data: 2022-09-13T07:35:50.681 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.682 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.682 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.683 DEBUG:boto:Params: {} 2022-09-13T07:35:50.683 DEBUG:boto:Token: None 2022-09-13T07:35:50.684 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.685 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xlXAIshNHup4DVfaOTdDU7Oj30Q= 2022-09-13T07:35:50.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xlXAIshNHup4DVfaOTdDU7Oj30Q=', 'Content-Length': '0'} 2022-09-13T07:35:50.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca37c69a7ec3d812-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.688 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:50.688 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:50.689 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.689 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.690 DEBUG:boto:Method: HEAD 2022-09-13T07:35:50.690 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.691 DEBUG:boto:Data: 2022-09-13T07:35:50.691 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.692 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.692 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.693 DEBUG:boto:Params: {} 2022-09-13T07:35:50.693 DEBUG:boto:Token: None 2022-09-13T07:35:50.694 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.694 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xlXAIshNHup4DVfaOTdDU7Oj30Q= 2022-09-13T07:35:50.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xlXAIshNHup4DVfaOTdDU7Oj30Q=', 'Content-Length': '0'} 2022-09-13T07:35:50.697 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d12afe66769f642b-00632032d6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:50.698 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:50.698 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:50.699 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.700 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.700 DEBUG:boto:Method: GET 2022-09-13T07:35:50.701 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.701 DEBUG:boto:Data: 2022-09-13T07:35:50.702 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.702 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.703 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.703 DEBUG:boto:Params: {} 2022-09-13T07:35:50.704 DEBUG:boto:Token: None 2022-09-13T07:35:50.704 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.705 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TkQGLfF61t3oQpVKtF/eKInBgcc= 2022-09-13T07:35:50.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TkQGLfF61t3oQpVKtF/eKInBgcc=', 'Content-Length': '0'} 2022-09-13T07:35:50.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee42d551127de2f1-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.708 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:50.709 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:50.709 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.709 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.710 DEBUG:boto:Method: GET 2022-09-13T07:35:50.711 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.711 DEBUG:boto:Data: 2022-09-13T07:35:50.712 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.712 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.713 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.713 DEBUG:boto:Params: {} 2022-09-13T07:35:50.714 DEBUG:boto:Token: None 2022-09-13T07:35:50.714 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.715 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TkQGLfF61t3oQpVKtF/eKInBgcc= 2022-09-13T07:35:50.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TkQGLfF61t3oQpVKtF/eKInBgcc=', 'Content-Length': '0'} 2022-09-13T07:35:50.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009e76eca02978f5fb-00632032d6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.718 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:50.719 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:50.719 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:50.720 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:50.720 DEBUG:boto:Method: GET 2022-09-13T07:35:50.721 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:35:50.721 DEBUG:boto:Data: 2022-09-13T07:35:50.722 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.722 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.723 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.723 DEBUG:boto:Params: {} 2022-09-13T07:35:50.724 DEBUG:boto:Token: None 2022-09-13T07:35:50.724 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/ 2022-09-13T07:35:50.725 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iXw5Oita+XzwrpkntPi4Ini5AyM= 2022-09-13T07:35:50.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iXw5Oita+XzwrpkntPi4Ini5AyM=', 'Content-Length': '0'} 2022-09-13T07:35:50.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aab10bad7029e4ed-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.728 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:50.729 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:50.730 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:50.730 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:50.731 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:50.731 DEBUG:boto:Method: GET 2022-09-13T07:35:50.732 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:35:50.732 DEBUG:boto:Data: 2022-09-13T07:35:50.733 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.734 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.734 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.735 DEBUG:boto:Params: {} 2022-09-13T07:35:50.735 DEBUG:boto:Token: None 2022-09-13T07:35:50.736 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/ 2022-09-13T07:35:50.736 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iXw5Oita+XzwrpkntPi4Ini5AyM= 2022-09-13T07:35:50.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iXw5Oita+XzwrpkntPi4Ini5AyM=', 'Content-Length': '0'} 2022-09-13T07:35:50.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032abe81c1d78235d-00632032d6-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.740 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:50.740 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:50.741 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:50.741 DEBUG:boto:Method: HEAD 2022-09-13T07:35:50.742 DEBUG:boto:Path: /yxvvuh-16/dummy 2022-09-13T07:35:50.742 DEBUG:boto:Data: 2022-09-13T07:35:50.743 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.743 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.744 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.744 DEBUG:boto:Params: {} 2022-09-13T07:35:50.745 DEBUG:boto:Token: None 2022-09-13T07:35:50.746 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/dummy 2022-09-13T07:35:50.746 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ph3BZdnd7fpOjM3+kdfCY1Txsks= 2022-09-13T07:35:50.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ph3BZdnd7fpOjM3+kdfCY1Txsks=', 'Content-Length': '0'} 2022-09-13T07:35:50.749 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c0a821c5b9e529b1-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.750 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:50.750 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:50.751 DEBUG:boto:Method: HEAD 2022-09-13T07:35:50.751 DEBUG:boto:Path: /yxvvuh-16/dummy 2022-09-13T07:35:50.752 DEBUG:boto:Data: 2022-09-13T07:35:50.752 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.753 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.753 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.754 DEBUG:boto:Params: {} 2022-09-13T07:35:50.754 DEBUG:boto:Token: None 2022-09-13T07:35:50.755 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/dummy 2022-09-13T07:35:50.756 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ph3BZdnd7fpOjM3+kdfCY1Txsks= 2022-09-13T07:35:50.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ph3BZdnd7fpOjM3+kdfCY1Txsks=', 'Content-Length': '0'} 2022-09-13T07:35:50.758 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001df08f7140f0ebab-00632032d6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:50.759 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:50.760 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:50.760 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.761 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.761 DEBUG:boto:Method: GET 2022-09-13T07:35:50.762 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.762 DEBUG:boto:Data: 2022-09-13T07:35:50.763 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.763 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.764 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.764 DEBUG:boto:Params: {} 2022-09-13T07:35:50.765 DEBUG:boto:Token: None 2022-09-13T07:35:50.765 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.766 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TkQGLfF61t3oQpVKtF/eKInBgcc= 2022-09-13T07:35:50.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TkQGLfF61t3oQpVKtF/eKInBgcc=', 'Content-Length': '0'} 2022-09-13T07:35:50.769 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000569f2638fa9196ec-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.770 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:50.770 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:50.771 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.771 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.772 DEBUG:boto:Method: GET 2022-09-13T07:35:50.772 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.773 DEBUG:boto:Data: 2022-09-13T07:35:50.773 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.774 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.774 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.775 DEBUG:boto:Params: {} 2022-09-13T07:35:50.775 DEBUG:boto:Token: None 2022-09-13T07:35:50.776 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:50.776 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TkQGLfF61t3oQpVKtF/eKInBgcc= 2022-09-13T07:35:50.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TkQGLfF61t3oQpVKtF/eKInBgcc=', 'Content-Length': '0'} 2022-09-13T07:35:50.779 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cdca586087ba50f8-00632032d6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.780 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:50.780 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:50.781 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:50.781 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:50.782 DEBUG:boto:Method: GET 2022-09-13T07:35:50.782 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:35:50.783 DEBUG:boto:Data: 2022-09-13T07:35:50.783 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.784 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.784 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.785 DEBUG:boto:Params: {} 2022-09-13T07:35:50.785 DEBUG:boto:Token: None 2022-09-13T07:35:50.786 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/ 2022-09-13T07:35:50.786 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iXw5Oita+XzwrpkntPi4Ini5AyM= 2022-09-13T07:35:50.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iXw5Oita+XzwrpkntPi4Ini5AyM=', 'Content-Length': '0'} 2022-09-13T07:35:50.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000306674306b131c76-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.790 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:50.791 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:50.791 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:50.792 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:50.792 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:50.793 DEBUG:boto:Method: GET 2022-09-13T07:35:50.793 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:35:50.794 DEBUG:boto:Data: 2022-09-13T07:35:50.794 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.795 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.795 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.796 DEBUG:boto:Params: {} 2022-09-13T07:35:50.797 DEBUG:boto:Token: None 2022-09-13T07:35:50.797 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/ 2022-09-13T07:35:50.798 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iXw5Oita+XzwrpkntPi4Ini5AyM= 2022-09-13T07:35:50.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iXw5Oita+XzwrpkntPi4Ini5AyM=', 'Content-Length': '0'} 2022-09-13T07:35:50.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1c1e4f43a3861c3-00632032d6-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.803 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:50.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:50.805 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.805 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.806 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.806 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.807 DEBUG:boto:Method: GET 2022-09-13T07:35:50.807 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.808 DEBUG:boto:Data: 2022-09-13T07:35:50.808 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.809 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.810 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.810 DEBUG:boto:Params: {} 2022-09-13T07:35:50.811 DEBUG:boto:Token: None 2022-09-13T07:35:50.811 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.812 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MANRUNvECStqIRnF5xX1SOiHbAI= 2022-09-13T07:35:50.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MANRUNvECStqIRnF5xX1SOiHbAI=', 'Content-Length': '0'} 2022-09-13T07:35:50.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eecb0a2a6ce876dc-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.815 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.816 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.816 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.817 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.817 DEBUG:boto:Method: GET 2022-09-13T07:35:50.818 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.818 DEBUG:boto:Data: 2022-09-13T07:35:50.819 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.819 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.820 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.820 DEBUG:boto:Params: {} 2022-09-13T07:35:50.821 DEBUG:boto:Token: None 2022-09-13T07:35:50.822 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.823 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MANRUNvECStqIRnF5xX1SOiHbAI= 2022-09-13T07:35:50.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MANRUNvECStqIRnF5xX1SOiHbAI=', 'Content-Length': '0'} 2022-09-13T07:35:50.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e4a2e38d71a40d9-00632032d6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.866 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.867 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.868 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.869 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.870 DEBUG:boto:Method: HEAD 2022-09-13T07:35:50.871 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.872 DEBUG:boto:Data: 2022-09-13T07:35:50.872 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.873 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.874 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.875 DEBUG:boto:Params: {} 2022-09-13T07:35:50.876 DEBUG:boto:Token: None 2022-09-13T07:35:50.877 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.878 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Xsr1PJ02GjGgwIOjXhoUJZr3/+Q= 2022-09-13T07:35:50.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Xsr1PJ02GjGgwIOjXhoUJZr3/+Q=', 'Content-Length': '0'} 2022-09-13T07:35:50.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003def2a1fc9b15013-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.883 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.883 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.884 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.884 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.885 DEBUG:boto:Method: HEAD 2022-09-13T07:35:50.885 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.886 DEBUG:boto:Data: 2022-09-13T07:35:50.886 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.887 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.887 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.888 DEBUG:boto:Params: {} 2022-09-13T07:35:50.888 DEBUG:boto:Token: None 2022-09-13T07:35:50.889 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.890 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Xsr1PJ02GjGgwIOjXhoUJZr3/+Q= 2022-09-13T07:35:50.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Xsr1PJ02GjGgwIOjXhoUJZr3/+Q=', 'Content-Length': '0'} 2022-09-13T07:35:50.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005f66c23529172e1-00632032d6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:50.894 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.894 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.895 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.895 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.896 DEBUG:boto:Method: GET 2022-09-13T07:35:50.897 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.897 DEBUG:boto:Data: 2022-09-13T07:35:50.898 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.898 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.899 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.899 DEBUG:boto:Params: {} 2022-09-13T07:35:50.900 DEBUG:boto:Token: None 2022-09-13T07:35:50.901 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.901 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MANRUNvECStqIRnF5xX1SOiHbAI= 2022-09-13T07:35:50.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MANRUNvECStqIRnF5xX1SOiHbAI=', 'Content-Length': '0'} 2022-09-13T07:35:50.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042f32058c3d5ee59-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.905 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.905 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.906 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.906 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.907 DEBUG:boto:Method: GET 2022-09-13T07:35:50.907 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.908 DEBUG:boto:Data: 2022-09-13T07:35:50.909 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.909 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.910 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.910 DEBUG:boto:Params: {} 2022-09-13T07:35:50.911 DEBUG:boto:Token: None 2022-09-13T07:35:50.911 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.912 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MANRUNvECStqIRnF5xX1SOiHbAI= 2022-09-13T07:35:50.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MANRUNvECStqIRnF5xX1SOiHbAI=', 'Content-Length': '0'} 2022-09-13T07:35:50.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030ffd743951c7991-00632032d6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.915 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:50.916 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:50.916 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:50.917 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:50.917 DEBUG:boto:Method: GET 2022-09-13T07:35:50.918 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:50.918 DEBUG:boto:Data: 2022-09-13T07:35:50.919 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.920 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.920 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.921 DEBUG:boto:Params: {} 2022-09-13T07:35:50.922 DEBUG:boto:Token: None 2022-09-13T07:35:50.922 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/ 2022-09-13T07:35:50.923 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iXw5Oita+XzwrpkntPi4Ini5AyM= 2022-09-13T07:35:50.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iXw5Oita+XzwrpkntPi4Ini5AyM=', 'Content-Length': '0'} 2022-09-13T07:35:50.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db5d6a63b6755e68-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.927 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:50.928 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:50.928 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:50.929 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:50.929 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:50.930 DEBUG:boto:Method: GET 2022-09-13T07:35:50.930 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:50.931 DEBUG:boto:Data: 2022-09-13T07:35:50.931 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.932 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.932 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.933 DEBUG:boto:Params: {} 2022-09-13T07:35:50.933 DEBUG:boto:Token: None 2022-09-13T07:35:50.934 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/ 2022-09-13T07:35:50.934 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iXw5Oita+XzwrpkntPi4Ini5AyM= 2022-09-13T07:35:50.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iXw5Oita+XzwrpkntPi4Ini5AyM=', 'Content-Length': '0'} 2022-09-13T07:35:50.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000374b3201f6543697-00632032d6-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.938 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:50.939 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.940 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.940 DEBUG:boto:Method: HEAD 2022-09-13T07:35:50.941 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.941 DEBUG:boto:Data: 2022-09-13T07:35:50.942 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.942 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.943 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.943 DEBUG:boto:Params: {} 2022-09-13T07:35:50.944 DEBUG:boto:Token: None 2022-09-13T07:35:50.944 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.945 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:N6SAHuIFz2GrAw8JAKv+yM+VOKY= 2022-09-13T07:35:50.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:N6SAHuIFz2GrAw8JAKv+yM+VOKY=', 'Content-Length': '0'} 2022-09-13T07:35:50.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074084db8ca2de1bd-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.948 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.949 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.949 DEBUG:boto:Method: HEAD 2022-09-13T07:35:50.950 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.951 DEBUG:boto:Data: 2022-09-13T07:35:50.951 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.952 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.952 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.952 DEBUG:boto:Params: {} 2022-09-13T07:35:50.953 DEBUG:boto:Token: None 2022-09-13T07:35:50.954 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.954 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:N6SAHuIFz2GrAw8JAKv+yM+VOKY= 2022-09-13T07:35:50.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:N6SAHuIFz2GrAw8JAKv+yM+VOKY=', 'Content-Length': '0'} 2022-09-13T07:35:50.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000079f1d58a006a5a-00632032d6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:50.958 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.959 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.959 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.960 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.961 DEBUG:boto:Method: GET 2022-09-13T07:35:50.961 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.962 DEBUG:boto:Data: 2022-09-13T07:35:50.963 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.963 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.964 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.965 DEBUG:boto:Params: {} 2022-09-13T07:35:50.965 DEBUG:boto:Token: None 2022-09-13T07:35:50.966 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.967 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MANRUNvECStqIRnF5xX1SOiHbAI= 2022-09-13T07:35:50.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MANRUNvECStqIRnF5xX1SOiHbAI=', 'Content-Length': '0'} 2022-09-13T07:35:50.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8a67825ef9dedc2-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.971 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.971 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:50.972 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.973 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.973 DEBUG:boto:Method: GET 2022-09-13T07:35:50.974 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.975 DEBUG:boto:Data: 2022-09-13T07:35:50.976 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.977 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:50.978 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.979 DEBUG:boto:Params: {} 2022-09-13T07:35:50.980 DEBUG:boto:Token: None 2022-09-13T07:35:50.981 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:50.982 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MANRUNvECStqIRnF5xX1SOiHbAI= 2022-09-13T07:35:50.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MANRUNvECStqIRnF5xX1SOiHbAI=', 'Content-Length': '0'} 2022-09-13T07:35:50.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074d5d48ab1945144-00632032d6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.985 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:50.985 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:50.986 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:50.986 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:50.987 DEBUG:boto:Method: GET 2022-09-13T07:35:50.988 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:50.988 DEBUG:boto:Data: 2022-09-13T07:35:50.989 DEBUG:boto:Headers: {} 2022-09-13T07:35:50.989 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:50.990 DEBUG:boto:Port: 8000 2022-09-13T07:35:50.990 DEBUG:boto:Params: {} 2022-09-13T07:35:50.991 DEBUG:boto:Token: None 2022-09-13T07:35:50.992 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:50 GMT /yxvvuh-16/ 2022-09-13T07:35:50.992 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iXw5Oita+XzwrpkntPi4Ini5AyM= 2022-09-13T07:35:50.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iXw5Oita+XzwrpkntPi4Ini5AyM=', 'Content-Length': '0'} 2022-09-13T07:35:50.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebc3108477da57c4-00632032d6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:50.996 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:50.997 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:50.997 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:50.998 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:50.998 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:50.999 DEBUG:boto:Method: GET 2022-09-13T07:35:50.999 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:51.000 DEBUG:boto:Data: 2022-09-13T07:35:51.000 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.001 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.001 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.002 DEBUG:boto:Params: {} 2022-09-13T07:35:51.003 DEBUG:boto:Token: None 2022-09-13T07:35:51.003 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.004 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0= 2022-09-13T07:35:51.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0=', 'Content-Length': '0'} 2022-09-13T07:35:51.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000487db051458fbb5d-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.008 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:51.010 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.010 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.011 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.011 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.012 DEBUG:boto:Method: GET 2022-09-13T07:35:51.013 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.013 DEBUG:boto:Data: 2022-09-13T07:35:51.014 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.014 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:51.015 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.015 DEBUG:boto:Params: {} 2022-09-13T07:35:51.016 DEBUG:boto:Token: None 2022-09-13T07:35:51.016 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.017 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WbPRRwMZYi4VsCnD0lfmOS4XbTs= 2022-09-13T07:35:51.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WbPRRwMZYi4VsCnD0lfmOS4XbTs=', 'Content-Length': '0'} 2022-09-13T07:35:51.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006696bbbcc9a56b75-00632032d7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.062 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.063 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.063 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.064 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.064 DEBUG:boto:Method: GET 2022-09-13T07:35:51.065 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.065 DEBUG:boto:Data: 2022-09-13T07:35:51.066 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.066 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.067 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.067 DEBUG:boto:Params: {} 2022-09-13T07:35:51.068 DEBUG:boto:Token: None 2022-09-13T07:35:51.068 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.069 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WbPRRwMZYi4VsCnD0lfmOS4XbTs= 2022-09-13T07:35:51.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WbPRRwMZYi4VsCnD0lfmOS4XbTs=', 'Content-Length': '0'} 2022-09-13T07:35:51.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e950cb1bc96397a-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.073 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.074 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.075 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.076 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.077 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.078 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.079 DEBUG:boto:Data: 2022-09-13T07:35:51.080 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.080 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:51.081 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.082 DEBUG:boto:Params: {} 2022-09-13T07:35:51.082 DEBUG:boto:Token: None 2022-09-13T07:35:51.083 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.084 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yp0f0SjYeSAFU4OfcMAza+YvFjU= 2022-09-13T07:35:51.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yp0f0SjYeSAFU4OfcMAza+YvFjU=', 'Content-Length': '0'} 2022-09-13T07:35:51.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006dc33ec5a390ee6b-00632032d7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.087 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.088 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.088 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.089 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.089 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.090 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.090 DEBUG:boto:Data: 2022-09-13T07:35:51.091 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.091 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.091 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.092 DEBUG:boto:Params: {} 2022-09-13T07:35:51.093 DEBUG:boto:Token: None 2022-09-13T07:35:51.093 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.094 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yp0f0SjYeSAFU4OfcMAza+YvFjU= 2022-09-13T07:35:51.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yp0f0SjYeSAFU4OfcMAza+YvFjU=', 'Content-Length': '0'} 2022-09-13T07:35:51.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2f514e3048ae6f5-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:51.097 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.098 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.098 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.099 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.099 DEBUG:boto:Method: GET 2022-09-13T07:35:51.100 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.100 DEBUG:boto:Data: 2022-09-13T07:35:51.101 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.101 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:51.102 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.103 DEBUG:boto:Params: {} 2022-09-13T07:35:51.103 DEBUG:boto:Token: None 2022-09-13T07:35:51.104 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.104 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WbPRRwMZYi4VsCnD0lfmOS4XbTs= 2022-09-13T07:35:51.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WbPRRwMZYi4VsCnD0lfmOS4XbTs=', 'Content-Length': '0'} 2022-09-13T07:35:51.107 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a12f34e33fb82523-00632032d7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.107 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.108 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.108 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.109 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.109 DEBUG:boto:Method: GET 2022-09-13T07:35:51.110 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.110 DEBUG:boto:Data: 2022-09-13T07:35:51.111 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.111 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.112 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.112 DEBUG:boto:Params: {} 2022-09-13T07:35:51.113 DEBUG:boto:Token: None 2022-09-13T07:35:51.114 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.114 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WbPRRwMZYi4VsCnD0lfmOS4XbTs= 2022-09-13T07:35:51.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WbPRRwMZYi4VsCnD0lfmOS4XbTs=', 'Content-Length': '0'} 2022-09-13T07:35:51.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f620ba0be4c27f9-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.118 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.118 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.119 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:51.119 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:51.120 DEBUG:boto:Method: GET 2022-09-13T07:35:51.120 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:51.121 DEBUG:boto:Data: 2022-09-13T07:35:51.121 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.122 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:51.122 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.123 DEBUG:boto:Params: {} 2022-09-13T07:35:51.123 DEBUG:boto:Token: None 2022-09-13T07:35:51.124 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.124 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0= 2022-09-13T07:35:51.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0=', 'Content-Length': '0'} 2022-09-13T07:35:51.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c43fe423ec39768c-00632032d7-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.174 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.175 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.176 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.176 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:51.177 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:51.177 DEBUG:boto:Method: GET 2022-09-13T07:35:51.178 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:51.178 DEBUG:boto:Data: 2022-09-13T07:35:51.179 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.179 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.180 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.180 DEBUG:boto:Params: {} 2022-09-13T07:35:51.181 DEBUG:boto:Token: None 2022-09-13T07:35:51.181 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.182 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0= 2022-09-13T07:35:51.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0=', 'Content-Length': '0'} 2022-09-13T07:35:51.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f4fc0bc7caedfd0-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.185 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.186 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.187 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.187 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.188 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.188 DEBUG:boto:Data: 2022-09-13T07:35:51.189 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.189 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:51.190 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.190 DEBUG:boto:Params: {} 2022-09-13T07:35:51.191 DEBUG:boto:Token: None 2022-09-13T07:35:51.191 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.192 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/iLsywQECrTbNn9umAJzaDHtkys= 2022-09-13T07:35:51.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/iLsywQECrTbNn9umAJzaDHtkys=', 'Content-Length': '0'} 2022-09-13T07:35:51.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7f3a06a97864c48-00632032d7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.195 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.196 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.196 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.196 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.197 DEBUG:boto:Data: 2022-09-13T07:35:51.197 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.198 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.198 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.199 DEBUG:boto:Params: {} 2022-09-13T07:35:51.199 DEBUG:boto:Token: None 2022-09-13T07:35:51.200 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.200 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/iLsywQECrTbNn9umAJzaDHtkys= 2022-09-13T07:35:51.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/iLsywQECrTbNn9umAJzaDHtkys=', 'Content-Length': '0'} 2022-09-13T07:35:51.203 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4eab4c1a65aaf51-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:51.204 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.205 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.205 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.206 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.206 DEBUG:boto:Method: GET 2022-09-13T07:35:51.207 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.208 DEBUG:boto:Data: 2022-09-13T07:35:51.208 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.209 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:51.209 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.210 DEBUG:boto:Params: {} 2022-09-13T07:35:51.210 DEBUG:boto:Token: None 2022-09-13T07:35:51.211 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.211 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WbPRRwMZYi4VsCnD0lfmOS4XbTs= 2022-09-13T07:35:51.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WbPRRwMZYi4VsCnD0lfmOS4XbTs=', 'Content-Length': '0'} 2022-09-13T07:35:51.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd3476c449f455b0-00632032d7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.215 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.215 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:51.216 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.216 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.217 DEBUG:boto:Method: GET 2022-09-13T07:35:51.217 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.218 DEBUG:boto:Data: 2022-09-13T07:35:51.218 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.219 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.219 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.220 DEBUG:boto:Params: {} 2022-09-13T07:35:51.220 DEBUG:boto:Token: None 2022-09-13T07:35:51.221 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:51.222 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WbPRRwMZYi4VsCnD0lfmOS4XbTs= 2022-09-13T07:35:51.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WbPRRwMZYi4VsCnD0lfmOS4XbTs=', 'Content-Length': '0'} 2022-09-13T07:35:51.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af03244bb27de107-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.225 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.225 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.226 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:51.226 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:51.227 DEBUG:boto:Method: GET 2022-09-13T07:35:51.227 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:51.228 DEBUG:boto:Data: 2022-09-13T07:35:51.228 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.229 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:51.229 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.230 DEBUG:boto:Params: {} 2022-09-13T07:35:51.230 DEBUG:boto:Token: None 2022-09-13T07:35:51.231 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.232 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0= 2022-09-13T07:35:51.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0=', 'Content-Length': '0'} 2022-09-13T07:35:51.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce2faf26806cb89c-00632032d7-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.278 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.279 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.280 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.280 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:51.281 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:51.281 DEBUG:boto:Method: GET 2022-09-13T07:35:51.281 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:51.282 DEBUG:boto:Data: 2022-09-13T07:35:51.283 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.283 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.283 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.284 DEBUG:boto:Params: {} 2022-09-13T07:35:51.285 DEBUG:boto:Token: None 2022-09-13T07:35:51.285 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.286 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0= 2022-09-13T07:35:51.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0=', 'Content-Length': '0'} 2022-09-13T07:35:51.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b96b2584ed2eb2d3-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.289 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:51.291 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.291 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.292 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.292 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.293 DEBUG:boto:Method: GET 2022-09-13T07:35:51.293 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.294 DEBUG:boto:Data: 2022-09-13T07:35:51.294 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.295 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:51.295 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.296 DEBUG:boto:Params: {} 2022-09-13T07:35:51.296 DEBUG:boto:Token: None 2022-09-13T07:35:51.297 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.298 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h8clgB2evzPgBRvb7g3JmjpYVu4= 2022-09-13T07:35:51.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h8clgB2evzPgBRvb7g3JmjpYVu4=', 'Content-Length': '0'} 2022-09-13T07:35:51.300 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c3b6dd292621d32-00632032d7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.301 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.302 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.302 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.303 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.303 DEBUG:boto:Method: GET 2022-09-13T07:35:51.304 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.304 DEBUG:boto:Data: 2022-09-13T07:35:51.305 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.305 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.306 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.306 DEBUG:boto:Params: {} 2022-09-13T07:35:51.307 DEBUG:boto:Token: None 2022-09-13T07:35:51.307 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.308 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h8clgB2evzPgBRvb7g3JmjpYVu4= 2022-09-13T07:35:51.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h8clgB2evzPgBRvb7g3JmjpYVu4=', 'Content-Length': '0'} 2022-09-13T07:35:51.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4a8a9468763cef8-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.312 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.312 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.313 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.313 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.314 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.314 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.315 DEBUG:boto:Data: 2022-09-13T07:35:51.315 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.316 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:51.316 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.317 DEBUG:boto:Params: {} 2022-09-13T07:35:51.317 DEBUG:boto:Token: None 2022-09-13T07:35:51.318 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.318 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WAde2aP4Mvi9E1ZOOidCqaTY3g4= 2022-09-13T07:35:51.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WAde2aP4Mvi9E1ZOOidCqaTY3g4=', 'Content-Length': '0'} 2022-09-13T07:35:51.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000257d3fc9edd97824-00632032d7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.321 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.322 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.323 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.323 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.324 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.324 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.324 DEBUG:boto:Data: 2022-09-13T07:35:51.325 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.325 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.326 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.326 DEBUG:boto:Params: {} 2022-09-13T07:35:51.327 DEBUG:boto:Token: None 2022-09-13T07:35:51.327 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.328 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WAde2aP4Mvi9E1ZOOidCqaTY3g4= 2022-09-13T07:35:51.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WAde2aP4Mvi9E1ZOOidCqaTY3g4=', 'Content-Length': '0'} 2022-09-13T07:35:51.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a4c36db7a1b80c1-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:51.331 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.332 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.333 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.333 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.334 DEBUG:boto:Method: GET 2022-09-13T07:35:51.334 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.335 DEBUG:boto:Data: 2022-09-13T07:35:51.335 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.336 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:51.336 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.337 DEBUG:boto:Params: {} 2022-09-13T07:35:51.337 DEBUG:boto:Token: None 2022-09-13T07:35:51.338 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.339 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h8clgB2evzPgBRvb7g3JmjpYVu4= 2022-09-13T07:35:51.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h8clgB2evzPgBRvb7g3JmjpYVu4=', 'Content-Length': '0'} 2022-09-13T07:35:51.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f7899d5af9716eb-00632032d7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.342 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.342 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.343 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.343 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.344 DEBUG:boto:Method: GET 2022-09-13T07:35:51.344 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.345 DEBUG:boto:Data: 2022-09-13T07:35:51.345 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.346 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.346 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.347 DEBUG:boto:Params: {} 2022-09-13T07:35:51.348 DEBUG:boto:Token: None 2022-09-13T07:35:51.348 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.349 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h8clgB2evzPgBRvb7g3JmjpYVu4= 2022-09-13T07:35:51.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h8clgB2evzPgBRvb7g3JmjpYVu4=', 'Content-Length': '0'} 2022-09-13T07:35:51.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3b3192097a4b100-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.394 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.395 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.395 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:51.396 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:51.396 DEBUG:boto:Method: GET 2022-09-13T07:35:51.397 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:51.397 DEBUG:boto:Data: 2022-09-13T07:35:51.398 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.399 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:51.399 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.400 DEBUG:boto:Params: {} 2022-09-13T07:35:51.400 DEBUG:boto:Token: None 2022-09-13T07:35:51.401 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.402 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0= 2022-09-13T07:35:51.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0=', 'Content-Length': '0'} 2022-09-13T07:35:51.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084773aca4d8eafce-00632032d7-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.405 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.406 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.407 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.407 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:51.408 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:51.408 DEBUG:boto:Method: GET 2022-09-13T07:35:51.409 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:51.409 DEBUG:boto:Data: 2022-09-13T07:35:51.410 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.411 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.411 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.412 DEBUG:boto:Params: {} 2022-09-13T07:35:51.412 DEBUG:boto:Token: None 2022-09-13T07:35:51.413 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.413 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0= 2022-09-13T07:35:51.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0=', 'Content-Length': '0'} 2022-09-13T07:35:51.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029b15152c9df6768-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.417 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.418 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.419 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.419 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.420 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.420 DEBUG:boto:Data: 2022-09-13T07:35:51.421 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.421 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:51.422 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.422 DEBUG:boto:Params: {} 2022-09-13T07:35:51.423 DEBUG:boto:Token: None 2022-09-13T07:35:51.423 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.424 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pdTTnDv/2oUjbHmD/pgkShAGVXM= 2022-09-13T07:35:51.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pdTTnDv/2oUjbHmD/pgkShAGVXM=', 'Content-Length': '0'} 2022-09-13T07:35:51.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c004ef2c07d899d-00632032d7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.428 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.428 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.429 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.429 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.430 DEBUG:boto:Data: 2022-09-13T07:35:51.430 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.431 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.431 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.432 DEBUG:boto:Params: {} 2022-09-13T07:35:51.432 DEBUG:boto:Token: None 2022-09-13T07:35:51.433 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.433 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pdTTnDv/2oUjbHmD/pgkShAGVXM= 2022-09-13T07:35:51.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pdTTnDv/2oUjbHmD/pgkShAGVXM=', 'Content-Length': '0'} 2022-09-13T07:35:51.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ac6fbe4d06cc560-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:51.437 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.438 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.438 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.439 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.439 DEBUG:boto:Method: GET 2022-09-13T07:35:51.440 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.440 DEBUG:boto:Data: 2022-09-13T07:35:51.441 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.441 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:51.442 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.443 DEBUG:boto:Params: {} 2022-09-13T07:35:51.443 DEBUG:boto:Token: None 2022-09-13T07:35:51.444 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.444 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h8clgB2evzPgBRvb7g3JmjpYVu4= 2022-09-13T07:35:51.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h8clgB2evzPgBRvb7g3JmjpYVu4=', 'Content-Length': '0'} 2022-09-13T07:35:51.447 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f15f2fcf7b07823-00632032d7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.448 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.448 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:51.449 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.449 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.450 DEBUG:boto:Method: GET 2022-09-13T07:35:51.450 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.451 DEBUG:boto:Data: 2022-09-13T07:35:51.451 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.452 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.452 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.453 DEBUG:boto:Params: {} 2022-09-13T07:35:51.453 DEBUG:boto:Token: None 2022-09-13T07:35:51.454 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:51.455 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h8clgB2evzPgBRvb7g3JmjpYVu4= 2022-09-13T07:35:51.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h8clgB2evzPgBRvb7g3JmjpYVu4=', 'Content-Length': '0'} 2022-09-13T07:35:51.457 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007200a237473307b9-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.458 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.459 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.459 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:51.460 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:51.460 DEBUG:boto:Method: GET 2022-09-13T07:35:51.461 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:51.461 DEBUG:boto:Data: 2022-09-13T07:35:51.462 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.462 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:51.463 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.463 DEBUG:boto:Params: {} 2022-09-13T07:35:51.464 DEBUG:boto:Token: None 2022-09-13T07:35:51.465 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.465 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0= 2022-09-13T07:35:51.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0=', 'Content-Length': '0'} 2022-09-13T07:35:51.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eecfc3cbd943c410-00632032d7-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.510 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.511 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.512 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.512 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:51.513 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:51.513 DEBUG:boto:Method: GET 2022-09-13T07:35:51.514 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:51.514 DEBUG:boto:Data: 2022-09-13T07:35:51.515 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.515 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.516 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.516 DEBUG:boto:Params: {} 2022-09-13T07:35:51.517 DEBUG:boto:Token: None 2022-09-13T07:35:51.518 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.518 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0= 2022-09-13T07:35:51.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0=', 'Content-Length': '0'} 2022-09-13T07:35:51.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf11c4f1d2ed9521-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.522 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.523 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-16 zones={test-zone2, test-zone1} 2022-09-13T07:35:51.524 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-16 zones={test-zone3, test-zone1} 2022-09-13T07:35:51.524 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.525 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.525 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.526 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:35:51.526 DEBUG:boto:Data: 2022-09-13T07:35:51.527 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.527 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.528 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.528 DEBUG:boto:Params: {} 2022-09-13T07:35:51.529 DEBUG:boto:Token: None 2022-09-13T07:35:51.530 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.530 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GhXKwEpMCq+ooDWRuiOmmOwuSb4= 2022-09-13T07:35:51.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GhXKwEpMCq+ooDWRuiOmmOwuSb4=', 'Content-Length': '0'} 2022-09-13T07:35:51.533 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9ca677fa77ae4b4-00632032d7-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.534 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.535 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.535 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.536 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:35:51.536 DEBUG:boto:Data: 2022-09-13T07:35:51.537 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.537 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.538 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.539 DEBUG:boto:Params: {} 2022-09-13T07:35:51.540 DEBUG:boto:Token: None 2022-09-13T07:35:51.541 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.542 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GhXKwEpMCq+ooDWRuiOmmOwuSb4= 2022-09-13T07:35:51.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GhXKwEpMCq+ooDWRuiOmmOwuSb4=', 'Content-Length': '0'} 2022-09-13T07:35:51.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024ba9af71b3dd95e-00632032d7-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.556 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:35:51.556 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.557 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.558 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:51.559 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:51.559 DEBUG:boto:Method: GET 2022-09-13T07:35:51.560 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:51.561 DEBUG:boto:Data: 2022-09-13T07:35:51.562 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.563 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.564 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.564 DEBUG:boto:Params: {} 2022-09-13T07:35:51.565 DEBUG:boto:Token: None 2022-09-13T07:35:51.566 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/?versions 2022-09-13T07:35:51.567 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bsv8qMltItEJw45l5usYORhqR2k= 2022-09-13T07:35:51.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bsv8qMltItEJw45l5usYORhqR2k=', 'Content-Length': '0'} 2022-09-13T07:35:51.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2f1e040dfaf8f10-00632032d7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.571 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.572 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:35:51.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:35:51.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:35:51.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:35:51.574 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:35:51.575 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.575 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.576 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:51.576 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:51.577 DEBUG:boto:Method: GET 2022-09-13T07:35:51.577 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:51.578 DEBUG:boto:Data: 2022-09-13T07:35:51.578 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.579 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.579 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.580 DEBUG:boto:Params: {} 2022-09-13T07:35:51.581 DEBUG:boto:Token: None 2022-09-13T07:35:51.581 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/?versions 2022-09-13T07:35:51.582 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bsv8qMltItEJw45l5usYORhqR2k= 2022-09-13T07:35:51.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bsv8qMltItEJw45l5usYORhqR2k=', 'Content-Length': '0'} 2022-09-13T07:35:51.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086e7bd7d14f6130d-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.586 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.587 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:35:51.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:35:51.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:35:51.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:35:51.589 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.590 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.590 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:51.591 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:51.591 DEBUG:boto:Method: GET 2022-09-13T07:35:51.592 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:51.592 DEBUG:boto:Data: 2022-09-13T07:35:51.593 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.593 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.594 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.594 DEBUG:boto:Params: {} 2022-09-13T07:35:51.595 DEBUG:boto:Token: None 2022-09-13T07:35:51.596 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/?versions 2022-09-13T07:35:51.596 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bsv8qMltItEJw45l5usYORhqR2k= 2022-09-13T07:35:51.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bsv8qMltItEJw45l5usYORhqR2k=', 'Content-Length': '0'} 2022-09-13T07:35:51.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d945d3598960403-00632032d7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.600 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.601 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.602 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.602 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:51.603 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:51.603 DEBUG:boto:Method: GET 2022-09-13T07:35:51.604 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:51.604 DEBUG:boto:Data: 2022-09-13T07:35:51.605 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.605 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.606 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.606 DEBUG:boto:Params: {} 2022-09-13T07:35:51.607 DEBUG:boto:Token: None 2022-09-13T07:35:51.607 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/?versions 2022-09-13T07:35:51.608 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bsv8qMltItEJw45l5usYORhqR2k= 2022-09-13T07:35:51.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bsv8qMltItEJw45l5usYORhqR2k=', 'Content-Length': '0'} 2022-09-13T07:35:51.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001179a277cb4cb243-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.612 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:51.613 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:51.614 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:51.614 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.615 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.615 DEBUG:boto:Method: GET 2022-09-13T07:35:51.616 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.616 DEBUG:boto:Data: 2022-09-13T07:35:51.617 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.617 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.618 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.618 DEBUG:boto:Params: {} 2022-09-13T07:35:51.619 DEBUG:boto:Token: None 2022-09-13T07:35:51.619 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.620 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MkTnVUPPocwQ2Adu5CqNIsdWzSA= 2022-09-13T07:35:51.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MkTnVUPPocwQ2Adu5CqNIsdWzSA=', 'Content-Length': '0'} 2022-09-13T07:35:51.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004106f1d359c22949-00632032d7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.623 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:51.624 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:51.624 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.625 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.625 DEBUG:boto:Method: GET 2022-09-13T07:35:51.626 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.626 DEBUG:boto:Data: 2022-09-13T07:35:51.627 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.627 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.628 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.628 DEBUG:boto:Params: {} 2022-09-13T07:35:51.629 DEBUG:boto:Token: None 2022-09-13T07:35:51.630 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.630 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MkTnVUPPocwQ2Adu5CqNIsdWzSA= 2022-09-13T07:35:51.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MkTnVUPPocwQ2Adu5CqNIsdWzSA=', 'Content-Length': '0'} 2022-09-13T07:35:51.633 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005c2cb13083986ea9-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.634 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:51.634 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:51.635 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.635 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.636 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.636 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.637 DEBUG:boto:Data: 2022-09-13T07:35:51.637 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.638 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.638 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.639 DEBUG:boto:Params: {} 2022-09-13T07:35:51.639 DEBUG:boto:Token: None 2022-09-13T07:35:51.640 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.640 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IUq0pOyuGIgdOo5D2PuNnVX5+sg= 2022-09-13T07:35:51.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IUq0pOyuGIgdOo5D2PuNnVX5+sg=', 'Content-Length': '0'} 2022-09-13T07:35:51.643 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000084880047e7798f2-00632032d7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.644 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:51.644 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:51.645 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.645 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.646 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.646 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.647 DEBUG:boto:Data: 2022-09-13T07:35:51.647 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.648 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.648 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.648 DEBUG:boto:Params: {} 2022-09-13T07:35:51.649 DEBUG:boto:Token: None 2022-09-13T07:35:51.650 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.650 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IUq0pOyuGIgdOo5D2PuNnVX5+sg= 2022-09-13T07:35:51.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IUq0pOyuGIgdOo5D2PuNnVX5+sg=', 'Content-Length': '0'} 2022-09-13T07:35:51.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000039476bb51ac37f5e-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:51.654 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:51.654 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:51.655 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.655 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.656 DEBUG:boto:Method: GET 2022-09-13T07:35:51.656 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.657 DEBUG:boto:Data: 2022-09-13T07:35:51.658 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.658 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.659 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.659 DEBUG:boto:Params: {} 2022-09-13T07:35:51.660 DEBUG:boto:Token: None 2022-09-13T07:35:51.660 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.661 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MkTnVUPPocwQ2Adu5CqNIsdWzSA= 2022-09-13T07:35:51.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MkTnVUPPocwQ2Adu5CqNIsdWzSA=', 'Content-Length': '0'} 2022-09-13T07:35:51.664 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006ad0c46db4de85f5-00632032d7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.665 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:51.665 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:51.666 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.666 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.667 DEBUG:boto:Method: GET 2022-09-13T07:35:51.667 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.668 DEBUG:boto:Data: 2022-09-13T07:35:51.668 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.669 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.669 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.670 DEBUG:boto:Params: {} 2022-09-13T07:35:51.671 DEBUG:boto:Token: None 2022-09-13T07:35:51.671 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.672 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MkTnVUPPocwQ2Adu5CqNIsdWzSA= 2022-09-13T07:35:51.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MkTnVUPPocwQ2Adu5CqNIsdWzSA=', 'Content-Length': '0'} 2022-09-13T07:35:51.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000348eae21345a3364-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.676 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.676 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.677 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:51.677 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:51.678 DEBUG:boto:Method: GET 2022-09-13T07:35:51.678 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:35:51.679 DEBUG:boto:Data: 2022-09-13T07:35:51.679 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.680 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.680 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.681 DEBUG:boto:Params: {} 2022-09-13T07:35:51.682 DEBUG:boto:Token: None 2022-09-13T07:35:51.682 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.683 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0= 2022-09-13T07:35:51.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0=', 'Content-Length': '0'} 2022-09-13T07:35:51.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001454fdbcde696623-00632032d7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.730 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.732 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.732 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.733 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:51.733 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:51.734 DEBUG:boto:Method: GET 2022-09-13T07:35:51.734 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:35:51.735 DEBUG:boto:Data: 2022-09-13T07:35:51.736 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.737 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.737 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.738 DEBUG:boto:Params: {} 2022-09-13T07:35:51.739 DEBUG:boto:Token: None 2022-09-13T07:35:51.739 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.740 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0= 2022-09-13T07:35:51.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0=', 'Content-Length': '0'} 2022-09-13T07:35:51.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054c79365221ea7c0-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.744 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.745 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:51.745 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:51.746 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.746 DEBUG:boto:Path: /yxvvuh-16/dummy 2022-09-13T07:35:51.747 DEBUG:boto:Data: 2022-09-13T07:35:51.747 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.748 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.748 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.749 DEBUG:boto:Params: {} 2022-09-13T07:35:51.749 DEBUG:boto:Token: None 2022-09-13T07:35:51.750 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/dummy 2022-09-13T07:35:51.750 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lvxIKw5fhglLLD0sbnqc7zKtD4c= 2022-09-13T07:35:51.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lvxIKw5fhglLLD0sbnqc7zKtD4c=', 'Content-Length': '0'} 2022-09-13T07:35:51.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c8591e1c042e6616-00632032d7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.754 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:51.755 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:51.756 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.756 DEBUG:boto:Path: /yxvvuh-16/dummy 2022-09-13T07:35:51.757 DEBUG:boto:Data: 2022-09-13T07:35:51.757 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.758 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.758 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.759 DEBUG:boto:Params: {} 2022-09-13T07:35:51.759 DEBUG:boto:Token: None 2022-09-13T07:35:51.760 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/dummy 2022-09-13T07:35:51.760 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lvxIKw5fhglLLD0sbnqc7zKtD4c= 2022-09-13T07:35:51.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lvxIKw5fhglLLD0sbnqc7zKtD4c=', 'Content-Length': '0'} 2022-09-13T07:35:51.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000069ad7257cda7886e-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:51.765 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:51.765 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:51.766 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.766 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.767 DEBUG:boto:Method: GET 2022-09-13T07:35:51.767 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.768 DEBUG:boto:Data: 2022-09-13T07:35:51.769 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.769 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.770 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.770 DEBUG:boto:Params: {} 2022-09-13T07:35:51.771 DEBUG:boto:Token: None 2022-09-13T07:35:51.771 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.772 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MkTnVUPPocwQ2Adu5CqNIsdWzSA= 2022-09-13T07:35:51.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MkTnVUPPocwQ2Adu5CqNIsdWzSA=', 'Content-Length': '0'} 2022-09-13T07:35:51.774 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023f7a5a81f5b82c0-00632032d7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.775 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:51.776 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:51.776 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.777 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.777 DEBUG:boto:Method: GET 2022-09-13T07:35:51.778 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.778 DEBUG:boto:Data: 2022-09-13T07:35:51.779 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.780 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.780 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.781 DEBUG:boto:Params: {} 2022-09-13T07:35:51.781 DEBUG:boto:Token: None 2022-09-13T07:35:51.782 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:51.783 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MkTnVUPPocwQ2Adu5CqNIsdWzSA= 2022-09-13T07:35:51.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MkTnVUPPocwQ2Adu5CqNIsdWzSA=', 'Content-Length': '0'} 2022-09-13T07:35:51.785 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000007fc5e08731568e-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.786 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.786 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.787 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:51.787 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:51.788 DEBUG:boto:Method: GET 2022-09-13T07:35:51.788 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:35:51.789 DEBUG:boto:Data: 2022-09-13T07:35:51.789 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.790 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.791 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.791 DEBUG:boto:Params: {} 2022-09-13T07:35:51.792 DEBUG:boto:Token: None 2022-09-13T07:35:51.792 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.793 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0= 2022-09-13T07:35:51.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0=', 'Content-Length': '0'} 2022-09-13T07:35:51.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cbe73b7244673ce-00632032d7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.797 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.797 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.798 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.798 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:51.799 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:51.800 DEBUG:boto:Method: GET 2022-09-13T07:35:51.800 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:35:51.801 DEBUG:boto:Data: 2022-09-13T07:35:51.801 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.802 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.802 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.803 DEBUG:boto:Params: {} 2022-09-13T07:35:51.803 DEBUG:boto:Token: None 2022-09-13T07:35:51.804 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.804 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0= 2022-09-13T07:35:51.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0=', 'Content-Length': '0'} 2022-09-13T07:35:51.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc03f4375f20d32e-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.808 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:51.810 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.810 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.811 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.811 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.812 DEBUG:boto:Method: GET 2022-09-13T07:35:51.812 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.813 DEBUG:boto:Data: 2022-09-13T07:35:51.813 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.814 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.814 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.815 DEBUG:boto:Params: {} 2022-09-13T07:35:51.816 DEBUG:boto:Token: None 2022-09-13T07:35:51.816 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.817 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bfFadFc4HtUAbmCtw/re+Gyy+aM= 2022-09-13T07:35:51.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bfFadFc4HtUAbmCtw/re+Gyy+aM=', 'Content-Length': '0'} 2022-09-13T07:35:51.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026f2001517ff7050-00632032d7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.824 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.825 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.826 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.827 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.828 DEBUG:boto:Method: GET 2022-09-13T07:35:51.828 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.829 DEBUG:boto:Data: 2022-09-13T07:35:51.830 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.831 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.832 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.833 DEBUG:boto:Params: {} 2022-09-13T07:35:51.834 DEBUG:boto:Token: None 2022-09-13T07:35:51.834 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.835 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bfFadFc4HtUAbmCtw/re+Gyy+aM= 2022-09-13T07:35:51.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bfFadFc4HtUAbmCtw/re+Gyy+aM=', 'Content-Length': '0'} 2022-09-13T07:35:51.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079e4fd0669ad3ca7-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.878 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.879 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.879 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.880 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.880 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.881 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.881 DEBUG:boto:Data: 2022-09-13T07:35:51.882 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.882 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.883 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.883 DEBUG:boto:Params: {} 2022-09-13T07:35:51.884 DEBUG:boto:Token: None 2022-09-13T07:35:51.885 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.885 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:K1lI7exm0o45nTv7lkQkdo6C8CA= 2022-09-13T07:35:51.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:K1lI7exm0o45nTv7lkQkdo6C8CA=', 'Content-Length': '0'} 2022-09-13T07:35:51.888 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de57d1ef35d3b249-00632032d7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.888 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.889 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.889 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.890 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.891 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.891 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.892 DEBUG:boto:Data: 2022-09-13T07:35:51.892 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.893 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.893 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.894 DEBUG:boto:Params: {} 2022-09-13T07:35:51.894 DEBUG:boto:Token: None 2022-09-13T07:35:51.895 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.896 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:K1lI7exm0o45nTv7lkQkdo6C8CA= 2022-09-13T07:35:51.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:K1lI7exm0o45nTv7lkQkdo6C8CA=', 'Content-Length': '0'} 2022-09-13T07:35:51.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000478b44d3f27bb143-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:51.900 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.900 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.901 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.901 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.902 DEBUG:boto:Method: GET 2022-09-13T07:35:51.902 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.903 DEBUG:boto:Data: 2022-09-13T07:35:51.903 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.904 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.904 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.905 DEBUG:boto:Params: {} 2022-09-13T07:35:51.905 DEBUG:boto:Token: None 2022-09-13T07:35:51.906 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.907 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bfFadFc4HtUAbmCtw/re+Gyy+aM= 2022-09-13T07:35:51.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bfFadFc4HtUAbmCtw/re+Gyy+aM=', 'Content-Length': '0'} 2022-09-13T07:35:51.909 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002114e393407ea6b8-00632032d7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.910 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.911 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.912 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.912 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.913 DEBUG:boto:Method: GET 2022-09-13T07:35:51.913 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.914 DEBUG:boto:Data: 2022-09-13T07:35:51.914 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.915 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.915 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.916 DEBUG:boto:Params: {} 2022-09-13T07:35:51.916 DEBUG:boto:Token: None 2022-09-13T07:35:51.917 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.917 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bfFadFc4HtUAbmCtw/re+Gyy+aM= 2022-09-13T07:35:51.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bfFadFc4HtUAbmCtw/re+Gyy+aM=', 'Content-Length': '0'} 2022-09-13T07:35:51.920 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000080b3c2613071ee2-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.921 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.922 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.922 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:51.923 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:51.923 DEBUG:boto:Method: GET 2022-09-13T07:35:51.924 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:51.924 DEBUG:boto:Data: 2022-09-13T07:35:51.925 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.926 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.926 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.927 DEBUG:boto:Params: {} 2022-09-13T07:35:51.927 DEBUG:boto:Token: None 2022-09-13T07:35:51.928 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.928 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0= 2022-09-13T07:35:51.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0=', 'Content-Length': '0'} 2022-09-13T07:35:51.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007044e1d8b37ccd7b-00632032d7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.932 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.933 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:51.934 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:51.934 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:51.935 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:51.936 DEBUG:boto:Method: GET 2022-09-13T07:35:51.936 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:51.937 DEBUG:boto:Data: 2022-09-13T07:35:51.937 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.938 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.938 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.939 DEBUG:boto:Params: {} 2022-09-13T07:35:51.939 DEBUG:boto:Token: None 2022-09-13T07:35:51.940 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/ 2022-09-13T07:35:51.940 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0= 2022-09-13T07:35:51.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b0/oRS8pcWKv0VCxWvk0uZX8nf0=', 'Content-Length': '0'} 2022-09-13T07:35:51.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000edd3c7881e5753a-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.944 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:51.945 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.946 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.946 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.947 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.947 DEBUG:boto:Data: 2022-09-13T07:35:51.948 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.948 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.949 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.949 DEBUG:boto:Params: {} 2022-09-13T07:35:51.950 DEBUG:boto:Token: None 2022-09-13T07:35:51.951 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.951 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0liOcUs30eqAQDlnvUxx5MDDkms= 2022-09-13T07:35:51.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0liOcUs30eqAQDlnvUxx5MDDkms=', 'Content-Length': '0'} 2022-09-13T07:35:51.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5856c08bedc9cfc-00632032d7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.955 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.955 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.956 DEBUG:boto:Method: HEAD 2022-09-13T07:35:51.957 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.957 DEBUG:boto:Data: 2022-09-13T07:35:51.958 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.958 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:51.959 DEBUG:boto:Port: 8000 2022-09-13T07:35:51.959 DEBUG:boto:Params: {} 2022-09-13T07:35:51.960 DEBUG:boto:Token: None 2022-09-13T07:35:51.960 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.961 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0liOcUs30eqAQDlnvUxx5MDDkms= 2022-09-13T07:35:51.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0liOcUs30eqAQDlnvUxx5MDDkms=', 'Content-Length': '0'} 2022-09-13T07:35:51.964 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cab8c2a72519906-00632032d7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:51.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:51.965 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.966 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:51.966 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.967 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.967 DEBUG:boto:Method: GET 2022-09-13T07:35:51.968 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.968 DEBUG:boto:Data: 2022-09-13T07:35:51.969 DEBUG:boto:Headers: {} 2022-09-13T07:35:51.969 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:51.970 DEBUG:boto:Port: 8001 2022-09-13T07:35:51.971 DEBUG:boto:Params: {} 2022-09-13T07:35:51.971 DEBUG:boto:Token: None 2022-09-13T07:35:51.972 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:51 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:51.972 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bfFadFc4HtUAbmCtw/re+Gyy+aM= 2022-09-13T07:35:51.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bfFadFc4HtUAbmCtw/re+Gyy+aM=', 'Content-Length': '0'} 2022-09-13T07:35:51.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000049d20503cf48d81-00632032d7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.018 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:52.019 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:52.020 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.021 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.022 DEBUG:boto:Method: GET 2022-09-13T07:35:52.023 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.024 DEBUG:boto:Data: 2022-09-13T07:35:52.025 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.026 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.027 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.028 DEBUG:boto:Params: {} 2022-09-13T07:35:52.029 DEBUG:boto:Token: None 2022-09-13T07:35:52.030 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.031 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3XDGI10P61KrlXGA81obXWq/Zck= 2022-09-13T07:35:52.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3XDGI10P61KrlXGA81obXWq/Zck=', 'Content-Length': '0'} 2022-09-13T07:35:52.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ff7480bc75959e8-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.036 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.037 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.038 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:52.039 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:52.040 DEBUG:boto:Method: GET 2022-09-13T07:35:52.041 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:52.042 DEBUG:boto:Data: 2022-09-13T07:35:52.043 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.044 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.045 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.046 DEBUG:boto:Params: {} 2022-09-13T07:35:52.047 DEBUG:boto:Token: None 2022-09-13T07:35:52.048 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.049 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk= 2022-09-13T07:35:52.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk=', 'Content-Length': '0'} 2022-09-13T07:35:52.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088e7731a02cfeb7c-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.054 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.056 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.057 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.058 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:52.059 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:52.060 DEBUG:boto:Method: GET 2022-09-13T07:35:52.061 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:52.062 DEBUG:boto:Data: 2022-09-13T07:35:52.063 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.064 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.065 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.066 DEBUG:boto:Params: {} 2022-09-13T07:35:52.067 DEBUG:boto:Token: None 2022-09-13T07:35:52.068 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.069 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk= 2022-09-13T07:35:52.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk=', 'Content-Length': '0'} 2022-09-13T07:35:52.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f44852551f0d1846-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.116 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:52.117 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.118 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.119 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.120 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.121 DEBUG:boto:Method: GET 2022-09-13T07:35:52.122 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.123 DEBUG:boto:Data: 2022-09-13T07:35:52.124 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.125 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.126 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.127 DEBUG:boto:Params: {} 2022-09-13T07:35:52.128 DEBUG:boto:Token: None 2022-09-13T07:35:52.130 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.131 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fBPGexS1x/hbTIk9QtwVHoCgKRc= 2022-09-13T07:35:52.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fBPGexS1x/hbTIk9QtwVHoCgKRc=', 'Content-Length': '0'} 2022-09-13T07:35:52.135 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b21a922c147d8898-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.136 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.137 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.138 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.139 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.140 DEBUG:boto:Method: GET 2022-09-13T07:35:52.141 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.142 DEBUG:boto:Data: 2022-09-13T07:35:52.143 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.144 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.145 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.146 DEBUG:boto:Params: {} 2022-09-13T07:35:52.147 DEBUG:boto:Token: None 2022-09-13T07:35:52.148 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.149 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fBPGexS1x/hbTIk9QtwVHoCgKRc= 2022-09-13T07:35:52.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fBPGexS1x/hbTIk9QtwVHoCgKRc=', 'Content-Length': '0'} 2022-09-13T07:35:52.153 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004324b2b1b00dc746-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.154 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.155 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.156 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.156 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.157 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.158 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.159 DEBUG:boto:Data: 2022-09-13T07:35:52.160 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.161 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.161 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.162 DEBUG:boto:Params: {} 2022-09-13T07:35:52.163 DEBUG:boto:Token: None 2022-09-13T07:35:52.165 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.166 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:JwLlZDotHI1Xz62ICpJny0LaWPk= 2022-09-13T07:35:52.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JwLlZDotHI1Xz62ICpJny0LaWPk=', 'Content-Length': '0'} 2022-09-13T07:35:52.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfe98bce7c0fe73b-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.171 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.172 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.173 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.174 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.175 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.175 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.176 DEBUG:boto:Data: 2022-09-13T07:35:52.177 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.178 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.179 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.180 DEBUG:boto:Params: {} 2022-09-13T07:35:52.181 DEBUG:boto:Token: None 2022-09-13T07:35:52.181 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.182 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:JwLlZDotHI1Xz62ICpJny0LaWPk= 2022-09-13T07:35:52.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JwLlZDotHI1Xz62ICpJny0LaWPk=', 'Content-Length': '0'} 2022-09-13T07:35:52.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3f80c95d426d917-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:52.187 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.188 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.188 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.189 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.189 DEBUG:boto:Method: GET 2022-09-13T07:35:52.190 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.190 DEBUG:boto:Data: 2022-09-13T07:35:52.191 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.191 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.192 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.192 DEBUG:boto:Params: {} 2022-09-13T07:35:52.193 DEBUG:boto:Token: None 2022-09-13T07:35:52.194 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.194 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fBPGexS1x/hbTIk9QtwVHoCgKRc= 2022-09-13T07:35:52.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fBPGexS1x/hbTIk9QtwVHoCgKRc=', 'Content-Length': '0'} 2022-09-13T07:35:52.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027e079fc655024b0-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.198 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.198 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.199 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.199 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.200 DEBUG:boto:Method: GET 2022-09-13T07:35:52.200 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.201 DEBUG:boto:Data: 2022-09-13T07:35:52.201 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.202 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.202 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.203 DEBUG:boto:Params: {} 2022-09-13T07:35:52.204 DEBUG:boto:Token: None 2022-09-13T07:35:52.204 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.205 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fBPGexS1x/hbTIk9QtwVHoCgKRc= 2022-09-13T07:35:52.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fBPGexS1x/hbTIk9QtwVHoCgKRc=', 'Content-Length': '0'} 2022-09-13T07:35:52.208 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9a0f8b807d1df60-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.209 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.209 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.210 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:52.210 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:52.211 DEBUG:boto:Method: GET 2022-09-13T07:35:52.211 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:52.212 DEBUG:boto:Data: 2022-09-13T07:35:52.212 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.213 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.214 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.214 DEBUG:boto:Params: {} 2022-09-13T07:35:52.215 DEBUG:boto:Token: None 2022-09-13T07:35:52.215 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.216 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk= 2022-09-13T07:35:52.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk=', 'Content-Length': '0'} 2022-09-13T07:35:52.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057b93d86a6093e9a-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.220 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.222 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.223 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.224 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:52.225 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:52.225 DEBUG:boto:Method: GET 2022-09-13T07:35:52.226 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:52.227 DEBUG:boto:Data: 2022-09-13T07:35:52.228 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.229 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.230 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.231 DEBUG:boto:Params: {} 2022-09-13T07:35:52.232 DEBUG:boto:Token: None 2022-09-13T07:35:52.233 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.234 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk= 2022-09-13T07:35:52.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk=', 'Content-Length': '0'} 2022-09-13T07:35:52.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033be8403f2a0ed87-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.240 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.241 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.242 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.243 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.244 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.244 DEBUG:boto:Data: 2022-09-13T07:35:52.245 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.245 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.246 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.246 DEBUG:boto:Params: {} 2022-09-13T07:35:52.247 DEBUG:boto:Token: None 2022-09-13T07:35:52.247 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.248 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h1v1SwZtw72K4Ld4tJI3/qhZDFI= 2022-09-13T07:35:52.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h1v1SwZtw72K4Ld4tJI3/qhZDFI=', 'Content-Length': '0'} 2022-09-13T07:35:52.251 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036a69c218e0f0189-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.251 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.252 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.252 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.253 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.253 DEBUG:boto:Data: 2022-09-13T07:35:52.254 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.254 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.255 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.255 DEBUG:boto:Params: {} 2022-09-13T07:35:52.256 DEBUG:boto:Token: None 2022-09-13T07:35:52.256 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.257 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h1v1SwZtw72K4Ld4tJI3/qhZDFI= 2022-09-13T07:35:52.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h1v1SwZtw72K4Ld4tJI3/qhZDFI=', 'Content-Length': '0'} 2022-09-13T07:35:52.260 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029fae11883b548f0-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:52.261 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.261 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.262 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.262 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.263 DEBUG:boto:Method: GET 2022-09-13T07:35:52.263 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.264 DEBUG:boto:Data: 2022-09-13T07:35:52.264 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.265 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.265 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.266 DEBUG:boto:Params: {} 2022-09-13T07:35:52.267 DEBUG:boto:Token: None 2022-09-13T07:35:52.267 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.268 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fBPGexS1x/hbTIk9QtwVHoCgKRc= 2022-09-13T07:35:52.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fBPGexS1x/hbTIk9QtwVHoCgKRc=', 'Content-Length': '0'} 2022-09-13T07:35:52.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7032ad765d9614b-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.318 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.319 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:52.319 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.320 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.321 DEBUG:boto:Method: GET 2022-09-13T07:35:52.321 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.322 DEBUG:boto:Data: 2022-09-13T07:35:52.322 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.323 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.323 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.324 DEBUG:boto:Params: {} 2022-09-13T07:35:52.325 DEBUG:boto:Token: None 2022-09-13T07:35:52.325 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:52.326 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fBPGexS1x/hbTIk9QtwVHoCgKRc= 2022-09-13T07:35:52.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fBPGexS1x/hbTIk9QtwVHoCgKRc=', 'Content-Length': '0'} 2022-09-13T07:35:52.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000978c148c7d043d1a-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.330 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.330 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.331 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:52.331 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:52.332 DEBUG:boto:Method: GET 2022-09-13T07:35:52.332 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:52.333 DEBUG:boto:Data: 2022-09-13T07:35:52.333 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.334 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.335 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.335 DEBUG:boto:Params: {} 2022-09-13T07:35:52.336 DEBUG:boto:Token: None 2022-09-13T07:35:52.337 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.338 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk= 2022-09-13T07:35:52.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk=', 'Content-Length': '0'} 2022-09-13T07:35:52.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029b5883df331ca00-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.343 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.344 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.344 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.345 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:52.345 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:52.346 DEBUG:boto:Method: GET 2022-09-13T07:35:52.346 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:52.347 DEBUG:boto:Data: 2022-09-13T07:35:52.348 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.348 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.349 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.349 DEBUG:boto:Params: {} 2022-09-13T07:35:52.350 DEBUG:boto:Token: None 2022-09-13T07:35:52.351 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.351 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk= 2022-09-13T07:35:52.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk=', 'Content-Length': '0'} 2022-09-13T07:35:52.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b534f0b3f87c7717-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.356 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:52.357 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.358 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.358 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.359 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.359 DEBUG:boto:Method: GET 2022-09-13T07:35:52.360 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.360 DEBUG:boto:Data: 2022-09-13T07:35:52.361 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.362 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.363 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.363 DEBUG:boto:Params: {} 2022-09-13T07:35:52.364 DEBUG:boto:Token: None 2022-09-13T07:35:52.364 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.365 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4QfydpoSy8/y0qCmKNNr/AG4Gxg= 2022-09-13T07:35:52.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4QfydpoSy8/y0qCmKNNr/AG4Gxg=', 'Content-Length': '0'} 2022-09-13T07:35:52.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003fae0ae8f7e526f7-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.369 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.370 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.371 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.371 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.372 DEBUG:boto:Method: GET 2022-09-13T07:35:52.372 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.373 DEBUG:boto:Data: 2022-09-13T07:35:52.373 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.374 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.374 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.375 DEBUG:boto:Params: {} 2022-09-13T07:35:52.375 DEBUG:boto:Token: None 2022-09-13T07:35:52.376 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.377 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4QfydpoSy8/y0qCmKNNr/AG4Gxg= 2022-09-13T07:35:52.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4QfydpoSy8/y0qCmKNNr/AG4Gxg=', 'Content-Length': '0'} 2022-09-13T07:35:52.380 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b594acdf9185d6e-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.422 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.423 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.424 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.424 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.425 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.425 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.426 DEBUG:boto:Data: 2022-09-13T07:35:52.426 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.427 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.427 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.427 DEBUG:boto:Params: {} 2022-09-13T07:35:52.428 DEBUG:boto:Token: None 2022-09-13T07:35:52.429 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.429 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ufA6XREjd85hT9qGFFU93aNnaPI= 2022-09-13T07:35:52.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ufA6XREjd85hT9qGFFU93aNnaPI=', 'Content-Length': '0'} 2022-09-13T07:35:52.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f228ecfaf13204e-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.433 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.433 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.434 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.434 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.435 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.435 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.435 DEBUG:boto:Data: 2022-09-13T07:35:52.436 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.436 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.437 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.437 DEBUG:boto:Params: {} 2022-09-13T07:35:52.438 DEBUG:boto:Token: None 2022-09-13T07:35:52.439 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.439 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ufA6XREjd85hT9qGFFU93aNnaPI= 2022-09-13T07:35:52.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ufA6XREjd85hT9qGFFU93aNnaPI=', 'Content-Length': '0'} 2022-09-13T07:35:52.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7d4c3a383a1081b-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:52.443 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.444 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.444 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.445 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.445 DEBUG:boto:Method: GET 2022-09-13T07:35:52.446 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.447 DEBUG:boto:Data: 2022-09-13T07:35:52.447 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.448 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.448 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.449 DEBUG:boto:Params: {} 2022-09-13T07:35:52.450 DEBUG:boto:Token: None 2022-09-13T07:35:52.450 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.451 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4QfydpoSy8/y0qCmKNNr/AG4Gxg= 2022-09-13T07:35:52.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4QfydpoSy8/y0qCmKNNr/AG4Gxg=', 'Content-Length': '0'} 2022-09-13T07:35:52.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c5e9fb05f134255-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.454 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.455 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.456 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.456 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.457 DEBUG:boto:Method: GET 2022-09-13T07:35:52.457 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.458 DEBUG:boto:Data: 2022-09-13T07:35:52.458 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.459 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.460 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.460 DEBUG:boto:Params: {} 2022-09-13T07:35:52.461 DEBUG:boto:Token: None 2022-09-13T07:35:52.461 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.462 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4QfydpoSy8/y0qCmKNNr/AG4Gxg= 2022-09-13T07:35:52.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4QfydpoSy8/y0qCmKNNr/AG4Gxg=', 'Content-Length': '0'} 2022-09-13T07:35:52.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f4d7fb794c222e3-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.465 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.466 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.467 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:52.467 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:52.468 DEBUG:boto:Method: GET 2022-09-13T07:35:52.468 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:52.469 DEBUG:boto:Data: 2022-09-13T07:35:52.469 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.470 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.470 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.471 DEBUG:boto:Params: {} 2022-09-13T07:35:52.471 DEBUG:boto:Token: None 2022-09-13T07:35:52.472 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.473 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk= 2022-09-13T07:35:52.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk=', 'Content-Length': '0'} 2022-09-13T07:35:52.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000760e85e98ebaf355-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.478 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.479 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.480 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.481 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:52.482 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:52.483 DEBUG:boto:Method: GET 2022-09-13T07:35:52.484 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:52.485 DEBUG:boto:Data: 2022-09-13T07:35:52.486 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.487 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.488 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.488 DEBUG:boto:Params: {} 2022-09-13T07:35:52.490 DEBUG:boto:Token: None 2022-09-13T07:35:52.491 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.492 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk= 2022-09-13T07:35:52.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk=', 'Content-Length': '0'} 2022-09-13T07:35:52.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000033301af069b4e84-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.497 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.499 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.500 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.501 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.501 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.502 DEBUG:boto:Data: 2022-09-13T07:35:52.503 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.504 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.505 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.506 DEBUG:boto:Params: {} 2022-09-13T07:35:52.507 DEBUG:boto:Token: None 2022-09-13T07:35:52.508 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.509 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rXjxcDC0Rz/YQ2NtKy91gMbcMgI= 2022-09-13T07:35:52.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rXjxcDC0Rz/YQ2NtKy91gMbcMgI=', 'Content-Length': '0'} 2022-09-13T07:35:52.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f89e0d76bea0943-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.514 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.515 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.516 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.516 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.517 DEBUG:boto:Data: 2022-09-13T07:35:52.517 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.518 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.518 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.519 DEBUG:boto:Params: {} 2022-09-13T07:35:52.519 DEBUG:boto:Token: None 2022-09-13T07:35:52.520 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.521 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rXjxcDC0Rz/YQ2NtKy91gMbcMgI= 2022-09-13T07:35:52.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rXjxcDC0Rz/YQ2NtKy91gMbcMgI=', 'Content-Length': '0'} 2022-09-13T07:35:52.524 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db1166f03ab84437-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:52.525 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.526 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.526 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.527 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.527 DEBUG:boto:Method: GET 2022-09-13T07:35:52.528 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.529 DEBUG:boto:Data: 2022-09-13T07:35:52.529 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.530 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.530 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.531 DEBUG:boto:Params: {} 2022-09-13T07:35:52.531 DEBUG:boto:Token: None 2022-09-13T07:35:52.532 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.532 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4QfydpoSy8/y0qCmKNNr/AG4Gxg= 2022-09-13T07:35:52.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4QfydpoSy8/y0qCmKNNr/AG4Gxg=', 'Content-Length': '0'} 2022-09-13T07:35:52.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000258544f08acda8cd-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.536 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.536 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:52.537 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.537 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.538 DEBUG:boto:Method: GET 2022-09-13T07:35:52.538 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.539 DEBUG:boto:Data: 2022-09-13T07:35:52.539 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.540 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.549 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.549 DEBUG:boto:Params: {} 2022-09-13T07:35:52.550 DEBUG:boto:Token: None 2022-09-13T07:35:52.551 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:52.551 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4QfydpoSy8/y0qCmKNNr/AG4Gxg= 2022-09-13T07:35:52.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4QfydpoSy8/y0qCmKNNr/AG4Gxg=', 'Content-Length': '0'} 2022-09-13T07:35:52.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004420761c4dd3c491-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.555 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.556 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.557 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:52.557 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:52.558 DEBUG:boto:Method: GET 2022-09-13T07:35:52.558 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:52.559 DEBUG:boto:Data: 2022-09-13T07:35:52.559 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.560 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.560 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.561 DEBUG:boto:Params: {} 2022-09-13T07:35:52.561 DEBUG:boto:Token: None 2022-09-13T07:35:52.562 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.563 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk= 2022-09-13T07:35:52.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk=', 'Content-Length': '0'} 2022-09-13T07:35:52.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000956f56802b752778-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.610 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.611 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.612 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.612 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:52.613 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:52.613 DEBUG:boto:Method: GET 2022-09-13T07:35:52.614 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:52.615 DEBUG:boto:Data: 2022-09-13T07:35:52.615 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.616 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:35:52.616 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.617 DEBUG:boto:Params: {} 2022-09-13T07:35:52.618 DEBUG:boto:Token: None 2022-09-13T07:35:52.618 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.619 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk= 2022-09-13T07:35:52.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk=', 'Content-Length': '0'} 2022-09-13T07:35:52.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7ec1df6bd997c5e-00632032d8-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.623 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.624 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-16 zones={test-zone3, test-zone1} 2022-09-13T07:35:52.624 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-16 zones={test-zone3, test-zone2} 2022-09-13T07:35:52.625 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.625 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.626 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.626 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:35:52.627 DEBUG:boto:Data: 2022-09-13T07:35:52.627 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.628 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.628 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.629 DEBUG:boto:Params: {} 2022-09-13T07:35:52.629 DEBUG:boto:Token: None 2022-09-13T07:35:52.630 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.631 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:76Ft5dmD1qzmOvjuq81T5IE/s1A= 2022-09-13T07:35:52.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:76Ft5dmD1qzmOvjuq81T5IE/s1A=', 'Content-Length': '0'} 2022-09-13T07:35:52.633 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047240deb838cd824-00632032d8-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.634 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.634 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.635 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.635 DEBUG:boto:Path: /yxvvuh-16/ 2022-09-13T07:35:52.636 DEBUG:boto:Data: 2022-09-13T07:35:52.636 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.637 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:52.637 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.638 DEBUG:boto:Params: {} 2022-09-13T07:35:52.639 DEBUG:boto:Token: None 2022-09-13T07:35:52.639 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.640 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:76Ft5dmD1qzmOvjuq81T5IE/s1A= 2022-09-13T07:35:52.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:76Ft5dmD1qzmOvjuq81T5IE/s1A=', 'Content-Length': '0'} 2022-09-13T07:35:52.643 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8864d5f906e0d29-00632032d8-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.643 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:35:52.644 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.644 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.645 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:52.645 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:52.646 DEBUG:boto:Method: GET 2022-09-13T07:35:52.646 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:52.647 DEBUG:boto:Data: 2022-09-13T07:35:52.647 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.648 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.648 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.649 DEBUG:boto:Params: {} 2022-09-13T07:35:52.650 DEBUG:boto:Token: None 2022-09-13T07:35:52.651 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/?versions 2022-09-13T07:35:52.651 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:223uIoSmvlzmSqwxxe28AD6KWfg= 2022-09-13T07:35:52.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:223uIoSmvlzmSqwxxe28AD6KWfg=', 'Content-Length': '0'} 2022-09-13T07:35:52.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ace9287ccb202809-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.655 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.656 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:35:52.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:35:52.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:35:52.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:35:52.658 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:35:52.658 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.659 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.659 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:52.660 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:52.660 DEBUG:boto:Method: GET 2022-09-13T07:35:52.661 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:52.661 DEBUG:boto:Data: 2022-09-13T07:35:52.662 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.662 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:52.663 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.663 DEBUG:boto:Params: {} 2022-09-13T07:35:52.664 DEBUG:boto:Token: None 2022-09-13T07:35:52.665 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/?versions 2022-09-13T07:35:52.665 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:223uIoSmvlzmSqwxxe28AD6KWfg= 2022-09-13T07:35:52.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:223uIoSmvlzmSqwxxe28AD6KWfg=', 'Content-Length': '0'} 2022-09-13T07:35:52.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000882b544e88ace168-00632032d8-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.669 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.670 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:35:52.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:35:52.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:35:52.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:35:52.672 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.672 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.673 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:52.673 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:52.674 DEBUG:boto:Method: GET 2022-09-13T07:35:52.674 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:52.675 DEBUG:boto:Data: 2022-09-13T07:35:52.675 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.676 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.676 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.677 DEBUG:boto:Params: {} 2022-09-13T07:35:52.677 DEBUG:boto:Token: None 2022-09-13T07:35:52.678 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/?versions 2022-09-13T07:35:52.679 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:223uIoSmvlzmSqwxxe28AD6KWfg= 2022-09-13T07:35:52.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:223uIoSmvlzmSqwxxe28AD6KWfg=', 'Content-Length': '0'} 2022-09-13T07:35:52.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005de94c088759bae1-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.682 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.683 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.684 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.684 DEBUG:boto:path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:52.685 DEBUG:boto:auth_path=/yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:52.685 DEBUG:boto:Method: GET 2022-09-13T07:35:52.686 DEBUG:boto:Path: /yxvvuh-16/?versions&max-keys=999 2022-09-13T07:35:52.686 DEBUG:boto:Data: 2022-09-13T07:35:52.687 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.687 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:52.688 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.688 DEBUG:boto:Params: {} 2022-09-13T07:35:52.689 DEBUG:boto:Token: None 2022-09-13T07:35:52.689 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/?versions 2022-09-13T07:35:52.690 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:223uIoSmvlzmSqwxxe28AD6KWfg= 2022-09-13T07:35:52.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:223uIoSmvlzmSqwxxe28AD6KWfg=', 'Content-Length': '0'} 2022-09-13T07:35:52.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007de2edec035a5396-00632032d8-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.694 DEBUG:boto:b'yxvvuh-16999falsedummy.etDTuFBPqw.uinGXDCTSfhS3trdxF.true2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ZGjwo1CvajWuedcOOmCGZL6b2cVNLEatrue2022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2WCOObBDl7.yF8-bmXrBJcSuKTYoGfVntrue2022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3WmcMv3ItEDGVe6luGjXTaZkbYfhv0qNtrue2022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:52.696 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:52.696 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:52.697 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.697 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.698 DEBUG:boto:Method: GET 2022-09-13T07:35:52.698 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.699 DEBUG:boto:Data: 2022-09-13T07:35:52.699 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.700 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.700 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.701 DEBUG:boto:Params: {} 2022-09-13T07:35:52.702 DEBUG:boto:Token: None 2022-09-13T07:35:52.702 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.703 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7IpoB97cC6eJy0Gbz+uXHvu5Fqw= 2022-09-13T07:35:52.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7IpoB97cC6eJy0Gbz+uXHvu5Fqw=', 'Content-Length': '0'} 2022-09-13T07:35:52.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eaa6c9352d300b25-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.707 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:52.707 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:52.708 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.708 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.709 DEBUG:boto:Method: GET 2022-09-13T07:35:52.709 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.710 DEBUG:boto:Data: 2022-09-13T07:35:52.710 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.711 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:52.711 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.712 DEBUG:boto:Params: {} 2022-09-13T07:35:52.713 DEBUG:boto:Token: None 2022-09-13T07:35:52.713 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.714 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7IpoB97cC6eJy0Gbz+uXHvu5Fqw= 2022-09-13T07:35:52.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7IpoB97cC6eJy0Gbz+uXHvu5Fqw=', 'Content-Length': '0'} 2022-09-13T07:35:52.716 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007cb3dd54964241ba-00632032d8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.717 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:52.718 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:52.718 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.719 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.719 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.720 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.720 DEBUG:boto:Data: 2022-09-13T07:35:52.721 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.722 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.722 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.723 DEBUG:boto:Params: {} 2022-09-13T07:35:52.723 DEBUG:boto:Token: None 2022-09-13T07:35:52.724 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.724 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qXOD24JgWTb/3FE5/+AW5gLONYU= 2022-09-13T07:35:52.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qXOD24JgWTb/3FE5/+AW5gLONYU=', 'Content-Length': '0'} 2022-09-13T07:35:52.727 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa7be39acd4beedf-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.727 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:52.728 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:52.729 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.729 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.730 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.730 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.731 DEBUG:boto:Data: 2022-09-13T07:35:52.731 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.732 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:52.733 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.734 DEBUG:boto:Params: {} 2022-09-13T07:35:52.735 DEBUG:boto:Token: None 2022-09-13T07:35:52.735 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.736 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qXOD24JgWTb/3FE5/+AW5gLONYU= 2022-09-13T07:35:52.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qXOD24JgWTb/3FE5/+AW5gLONYU=', 'Content-Length': '0'} 2022-09-13T07:35:52.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060bef2422d2b788e-00632032d8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:52.740 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:52.740 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:52.741 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.742 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.742 DEBUG:boto:Method: GET 2022-09-13T07:35:52.743 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.743 DEBUG:boto:Data: 2022-09-13T07:35:52.744 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.744 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.745 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.745 DEBUG:boto:Params: {} 2022-09-13T07:35:52.746 DEBUG:boto:Token: None 2022-09-13T07:35:52.747 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.748 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7IpoB97cC6eJy0Gbz+uXHvu5Fqw= 2022-09-13T07:35:52.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7IpoB97cC6eJy0Gbz+uXHvu5Fqw=', 'Content-Length': '0'} 2022-09-13T07:35:52.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc7cf2218c08846c-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.752 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:52.752 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:52.753 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.754 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.755 DEBUG:boto:Method: GET 2022-09-13T07:35:52.755 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.756 DEBUG:boto:Data: 2022-09-13T07:35:52.757 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.757 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:52.758 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.758 DEBUG:boto:Params: {} 2022-09-13T07:35:52.759 DEBUG:boto:Token: None 2022-09-13T07:35:52.759 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.760 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7IpoB97cC6eJy0Gbz+uXHvu5Fqw= 2022-09-13T07:35:52.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7IpoB97cC6eJy0Gbz+uXHvu5Fqw=', 'Content-Length': '0'} 2022-09-13T07:35:52.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bd8547fbfc5f926e-00632032d8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.764 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.764 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.765 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:52.766 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:52.766 DEBUG:boto:Method: GET 2022-09-13T07:35:52.767 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:35:52.767 DEBUG:boto:Data: 2022-09-13T07:35:52.768 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.769 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.769 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.770 DEBUG:boto:Params: {} 2022-09-13T07:35:52.770 DEBUG:boto:Token: None 2022-09-13T07:35:52.771 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.772 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk= 2022-09-13T07:35:52.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk=', 'Content-Length': '0'} 2022-09-13T07:35:52.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ebc5f6227451647-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.822 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.823 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.824 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.824 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:52.825 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:52.826 DEBUG:boto:Method: GET 2022-09-13T07:35:52.827 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:35:52.827 DEBUG:boto:Data: 2022-09-13T07:35:52.828 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.829 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:52.830 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.831 DEBUG:boto:Params: {} 2022-09-13T07:35:52.832 DEBUG:boto:Token: None 2022-09-13T07:35:52.832 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.833 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk= 2022-09-13T07:35:52.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk=', 'Content-Length': '0'} 2022-09-13T07:35:52.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fa97d1bc205dcff-00632032d8-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.838 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.839 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:52.840 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:52.841 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.841 DEBUG:boto:Path: /yxvvuh-16/dummy 2022-09-13T07:35:52.842 DEBUG:boto:Data: 2022-09-13T07:35:52.842 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.843 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.844 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.844 DEBUG:boto:Params: {} 2022-09-13T07:35:52.845 DEBUG:boto:Token: None 2022-09-13T07:35:52.845 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/dummy 2022-09-13T07:35:52.846 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:D5hkR5toM4Ddp7TJPKKWojtk0EE= 2022-09-13T07:35:52.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:D5hkR5toM4Ddp7TJPKKWojtk0EE=', 'Content-Length': '0'} 2022-09-13T07:35:52.849 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006478a91a496aded0-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.850 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:52.851 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:52.852 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.852 DEBUG:boto:Path: /yxvvuh-16/dummy 2022-09-13T07:35:52.853 DEBUG:boto:Data: 2022-09-13T07:35:52.853 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.854 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:52.854 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.855 DEBUG:boto:Params: {} 2022-09-13T07:35:52.855 DEBUG:boto:Token: None 2022-09-13T07:35:52.856 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/dummy 2022-09-13T07:35:52.857 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:D5hkR5toM4Ddp7TJPKKWojtk0EE= 2022-09-13T07:35:52.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:D5hkR5toM4Ddp7TJPKKWojtk0EE=', 'Content-Length': '0'} 2022-09-13T07:35:52.860 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009193d3ce3a20460f-00632032d8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:35:52.862 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:52.862 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:52.863 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.864 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.865 DEBUG:boto:Method: GET 2022-09-13T07:35:52.865 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.866 DEBUG:boto:Data: 2022-09-13T07:35:52.867 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.868 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.869 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.870 DEBUG:boto:Params: {} 2022-09-13T07:35:52.871 DEBUG:boto:Token: None 2022-09-13T07:35:52.872 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.873 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7IpoB97cC6eJy0Gbz+uXHvu5Fqw= 2022-09-13T07:35:52.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7IpoB97cC6eJy0Gbz+uXHvu5Fqw=', 'Content-Length': '0'} 2022-09-13T07:35:52.876 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cbd849de1a5555c0-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.877 DEBUG:boto:path=/yxvvuh-16/dummy 2022-09-13T07:35:52.877 DEBUG:boto:auth_path=/yxvvuh-16/dummy 2022-09-13T07:35:52.878 DEBUG:boto:path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.878 DEBUG:boto:auth_path=/yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.879 DEBUG:boto:Method: GET 2022-09-13T07:35:52.880 DEBUG:boto:Path: /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.880 DEBUG:boto:Data: 2022-09-13T07:35:52.881 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.882 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:52.882 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.883 DEBUG:boto:Params: {} 2022-09-13T07:35:52.884 DEBUG:boto:Token: None 2022-09-13T07:35:52.884 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/dummy?versionId=.etDTuFBPqw.uinGXDCTSfhS3trdxF. 2022-09-13T07:35:52.885 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7IpoB97cC6eJy0Gbz+uXHvu5Fqw= 2022-09-13T07:35:52.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7IpoB97cC6eJy0Gbz+uXHvu5Fqw=', 'Content-Length': '0'} 2022-09-13T07:35:52.888 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:10 GMT'), ('x-amz-version-id', '.etDTuFBPqw.uinGXDCTSfhS3trdxF.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000083ea32e479bff667-00632032d8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.889 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.890 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.891 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:52.893 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:52.894 DEBUG:boto:Method: GET 2022-09-13T07:35:52.895 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:35:52.896 DEBUG:boto:Data: 2022-09-13T07:35:52.897 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.898 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.899 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.899 DEBUG:boto:Params: {} 2022-09-13T07:35:52.900 DEBUG:boto:Token: None 2022-09-13T07:35:52.900 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.901 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk= 2022-09-13T07:35:52.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk=', 'Content-Length': '0'} 2022-09-13T07:35:52.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039d09ef377a54b7c-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.905 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.906 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.907 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.907 DEBUG:boto:path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:52.908 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=dummy 2022-09-13T07:35:52.908 DEBUG:boto:Method: GET 2022-09-13T07:35:52.909 DEBUG:boto:Path: /yxvvuh-16/?prefix=dummy 2022-09-13T07:35:52.909 DEBUG:boto:Data: 2022-09-13T07:35:52.910 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.910 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:52.911 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.911 DEBUG:boto:Params: {} 2022-09-13T07:35:52.912 DEBUG:boto:Token: None 2022-09-13T07:35:52.913 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.913 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk= 2022-09-13T07:35:52.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk=', 'Content-Length': '0'} 2022-09-13T07:35:52.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036e3fbca87c023a2-00632032d8-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.918 DEBUG:boto:b'yxvvuh-16dummy1000falsedummy2022-09-13T07:31:10.661Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:52.919 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:52.920 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:52.920 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.921 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.921 DEBUG:boto:Method: GET 2022-09-13T07:35:52.922 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.922 DEBUG:boto:Data: 2022-09-13T07:35:52.923 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.923 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.924 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.924 DEBUG:boto:Params: {} 2022-09-13T07:35:52.925 DEBUG:boto:Token: None 2022-09-13T07:35:52.925 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.926 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3XDGI10P61KrlXGA81obXWq/Zck= 2022-09-13T07:35:52.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3XDGI10P61KrlXGA81obXWq/Zck=', 'Content-Length': '0'} 2022-09-13T07:35:52.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000590b77f31bb3672c-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.929 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:52.930 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:52.931 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.931 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.932 DEBUG:boto:Method: GET 2022-09-13T07:35:52.932 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.933 DEBUG:boto:Data: 2022-09-13T07:35:52.933 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.934 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:52.934 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.935 DEBUG:boto:Params: {} 2022-09-13T07:35:52.935 DEBUG:boto:Token: None 2022-09-13T07:35:52.936 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.936 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3XDGI10P61KrlXGA81obXWq/Zck= 2022-09-13T07:35:52.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3XDGI10P61KrlXGA81obXWq/Zck=', 'Content-Length': '0'} 2022-09-13T07:35:52.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064e5a5702e0bd21c-00632032d8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.940 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:52.941 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:52.941 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.942 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.942 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.943 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.943 DEBUG:boto:Data: 2022-09-13T07:35:52.944 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.944 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.945 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.945 DEBUG:boto:Params: {} 2022-09-13T07:35:52.946 DEBUG:boto:Token: None 2022-09-13T07:35:52.946 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.947 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HxZkdjRiwUQr+cIUruERn+rMUdE= 2022-09-13T07:35:52.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HxZkdjRiwUQr+cIUruERn+rMUdE=', 'Content-Length': '0'} 2022-09-13T07:35:52.949 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b767dafea9189407-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.950 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:52.951 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:52.951 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.951 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.952 DEBUG:boto:Method: HEAD 2022-09-13T07:35:52.952 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.953 DEBUG:boto:Data: 2022-09-13T07:35:52.953 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.954 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:52.955 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.955 DEBUG:boto:Params: {} 2022-09-13T07:35:52.956 DEBUG:boto:Token: None 2022-09-13T07:35:52.956 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.957 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HxZkdjRiwUQr+cIUruERn+rMUdE= 2022-09-13T07:35:52.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HxZkdjRiwUQr+cIUruERn+rMUdE=', 'Content-Length': '0'} 2022-09-13T07:35:52.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a951228b762419b0-00632032d8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:52.961 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:52.961 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:52.962 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.962 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.963 DEBUG:boto:Method: GET 2022-09-13T07:35:52.963 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.964 DEBUG:boto:Data: 2022-09-13T07:35:52.964 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.965 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.965 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.966 DEBUG:boto:Params: {} 2022-09-13T07:35:52.967 DEBUG:boto:Token: None 2022-09-13T07:35:52.967 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.968 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3XDGI10P61KrlXGA81obXWq/Zck= 2022-09-13T07:35:52.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3XDGI10P61KrlXGA81obXWq/Zck=', 'Content-Length': '0'} 2022-09-13T07:35:52.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd360afc594131c2-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.971 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:52.971 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:52.972 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.973 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.973 DEBUG:boto:Method: GET 2022-09-13T07:35:52.974 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.974 DEBUG:boto:Data: 2022-09-13T07:35:52.975 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.975 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:52.976 DEBUG:boto:Port: 8000 2022-09-13T07:35:52.976 DEBUG:boto:Params: {} 2022-09-13T07:35:52.977 DEBUG:boto:Token: None 2022-09-13T07:35:52.977 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:52.978 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3XDGI10P61KrlXGA81obXWq/Zck= 2022-09-13T07:35:52.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3XDGI10P61KrlXGA81obXWq/Zck=', 'Content-Length': '0'} 2022-09-13T07:35:52.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063685c1fa3506dd0-00632032d8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.982 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.982 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.983 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:52.983 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:52.984 DEBUG:boto:Method: GET 2022-09-13T07:35:52.985 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:52.985 DEBUG:boto:Data: 2022-09-13T07:35:52.986 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.986 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:52.987 DEBUG:boto:Port: 8001 2022-09-13T07:35:52.987 DEBUG:boto:Params: {} 2022-09-13T07:35:52.988 DEBUG:boto:Token: None 2022-09-13T07:35:52.989 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:52 GMT /yxvvuh-16/ 2022-09-13T07:35:52.989 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk= 2022-09-13T07:35:52.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NpW/Pwkzm+uNsmu2Jhr037/6rbk=', 'Content-Length': '0'} 2022-09-13T07:35:52.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb39feaeec1a8c88-00632032d8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:52.993 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:52.994 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:52.995 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:52.995 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:52.996 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:52.996 DEBUG:boto:Method: GET 2022-09-13T07:35:52.997 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:52.997 DEBUG:boto:Data: 2022-09-13T07:35:52.998 DEBUG:boto:Headers: {} 2022-09-13T07:35:52.998 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:52.999 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.000 DEBUG:boto:Params: {} 2022-09-13T07:35:53.000 DEBUG:boto:Token: None 2022-09-13T07:35:53.001 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/ 2022-09-13T07:35:53.002 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0= 2022-09-13T07:35:53.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0=', 'Content-Length': '0'} 2022-09-13T07:35:53.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c37d67bfae634e8-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.006 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:53.007 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:53.007 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:53.008 DEBUG:boto:Method: HEAD 2022-09-13T07:35:53.008 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1 2022-09-13T07:35:53.009 DEBUG:boto:Data: 2022-09-13T07:35:53.009 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.010 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.011 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.011 DEBUG:boto:Params: {} 2022-09-13T07:35:53.012 DEBUG:boto:Token: None 2022-09-13T07:35:53.012 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone1 2022-09-13T07:35:53.013 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:o7fThjYg52IfGo6Kl8E2RbF8XT8= 2022-09-13T07:35:53.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:o7fThjYg52IfGo6Kl8E2RbF8XT8=', 'Content-Length': '0'} 2022-09-13T07:35:53.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dcb80a4bfa7b50db-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.016 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:53.017 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:53.018 DEBUG:boto:Method: HEAD 2022-09-13T07:35:53.018 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1 2022-09-13T07:35:53.019 DEBUG:boto:Data: 2022-09-13T07:35:53.020 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.020 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.020 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.021 DEBUG:boto:Params: {} 2022-09-13T07:35:53.021 DEBUG:boto:Token: None 2022-09-13T07:35:53.022 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone1 2022-09-13T07:35:53.023 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:o7fThjYg52IfGo6Kl8E2RbF8XT8= 2022-09-13T07:35:53.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:o7fThjYg52IfGo6Kl8E2RbF8XT8=', 'Content-Length': '0'} 2022-09-13T07:35:53.026 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e772d2fea94596d-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:35:53.027 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:53.028 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:53.028 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:53.029 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:53.029 DEBUG:boto:Method: GET 2022-09-13T07:35:53.030 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:53.030 DEBUG:boto:Data: 2022-09-13T07:35:53.031 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.031 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.032 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.032 DEBUG:boto:Params: {} 2022-09-13T07:35:53.033 DEBUG:boto:Token: None 2022-09-13T07:35:53.033 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:53.034 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cX3HHyglcIQF0Lh5b7wy46CbJwQ= 2022-09-13T07:35:53.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cX3HHyglcIQF0Lh5b7wy46CbJwQ=', 'Content-Length': '0'} 2022-09-13T07:35:53.037 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000308f6c966e071c1c-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.038 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:53.038 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1 2022-09-13T07:35:53.039 DEBUG:boto:path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:53.039 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:53.040 DEBUG:boto:Method: GET 2022-09-13T07:35:53.040 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:53.041 DEBUG:boto:Data: 2022-09-13T07:35:53.041 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.042 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.042 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.043 DEBUG:boto:Params: {} 2022-09-13T07:35:53.043 DEBUG:boto:Token: None 2022-09-13T07:35:53.044 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone1?versionId=ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa 2022-09-13T07:35:53.045 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cX3HHyglcIQF0Lh5b7wy46CbJwQ= 2022-09-13T07:35:53.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cX3HHyglcIQF0Lh5b7wy46CbJwQ=', 'Content-Length': '0'} 2022-09-13T07:35:53.047 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'ZGjwo1CvajWuedcOOmCGZL6b2cVNLEa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2c3bd23f66170ac-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.048 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:53.049 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:53.049 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:53.050 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:53.050 DEBUG:boto:Method: GET 2022-09-13T07:35:53.051 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:53.051 DEBUG:boto:Data: 2022-09-13T07:35:53.052 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.053 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.053 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.054 DEBUG:boto:Params: {} 2022-09-13T07:35:53.054 DEBUG:boto:Token: None 2022-09-13T07:35:53.055 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/ 2022-09-13T07:35:53.056 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0= 2022-09-13T07:35:53.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0=', 'Content-Length': '0'} 2022-09-13T07:35:53.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abd4635a5a3ec51a-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.060 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:53.060 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:53.061 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:53.062 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:53.062 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:53.063 DEBUG:boto:Method: GET 2022-09-13T07:35:53.063 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone1 2022-09-13T07:35:53.064 DEBUG:boto:Data: 2022-09-13T07:35:53.064 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.065 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.066 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.066 DEBUG:boto:Params: {} 2022-09-13T07:35:53.067 DEBUG:boto:Token: None 2022-09-13T07:35:53.068 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/ 2022-09-13T07:35:53.068 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0= 2022-09-13T07:35:53.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0=', 'Content-Length': '0'} 2022-09-13T07:35:53.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000412466a8751201c-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.073 DEBUG:boto:b'yxvvuh-16obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:53.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:53.074 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.075 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.075 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.076 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.076 DEBUG:boto:Method: GET 2022-09-13T07:35:53.077 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.077 DEBUG:boto:Data: 2022-09-13T07:35:53.078 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.078 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.079 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.080 DEBUG:boto:Params: {} 2022-09-13T07:35:53.080 DEBUG:boto:Token: None 2022-09-13T07:35:53.081 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.082 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WQioLR1GK+vWYx//ud4cxlIwCTg= 2022-09-13T07:35:53.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WQioLR1GK+vWYx//ud4cxlIwCTg=', 'Content-Length': '0'} 2022-09-13T07:35:53.084 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e8a1180badfdc54-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.126 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.127 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.127 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.128 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.128 DEBUG:boto:Method: GET 2022-09-13T07:35:53.129 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.129 DEBUG:boto:Data: 2022-09-13T07:35:53.130 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.130 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.131 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.131 DEBUG:boto:Params: {} 2022-09-13T07:35:53.132 DEBUG:boto:Token: None 2022-09-13T07:35:53.133 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.133 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WQioLR1GK+vWYx//ud4cxlIwCTg= 2022-09-13T07:35:53.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WQioLR1GK+vWYx//ud4cxlIwCTg=', 'Content-Length': '0'} 2022-09-13T07:35:53.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cec55656c03c9d15-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.137 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.138 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.138 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.139 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.139 DEBUG:boto:Method: HEAD 2022-09-13T07:35:53.140 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.140 DEBUG:boto:Data: 2022-09-13T07:35:53.141 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.141 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.142 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.143 DEBUG:boto:Params: {} 2022-09-13T07:35:53.144 DEBUG:boto:Token: None 2022-09-13T07:35:53.145 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.145 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iMBRnHz5yVJOjiqtt5DOZr2MgC0= 2022-09-13T07:35:53.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iMBRnHz5yVJOjiqtt5DOZr2MgC0=', 'Content-Length': '0'} 2022-09-13T07:35:53.148 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e40217a57ab8425a-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.149 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.149 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.150 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.150 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.151 DEBUG:boto:Method: HEAD 2022-09-13T07:35:53.151 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.152 DEBUG:boto:Data: 2022-09-13T07:35:53.152 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.153 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.153 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.154 DEBUG:boto:Params: {} 2022-09-13T07:35:53.155 DEBUG:boto:Token: None 2022-09-13T07:35:53.155 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.156 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iMBRnHz5yVJOjiqtt5DOZr2MgC0= 2022-09-13T07:35:53.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iMBRnHz5yVJOjiqtt5DOZr2MgC0=', 'Content-Length': '0'} 2022-09-13T07:35:53.159 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1c3cd9af4d337de-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:53.160 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.161 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.161 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.162 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.162 DEBUG:boto:Method: GET 2022-09-13T07:35:53.163 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.163 DEBUG:boto:Data: 2022-09-13T07:35:53.164 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.165 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.165 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.166 DEBUG:boto:Params: {} 2022-09-13T07:35:53.166 DEBUG:boto:Token: None 2022-09-13T07:35:53.167 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.167 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WQioLR1GK+vWYx//ud4cxlIwCTg= 2022-09-13T07:35:53.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WQioLR1GK+vWYx//ud4cxlIwCTg=', 'Content-Length': '0'} 2022-09-13T07:35:53.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df3b33c952a08a36-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.171 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.171 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.172 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.172 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.173 DEBUG:boto:Method: GET 2022-09-13T07:35:53.173 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.174 DEBUG:boto:Data: 2022-09-13T07:35:53.174 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.175 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.175 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.176 DEBUG:boto:Params: {} 2022-09-13T07:35:53.177 DEBUG:boto:Token: None 2022-09-13T07:35:53.177 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.178 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WQioLR1GK+vWYx//ud4cxlIwCTg= 2022-09-13T07:35:53.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WQioLR1GK+vWYx//ud4cxlIwCTg=', 'Content-Length': '0'} 2022-09-13T07:35:53.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042da799d1f63c98d-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.222 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:53.223 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:53.223 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:53.224 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:53.224 DEBUG:boto:Method: GET 2022-09-13T07:35:53.225 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:53.225 DEBUG:boto:Data: 2022-09-13T07:35:53.226 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.227 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.227 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.228 DEBUG:boto:Params: {} 2022-09-13T07:35:53.229 DEBUG:boto:Token: None 2022-09-13T07:35:53.229 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/ 2022-09-13T07:35:53.230 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0= 2022-09-13T07:35:53.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0=', 'Content-Length': '0'} 2022-09-13T07:35:53.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000208742761e1c3cc5-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.234 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:53.235 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:53.236 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:53.236 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:53.237 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:53.238 DEBUG:boto:Method: GET 2022-09-13T07:35:53.238 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:53.239 DEBUG:boto:Data: 2022-09-13T07:35:53.240 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.240 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.241 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.242 DEBUG:boto:Params: {} 2022-09-13T07:35:53.242 DEBUG:boto:Token: None 2022-09-13T07:35:53.243 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/ 2022-09-13T07:35:53.243 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0= 2022-09-13T07:35:53.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0=', 'Content-Length': '0'} 2022-09-13T07:35:53.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c7dec6f8c63b531-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.249 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:53.250 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.251 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.252 DEBUG:boto:Method: HEAD 2022-09-13T07:35:53.253 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.253 DEBUG:boto:Data: 2022-09-13T07:35:53.254 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.255 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.257 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.257 DEBUG:boto:Params: {} 2022-09-13T07:35:53.258 DEBUG:boto:Token: None 2022-09-13T07:35:53.258 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.259 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cH5ZX2Wj9TfX1VNvpHPDstCuWs0= 2022-09-13T07:35:53.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cH5ZX2Wj9TfX1VNvpHPDstCuWs0=', 'Content-Length': '0'} 2022-09-13T07:35:53.262 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059621e6c8befb324-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.262 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.263 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.264 DEBUG:boto:Method: HEAD 2022-09-13T07:35:53.264 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.265 DEBUG:boto:Data: 2022-09-13T07:35:53.265 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.266 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.266 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.267 DEBUG:boto:Params: {} 2022-09-13T07:35:53.267 DEBUG:boto:Token: None 2022-09-13T07:35:53.268 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.269 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cH5ZX2Wj9TfX1VNvpHPDstCuWs0= 2022-09-13T07:35:53.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cH5ZX2Wj9TfX1VNvpHPDstCuWs0=', 'Content-Length': '0'} 2022-09-13T07:35:53.272 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c426bfbe0755009c-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:35:53.273 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.274 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.274 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.275 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.275 DEBUG:boto:Method: GET 2022-09-13T07:35:53.276 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.276 DEBUG:boto:Data: 2022-09-13T07:35:53.277 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.277 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.278 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.279 DEBUG:boto:Params: {} 2022-09-13T07:35:53.279 DEBUG:boto:Token: None 2022-09-13T07:35:53.280 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.281 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WQioLR1GK+vWYx//ud4cxlIwCTg= 2022-09-13T07:35:53.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WQioLR1GK+vWYx//ud4cxlIwCTg=', 'Content-Length': '0'} 2022-09-13T07:35:53.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c89ee149cb391a04-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.284 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.284 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2 2022-09-13T07:35:53.285 DEBUG:boto:path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.285 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.286 DEBUG:boto:Method: GET 2022-09-13T07:35:53.287 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.287 DEBUG:boto:Data: 2022-09-13T07:35:53.288 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.288 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.289 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.289 DEBUG:boto:Params: {} 2022-09-13T07:35:53.290 DEBUG:boto:Token: None 2022-09-13T07:35:53.291 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone2?versionId=WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn 2022-09-13T07:35:53.291 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WQioLR1GK+vWYx//ud4cxlIwCTg= 2022-09-13T07:35:53.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WQioLR1GK+vWYx//ud4cxlIwCTg=', 'Content-Length': '0'} 2022-09-13T07:35:53.294 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WCOObBDl7.yF8-bmXrBJcSuKTYoGfVn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020de8aefe16d5cbd-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.295 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:53.296 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:53.296 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:53.297 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:53.297 DEBUG:boto:Method: GET 2022-09-13T07:35:53.298 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:53.298 DEBUG:boto:Data: 2022-09-13T07:35:53.299 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.299 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.300 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.301 DEBUG:boto:Params: {} 2022-09-13T07:35:53.302 DEBUG:boto:Token: None 2022-09-13T07:35:53.302 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/ 2022-09-13T07:35:53.303 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0= 2022-09-13T07:35:53.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0=', 'Content-Length': '0'} 2022-09-13T07:35:53.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035a4e273618b4a35-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.307 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:53.308 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:53.309 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:53.309 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:53.310 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:53.310 DEBUG:boto:Method: GET 2022-09-13T07:35:53.311 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone2 2022-09-13T07:35:53.312 DEBUG:boto:Data: 2022-09-13T07:35:53.312 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.313 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.313 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.314 DEBUG:boto:Params: {} 2022-09-13T07:35:53.314 DEBUG:boto:Token: None 2022-09-13T07:35:53.315 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/ 2022-09-13T07:35:53.315 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0= 2022-09-13T07:35:53.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0=', 'Content-Length': '0'} 2022-09-13T07:35:53.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a1d36f43f92c5ad-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.362 DEBUG:boto:b'yxvvuh-16obj-test-zone21000falseobj-test-zone22022-09-13T07:32:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:53.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:53.364 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.365 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.366 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.366 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.367 DEBUG:boto:Method: GET 2022-09-13T07:35:53.368 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.369 DEBUG:boto:Data: 2022-09-13T07:35:53.370 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.371 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.372 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.373 DEBUG:boto:Params: {} 2022-09-13T07:35:53.375 DEBUG:boto:Token: None 2022-09-13T07:35:53.376 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.377 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZoNtw9JvhCIu4qIphzf4h12wxZg= 2022-09-13T07:35:53.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZoNtw9JvhCIu4qIphzf4h12wxZg=', 'Content-Length': '0'} 2022-09-13T07:35:53.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000105850be6ad924fb-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.382 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.382 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.383 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.384 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.384 DEBUG:boto:Method: GET 2022-09-13T07:35:53.385 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.386 DEBUG:boto:Data: 2022-09-13T07:35:53.386 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.387 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.387 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.388 DEBUG:boto:Params: {} 2022-09-13T07:35:53.388 DEBUG:boto:Token: None 2022-09-13T07:35:53.389 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.390 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZoNtw9JvhCIu4qIphzf4h12wxZg= 2022-09-13T07:35:53.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZoNtw9JvhCIu4qIphzf4h12wxZg=', 'Content-Length': '0'} 2022-09-13T07:35:53.393 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046f411cf67aba3db-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.394 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.394 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.395 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.395 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.396 DEBUG:boto:Method: HEAD 2022-09-13T07:35:53.396 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.397 DEBUG:boto:Data: 2022-09-13T07:35:53.397 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.398 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.398 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.399 DEBUG:boto:Params: {} 2022-09-13T07:35:53.400 DEBUG:boto:Token: None 2022-09-13T07:35:53.400 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.401 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Panmb+heCC1yBeFof/9/ks3XTBM= 2022-09-13T07:35:53.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Panmb+heCC1yBeFof/9/ks3XTBM=', 'Content-Length': '0'} 2022-09-13T07:35:53.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5248fe59237d66d-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.405 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.405 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.406 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.406 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.407 DEBUG:boto:Method: HEAD 2022-09-13T07:35:53.407 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.408 DEBUG:boto:Data: 2022-09-13T07:35:53.408 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.409 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.410 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.410 DEBUG:boto:Params: {} 2022-09-13T07:35:53.411 DEBUG:boto:Token: None 2022-09-13T07:35:53.412 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.412 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Panmb+heCC1yBeFof/9/ks3XTBM= 2022-09-13T07:35:53.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Panmb+heCC1yBeFof/9/ks3XTBM=', 'Content-Length': '0'} 2022-09-13T07:35:53.415 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bde29f51db0cda4-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:53.416 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.417 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.417 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.418 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.419 DEBUG:boto:Method: GET 2022-09-13T07:35:53.419 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.420 DEBUG:boto:Data: 2022-09-13T07:35:53.420 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.421 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.421 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.422 DEBUG:boto:Params: {} 2022-09-13T07:35:53.423 DEBUG:boto:Token: None 2022-09-13T07:35:53.423 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.424 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZoNtw9JvhCIu4qIphzf4h12wxZg= 2022-09-13T07:35:53.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZoNtw9JvhCIu4qIphzf4h12wxZg=', 'Content-Length': '0'} 2022-09-13T07:35:53.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000becc6039bc55564-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.427 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.428 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.428 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.429 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.430 DEBUG:boto:Method: GET 2022-09-13T07:35:53.430 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.431 DEBUG:boto:Data: 2022-09-13T07:35:53.432 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.432 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.433 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.433 DEBUG:boto:Params: {} 2022-09-13T07:35:53.434 DEBUG:boto:Token: None 2022-09-13T07:35:53.434 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.435 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZoNtw9JvhCIu4qIphzf4h12wxZg= 2022-09-13T07:35:53.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZoNtw9JvhCIu4qIphzf4h12wxZg=', 'Content-Length': '0'} 2022-09-13T07:35:53.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058fb0d21b04b93b2-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.439 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:53.440 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:53.441 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:53.441 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:53.442 DEBUG:boto:Method: GET 2022-09-13T07:35:53.442 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:53.443 DEBUG:boto:Data: 2022-09-13T07:35:53.444 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.445 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.445 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.446 DEBUG:boto:Params: {} 2022-09-13T07:35:53.447 DEBUG:boto:Token: None 2022-09-13T07:35:53.447 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/ 2022-09-13T07:35:53.448 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0= 2022-09-13T07:35:53.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0=', 'Content-Length': '0'} 2022-09-13T07:35:53.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ebfd4040cf4052e-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.452 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:53.453 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:53.454 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:53.454 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:53.455 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:53.455 DEBUG:boto:Method: GET 2022-09-13T07:35:53.456 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:53.456 DEBUG:boto:Data: 2022-09-13T07:35:53.457 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.458 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.458 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.459 DEBUG:boto:Params: {} 2022-09-13T07:35:53.460 DEBUG:boto:Token: None 2022-09-13T07:35:53.460 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/ 2022-09-13T07:35:53.461 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0= 2022-09-13T07:35:53.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0=', 'Content-Length': '0'} 2022-09-13T07:35:53.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ae5364cf10b6615-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.506 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:53.507 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.507 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.508 DEBUG:boto:Method: HEAD 2022-09-13T07:35:53.509 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.509 DEBUG:boto:Data: 2022-09-13T07:35:53.510 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.510 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.511 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.511 DEBUG:boto:Params: {} 2022-09-13T07:35:53.512 DEBUG:boto:Token: None 2022-09-13T07:35:53.513 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.513 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XmFaMD35Zg88YiZ05sQi6vbKv+8= 2022-09-13T07:35:53.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XmFaMD35Zg88YiZ05sQi6vbKv+8=', 'Content-Length': '0'} 2022-09-13T07:35:53.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063476219b9d715f2-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.518 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.518 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.519 DEBUG:boto:Method: HEAD 2022-09-13T07:35:53.519 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.520 DEBUG:boto:Data: 2022-09-13T07:35:53.520 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.521 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.521 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.522 DEBUG:boto:Params: {} 2022-09-13T07:35:53.523 DEBUG:boto:Token: None 2022-09-13T07:35:53.523 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.524 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XmFaMD35Zg88YiZ05sQi6vbKv+8= 2022-09-13T07:35:53.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XmFaMD35Zg88YiZ05sQi6vbKv+8=', 'Content-Length': '0'} 2022-09-13T07:35:53.527 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c7d9695e05d9e6a-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:35:53.528 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.529 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.529 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.530 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.531 DEBUG:boto:Method: GET 2022-09-13T07:35:53.532 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.532 DEBUG:boto:Data: 2022-09-13T07:35:53.533 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.533 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.534 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.534 DEBUG:boto:Params: {} 2022-09-13T07:35:53.535 DEBUG:boto:Token: None 2022-09-13T07:35:53.535 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.536 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZoNtw9JvhCIu4qIphzf4h12wxZg= 2022-09-13T07:35:53.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZoNtw9JvhCIu4qIphzf4h12wxZg=', 'Content-Length': '0'} 2022-09-13T07:35:53.539 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a749a9810d0abb12-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.540 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.540 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3 2022-09-13T07:35:53.541 DEBUG:boto:path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.541 DEBUG:boto:auth_path=/yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.542 DEBUG:boto:Method: GET 2022-09-13T07:35:53.542 DEBUG:boto:Path: /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.543 DEBUG:boto:Data: 2022-09-13T07:35:53.543 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.544 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.544 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.545 DEBUG:boto:Params: {} 2022-09-13T07:35:53.545 DEBUG:boto:Token: None 2022-09-13T07:35:53.546 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/obj-test-zone3?versionId=WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN 2022-09-13T07:35:53.546 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZoNtw9JvhCIu4qIphzf4h12wxZg= 2022-09-13T07:35:53.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZoNtw9JvhCIu4qIphzf4h12wxZg=', 'Content-Length': '0'} 2022-09-13T07:35:53.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'WmcMv3ItEDGVe6luGjXTaZkbYfhv0qN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000045f97042de8f94a-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.550 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:53.550 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:53.551 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:53.552 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:53.553 DEBUG:boto:Method: GET 2022-09-13T07:35:53.553 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:53.554 DEBUG:boto:Data: 2022-09-13T07:35:53.555 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.555 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:35:53.556 DEBUG:boto:Port: 8001 2022-09-13T07:35:53.556 DEBUG:boto:Params: {} 2022-09-13T07:35:53.557 DEBUG:boto:Token: None 2022-09-13T07:35:53.557 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/ 2022-09-13T07:35:53.558 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0= 2022-09-13T07:35:53.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0=', 'Content-Length': '0'} 2022-09-13T07:35:53.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbdde03157dff74b-00632032d9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.602 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:53.603 DEBUG:boto:path=/yxvvuh-16/ 2022-09-13T07:35:53.604 DEBUG:boto:auth_path=/yxvvuh-16/ 2022-09-13T07:35:53.604 DEBUG:boto:path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:53.605 DEBUG:boto:auth_path=/yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:53.605 DEBUG:boto:Method: GET 2022-09-13T07:35:53.606 DEBUG:boto:Path: /yxvvuh-16/?prefix=obj-test-zone3 2022-09-13T07:35:53.606 DEBUG:boto:Data: 2022-09-13T07:35:53.607 DEBUG:boto:Headers: {} 2022-09-13T07:35:53.607 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:35:53.608 DEBUG:boto:Port: 8000 2022-09-13T07:35:53.608 DEBUG:boto:Params: {} 2022-09-13T07:35:53.609 DEBUG:boto:Token: None 2022-09-13T07:35:53.609 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:35:53 GMT /yxvvuh-16/ 2022-09-13T07:35:53.610 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0= 2022-09-13T07:35:53.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:35:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1AB8qzRf3qQz5ArhgnGq08IqcE0=', 'Content-Length': '0'} 2022-09-13T07:35:53.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e41b8aea880fe366-00632032d9-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:35:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:35:53.614 DEBUG:boto:b'yxvvuh-16obj-test-zone31000falseobj-test-zone32022-09-13T07:32:06.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:35:53.614 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-16 zones={test-zone3, test-zone2} 2022-09-13T07:35:53.615 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-17 2022-09-13T07:35:53.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:53.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:53.617 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:35:53.680 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:35:53.681 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:35:53.718 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:35:53.714+0000 7f6a524cd600 1 bucket sync caught up with source: 2022-09-13T07:35:53.719 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000011.32.5, , 00000000004.34.5, , 00000000008.21.5, , , , , ] 2022-09-13T07:35:53.719 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000011.32.5, , 00000000004.34.5, , 00000000008.21.5, , , , , ] 2022-09-13T07:35:53.719 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:35:53.714+0000 7f6a524cd600 0 bucket checkpoint complete 2022-09-13T07:35:53.733 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-17 2022-09-13T07:35:53.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:53.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:53.735 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:35:53.794 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:35:53.794 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:35:53.842 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:53.839+0000 7fbac3238600 1 bucket sync caught up with source: 2022-09-13T07:35:53.843 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000011.32.5, , 00000000004.34.5, , 00000000008.21.5, , , , , ] 2022-09-13T07:35:53.843 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000011.32.5, , 00000000004.34.5, , 00000000008.21.5, , , , , ] 2022-09-13T07:35:53.843 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:53.839+0000 7fbac3238600 0 bucket checkpoint complete 2022-09-13T07:35:53.866 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-17 2022-09-13T07:35:53.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:53.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:35:53.868 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:35:53.926 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:35:53.926 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:35:53.974 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:53.971+0000 7f4aa2194600 1 waiting for incremental sync to catch up: 2022-09-13T07:35:53.974 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.45.5, , 00000000010.34.5, , 00000000008.33.9, , , , , ] 2022-09-13T07:35:53.974 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.45.5, , 00000000011.43.5, , 00000000009.36.5, , , , , ] 2022-09-13T07:35:58.975 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:35:58.972+0000 7f4aa2194600 1 waiting for incremental sync to catch up: 2022-09-13T07:35:58.976 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.45.5, , 00000000010.34.5, , 00000000008.33.9, , , , , ] 2022-09-13T07:35:58.976 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.45.5, , 00000000011.43.5, , 00000000009.36.5, , , , , ] 2022-09-13T07:36:03.976 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:03.973+0000 7f4aa2194600 1 waiting for incremental sync to catch up: 2022-09-13T07:36:03.977 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.45.5, , 00000000010.34.5, , 00000000008.33.9, , , , , ] 2022-09-13T07:36:03.977 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.45.5, , 00000000011.43.5, , 00000000009.36.5, , , , , ] 2022-09-13T07:36:08.978 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:08.974+0000 7f4aa2194600 1 waiting for incremental sync to catch up: 2022-09-13T07:36:08.978 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.45.5, , 00000000010.34.5, , 00000000009.36.5, , , , , ] 2022-09-13T07:36:08.979 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.45.5, , 00000000011.43.5, , 00000000009.36.5, , , , , ] 2022-09-13T07:36:13.979 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:13.976+0000 7f4aa2194600 1 waiting for incremental sync to catch up: 2022-09-13T07:36:13.980 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.45.5, , 00000000010.34.5, , 00000000009.36.5, , , , , ] 2022-09-13T07:36:13.980 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.45.5, , 00000000011.43.5, , 00000000009.36.5, , , , , ] 2022-09-13T07:36:18.981 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:18.977+0000 7f4aa2194600 1 waiting for incremental sync to catch up: 2022-09-13T07:36:18.981 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.45.5, , 00000000010.34.5, , 00000000009.36.5, , , , , ] 2022-09-13T07:36:18.982 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.45.5, , 00000000011.43.5, , 00000000009.36.5, , , , , ] 2022-09-13T07:36:23.982 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:23.978+0000 7f4aa2194600 1 waiting for incremental sync to catch up: 2022-09-13T07:36:23.983 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.45.5, , 00000000010.34.5, , 00000000009.36.5, , , , , ] 2022-09-13T07:36:23.983 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.45.5, , 00000000011.43.5, , 00000000009.36.5, , , , , ] 2022-09-13T07:36:28.984 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:28.980+0000 7f4aa2194600 1 bucket sync caught up with source: 2022-09-13T07:36:28.984 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.45.5, , 00000000011.43.5, , 00000000009.36.5, , , , , ] 2022-09-13T07:36:28.985 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.45.5, , 00000000011.43.5, , 00000000009.36.5, , , , , ] 2022-09-13T07:36:28.985 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:28.980+0000 7f4aa2194600 0 bucket checkpoint complete 2022-09-13T07:36:29.004 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-17 2022-09-13T07:36:29.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:29.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:29.007 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:36:29.071 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:36:29.072 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:36:29.128 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:29.124+0000 7fd2b177f600 1 bucket sync caught up with source: 2022-09-13T07:36:29.129 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.45.5, , 00000000011.43.5, , 00000000009.36.5, , , , , ] 2022-09-13T07:36:29.129 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.45.5, , 00000000011.43.5, , 00000000009.36.5, , , , , ] 2022-09-13T07:36:29.129 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:29.124+0000 7fd2b177f600 0 bucket checkpoint complete 2022-09-13T07:36:29.144 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-17 2022-09-13T07:36:29.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:29.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:29.146 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:36:29.204 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:36:29.204 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:36:29.252 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:29.248+0000 7f51792e5600 1 bucket sync caught up with source: 2022-09-13T07:36:29.253 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.24.5, , 00000000004.26.5, , 00000000015.59.5, , , , , ] 2022-09-13T07:36:29.253 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.24.5, , 00000000004.26.5, , 00000000015.59.5, , , , , ] 2022-09-13T07:36:29.253 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:29.248+0000 7f51792e5600 0 bucket checkpoint complete 2022-09-13T07:36:29.265 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-17 2022-09-13T07:36:29.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:29.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:29.267 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:36:29.330 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:36:29.331 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:36:29.385 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:36:29.381+0000 7f4c79be7600 1 bucket sync caught up with source: 2022-09-13T07:36:29.385 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000004.24.5, , 00000000004.26.5, , 00000000015.59.5, , , , , ] 2022-09-13T07:36:29.386 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000004.24.5, , 00000000004.26.5, , 00000000015.59.5, , , , , ] 2022-09-13T07:36:29.386 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:36:29.381+0000 7f4c79be7600 0 bucket checkpoint complete 2022-09-13T07:36:29.409 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-17 zones={test-zone2, test-zone1} 2022-09-13T07:36:29.411 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:29.412 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:29.412 DEBUG:boto:Method: HEAD 2022-09-13T07:36:29.413 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:36:29.414 DEBUG:boto:Data: 2022-09-13T07:36:29.415 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.416 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.417 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.418 DEBUG:boto:Params: {} 2022-09-13T07:36:29.419 DEBUG:boto:Token: None 2022-09-13T07:36:29.420 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/ 2022-09-13T07:36:29.421 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NKm6KRpQHtNUho67d08mOZlMygM= 2022-09-13T07:36:29.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NKm6KRpQHtNUho67d08mOZlMygM=', 'Content-Length': '0'} 2022-09-13T07:36:29.425 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056a0db84dad5e2e6-00632032fd-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.426 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:29.427 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:29.427 DEBUG:boto:Method: HEAD 2022-09-13T07:36:29.428 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:36:29.428 DEBUG:boto:Data: 2022-09-13T07:36:29.428 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.429 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.429 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.430 DEBUG:boto:Params: {} 2022-09-13T07:36:29.431 DEBUG:boto:Token: None 2022-09-13T07:36:29.431 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/ 2022-09-13T07:36:29.432 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NKm6KRpQHtNUho67d08mOZlMygM= 2022-09-13T07:36:29.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NKm6KRpQHtNUho67d08mOZlMygM=', 'Content-Length': '0'} 2022-09-13T07:36:29.435 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000549651635f1ace9f-00632032fd-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.436 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:36:29.437 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:29.438 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:29.439 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:29.439 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:29.440 DEBUG:boto:Method: GET 2022-09-13T07:36:29.441 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:29.442 DEBUG:boto:Data: 2022-09-13T07:36:29.443 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.443 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.444 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.445 DEBUG:boto:Params: {} 2022-09-13T07:36:29.446 DEBUG:boto:Token: None 2022-09-13T07:36:29.447 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/?versions 2022-09-13T07:36:29.447 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j0fwo0n7lRWaVawpPeP2y59DtSI= 2022-09-13T07:36:29.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j0fwo0n7lRWaVawpPeP2y59DtSI=', 'Content-Length': '0'} 2022-09-13T07:36:29.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000105c42c670d7334f-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.451 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:29.452 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:36:29.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:36:29.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:36:29.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:36:29.454 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:36:29.455 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:29.455 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:29.456 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:29.456 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:29.457 DEBUG:boto:Method: GET 2022-09-13T07:36:29.457 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:29.458 DEBUG:boto:Data: 2022-09-13T07:36:29.458 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.459 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.459 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.460 DEBUG:boto:Params: {} 2022-09-13T07:36:29.460 DEBUG:boto:Token: None 2022-09-13T07:36:29.461 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/?versions 2022-09-13T07:36:29.462 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j0fwo0n7lRWaVawpPeP2y59DtSI= 2022-09-13T07:36:29.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j0fwo0n7lRWaVawpPeP2y59DtSI=', 'Content-Length': '0'} 2022-09-13T07:36:29.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f2e26c4a12f6c2c-00632032fd-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.465 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:29.466 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:36:29.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:36:29.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:36:29.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:36:29.468 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:29.469 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:29.469 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:29.470 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:29.470 DEBUG:boto:Method: GET 2022-09-13T07:36:29.471 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:29.471 DEBUG:boto:Data: 2022-09-13T07:36:29.472 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.472 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.473 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.473 DEBUG:boto:Params: {} 2022-09-13T07:36:29.474 DEBUG:boto:Token: None 2022-09-13T07:36:29.474 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/?versions 2022-09-13T07:36:29.475 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j0fwo0n7lRWaVawpPeP2y59DtSI= 2022-09-13T07:36:29.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j0fwo0n7lRWaVawpPeP2y59DtSI=', 'Content-Length': '0'} 2022-09-13T07:36:29.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ff2879a5780ab2c-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.478 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:29.479 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:29.480 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:29.480 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:29.481 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:29.481 DEBUG:boto:Method: GET 2022-09-13T07:36:29.482 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:29.483 DEBUG:boto:Data: 2022-09-13T07:36:29.483 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.484 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.484 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.484 DEBUG:boto:Params: {} 2022-09-13T07:36:29.485 DEBUG:boto:Token: None 2022-09-13T07:36:29.486 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/?versions 2022-09-13T07:36:29.486 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j0fwo0n7lRWaVawpPeP2y59DtSI= 2022-09-13T07:36:29.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j0fwo0n7lRWaVawpPeP2y59DtSI=', 'Content-Length': '0'} 2022-09-13T07:36:29.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e554a4cd9ad08945-00632032fd-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.490 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:29.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:29.492 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:29.492 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:29.493 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.493 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.494 DEBUG:boto:Method: GET 2022-09-13T07:36:29.495 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.495 DEBUG:boto:Data: 2022-09-13T07:36:29.496 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.496 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.497 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.497 DEBUG:boto:Params: {} 2022-09-13T07:36:29.498 DEBUG:boto:Token: None 2022-09-13T07:36:29.498 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.499 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F6fiUoqjdVyqPmNZYn8+FOFXCsE= 2022-09-13T07:36:29.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F6fiUoqjdVyqPmNZYn8+FOFXCsE=', 'Content-Length': '0'} 2022-09-13T07:36:29.501 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dcb4a28dbb6919ec-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.502 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:29.503 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:29.503 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.504 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.504 DEBUG:boto:Method: GET 2022-09-13T07:36:29.505 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.505 DEBUG:boto:Data: 2022-09-13T07:36:29.506 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.506 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.507 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.507 DEBUG:boto:Params: {} 2022-09-13T07:36:29.508 DEBUG:boto:Token: None 2022-09-13T07:36:29.509 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.509 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F6fiUoqjdVyqPmNZYn8+FOFXCsE= 2022-09-13T07:36:29.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F6fiUoqjdVyqPmNZYn8+FOFXCsE=', 'Content-Length': '0'} 2022-09-13T07:36:29.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d79292f05192f8a1-00632032fd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.513 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:29.513 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:29.514 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.514 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.515 DEBUG:boto:Method: HEAD 2022-09-13T07:36:29.515 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.516 DEBUG:boto:Data: 2022-09-13T07:36:29.516 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.517 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.517 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.518 DEBUG:boto:Params: {} 2022-09-13T07:36:29.518 DEBUG:boto:Token: None 2022-09-13T07:36:29.519 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.519 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zIZbqmCEq6PmIovRpzvHn8mRkLQ= 2022-09-13T07:36:29.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zIZbqmCEq6PmIovRpzvHn8mRkLQ=', 'Content-Length': '0'} 2022-09-13T07:36:29.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eeb69dca738c0307-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.523 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:29.523 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:29.524 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.524 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.525 DEBUG:boto:Method: HEAD 2022-09-13T07:36:29.525 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.526 DEBUG:boto:Data: 2022-09-13T07:36:29.526 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.527 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.527 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.528 DEBUG:boto:Params: {} 2022-09-13T07:36:29.528 DEBUG:boto:Token: None 2022-09-13T07:36:29.529 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.530 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zIZbqmCEq6PmIovRpzvHn8mRkLQ= 2022-09-13T07:36:29.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zIZbqmCEq6PmIovRpzvHn8mRkLQ=', 'Content-Length': '0'} 2022-09-13T07:36:29.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f0bb4be6edfeeed6-00632032fd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:29.534 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:29.534 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:29.535 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.535 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.536 DEBUG:boto:Method: GET 2022-09-13T07:36:29.536 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.537 DEBUG:boto:Data: 2022-09-13T07:36:29.537 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.538 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.538 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.539 DEBUG:boto:Params: {} 2022-09-13T07:36:29.539 DEBUG:boto:Token: None 2022-09-13T07:36:29.540 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.541 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F6fiUoqjdVyqPmNZYn8+FOFXCsE= 2022-09-13T07:36:29.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F6fiUoqjdVyqPmNZYn8+FOFXCsE=', 'Content-Length': '0'} 2022-09-13T07:36:29.543 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d5f30024b15465a4-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.544 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:29.544 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:29.545 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.545 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.546 DEBUG:boto:Method: GET 2022-09-13T07:36:29.546 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.547 DEBUG:boto:Data: 2022-09-13T07:36:29.548 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.548 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.549 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.549 DEBUG:boto:Params: {} 2022-09-13T07:36:29.550 DEBUG:boto:Token: None 2022-09-13T07:36:29.550 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.551 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F6fiUoqjdVyqPmNZYn8+FOFXCsE= 2022-09-13T07:36:29.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F6fiUoqjdVyqPmNZYn8+FOFXCsE=', 'Content-Length': '0'} 2022-09-13T07:36:29.553 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000faaa4bac99f73a43-00632032fd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.554 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:29.555 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:29.555 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:29.556 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:29.556 DEBUG:boto:Method: GET 2022-09-13T07:36:29.557 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:36:29.557 DEBUG:boto:Data: 2022-09-13T07:36:29.558 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.558 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.559 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.559 DEBUG:boto:Params: {} 2022-09-13T07:36:29.560 DEBUG:boto:Token: None 2022-09-13T07:36:29.561 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/ 2022-09-13T07:36:29.561 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HXr9m5PIHfLbybYL+AayokRREds= 2022-09-13T07:36:29.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HXr9m5PIHfLbybYL+AayokRREds=', 'Content-Length': '0'} 2022-09-13T07:36:29.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4460f3b560f5456-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.606 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:29.607 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:29.608 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:29.608 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:29.609 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:29.609 DEBUG:boto:Method: GET 2022-09-13T07:36:29.610 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:36:29.611 DEBUG:boto:Data: 2022-09-13T07:36:29.611 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.612 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.612 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.613 DEBUG:boto:Params: {} 2022-09-13T07:36:29.613 DEBUG:boto:Token: None 2022-09-13T07:36:29.614 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/ 2022-09-13T07:36:29.614 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HXr9m5PIHfLbybYL+AayokRREds= 2022-09-13T07:36:29.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HXr9m5PIHfLbybYL+AayokRREds=', 'Content-Length': '0'} 2022-09-13T07:36:29.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035ce842c3b55b4da-00632032fd-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.619 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:29.619 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:29.620 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:29.620 DEBUG:boto:Method: HEAD 2022-09-13T07:36:29.621 DEBUG:boto:Path: /yxvvuh-17/dummy 2022-09-13T07:36:29.621 DEBUG:boto:Data: 2022-09-13T07:36:29.622 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.622 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.623 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.623 DEBUG:boto:Params: {} 2022-09-13T07:36:29.624 DEBUG:boto:Token: None 2022-09-13T07:36:29.624 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/dummy 2022-09-13T07:36:29.625 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:v8Ujx6O6wEIdmcUxP18sMjQ5e3s= 2022-09-13T07:36:29.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:v8Ujx6O6wEIdmcUxP18sMjQ5e3s=', 'Content-Length': '0'} 2022-09-13T07:36:29.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c0c7999fdd7601f-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.629 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:29.629 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:29.630 DEBUG:boto:Method: HEAD 2022-09-13T07:36:29.630 DEBUG:boto:Path: /yxvvuh-17/dummy 2022-09-13T07:36:29.631 DEBUG:boto:Data: 2022-09-13T07:36:29.631 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.632 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.632 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.633 DEBUG:boto:Params: {} 2022-09-13T07:36:29.633 DEBUG:boto:Token: None 2022-09-13T07:36:29.634 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/dummy 2022-09-13T07:36:29.635 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:v8Ujx6O6wEIdmcUxP18sMjQ5e3s= 2022-09-13T07:36:29.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:v8Ujx6O6wEIdmcUxP18sMjQ5e3s=', 'Content-Length': '0'} 2022-09-13T07:36:29.637 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000052a7a62bae7fa673-00632032fd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:29.639 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:29.639 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:29.640 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.640 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.641 DEBUG:boto:Method: GET 2022-09-13T07:36:29.641 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.642 DEBUG:boto:Data: 2022-09-13T07:36:29.642 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.643 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.643 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.644 DEBUG:boto:Params: {} 2022-09-13T07:36:29.644 DEBUG:boto:Token: None 2022-09-13T07:36:29.645 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.646 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F6fiUoqjdVyqPmNZYn8+FOFXCsE= 2022-09-13T07:36:29.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F6fiUoqjdVyqPmNZYn8+FOFXCsE=', 'Content-Length': '0'} 2022-09-13T07:36:29.648 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c24ee46709b21292-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.649 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:29.649 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:29.650 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.650 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.651 DEBUG:boto:Method: GET 2022-09-13T07:36:29.651 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.652 DEBUG:boto:Data: 2022-09-13T07:36:29.653 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.653 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.654 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.654 DEBUG:boto:Params: {} 2022-09-13T07:36:29.655 DEBUG:boto:Token: None 2022-09-13T07:36:29.655 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:29.656 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F6fiUoqjdVyqPmNZYn8+FOFXCsE= 2022-09-13T07:36:29.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F6fiUoqjdVyqPmNZYn8+FOFXCsE=', 'Content-Length': '0'} 2022-09-13T07:36:29.658 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c5c1171ef8a8e74b-00632032fd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.659 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:29.660 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:29.661 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:29.661 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:29.662 DEBUG:boto:Method: GET 2022-09-13T07:36:29.662 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:36:29.663 DEBUG:boto:Data: 2022-09-13T07:36:29.664 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.664 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.665 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.665 DEBUG:boto:Params: {} 2022-09-13T07:36:29.666 DEBUG:boto:Token: None 2022-09-13T07:36:29.666 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/ 2022-09-13T07:36:29.667 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HXr9m5PIHfLbybYL+AayokRREds= 2022-09-13T07:36:29.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HXr9m5PIHfLbybYL+AayokRREds=', 'Content-Length': '0'} 2022-09-13T07:36:29.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdc662495fa7dc30-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.671 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:29.672 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:29.673 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:29.673 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:29.674 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:29.674 DEBUG:boto:Method: GET 2022-09-13T07:36:29.675 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:36:29.675 DEBUG:boto:Data: 2022-09-13T07:36:29.676 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.676 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.677 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.677 DEBUG:boto:Params: {} 2022-09-13T07:36:29.678 DEBUG:boto:Token: None 2022-09-13T07:36:29.678 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/ 2022-09-13T07:36:29.679 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HXr9m5PIHfLbybYL+AayokRREds= 2022-09-13T07:36:29.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HXr9m5PIHfLbybYL+AayokRREds=', 'Content-Length': '0'} 2022-09-13T07:36:29.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065a9fa9ade6130ae-00632032fd-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.683 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:29.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:29.684 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.685 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.685 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.686 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.686 DEBUG:boto:Method: GET 2022-09-13T07:36:29.687 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.687 DEBUG:boto:Data: 2022-09-13T07:36:29.688 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.688 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.689 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.689 DEBUG:boto:Params: {} 2022-09-13T07:36:29.690 DEBUG:boto:Token: None 2022-09-13T07:36:29.691 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.691 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S8GFq+d0Mg7deOAas5G24IEZ+Jo= 2022-09-13T07:36:29.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S8GFq+d0Mg7deOAas5G24IEZ+Jo=', 'Content-Length': '0'} 2022-09-13T07:36:29.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002cb0a48d4f44aa8a-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.695 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.695 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.696 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.696 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.697 DEBUG:boto:Method: GET 2022-09-13T07:36:29.698 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.698 DEBUG:boto:Data: 2022-09-13T07:36:29.699 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.699 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.700 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.700 DEBUG:boto:Params: {} 2022-09-13T07:36:29.701 DEBUG:boto:Token: None 2022-09-13T07:36:29.701 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.702 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S8GFq+d0Mg7deOAas5G24IEZ+Jo= 2022-09-13T07:36:29.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S8GFq+d0Mg7deOAas5G24IEZ+Jo=', 'Content-Length': '0'} 2022-09-13T07:36:29.704 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006645372b7b3ee638-00632032fd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.746 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.747 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.747 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.748 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.748 DEBUG:boto:Method: HEAD 2022-09-13T07:36:29.749 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.749 DEBUG:boto:Data: 2022-09-13T07:36:29.750 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.751 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.752 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.752 DEBUG:boto:Params: {} 2022-09-13T07:36:29.753 DEBUG:boto:Token: None 2022-09-13T07:36:29.753 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.754 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rA405uupNJgA8UmgVcNw4V+wqTY= 2022-09-13T07:36:29.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rA405uupNJgA8UmgVcNw4V+wqTY=', 'Content-Length': '0'} 2022-09-13T07:36:29.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e24ded1fcae163de-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.767 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.768 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.769 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.770 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.771 DEBUG:boto:Method: HEAD 2022-09-13T07:36:29.771 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.772 DEBUG:boto:Data: 2022-09-13T07:36:29.773 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.774 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.775 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.776 DEBUG:boto:Params: {} 2022-09-13T07:36:29.777 DEBUG:boto:Token: None 2022-09-13T07:36:29.778 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.779 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rA405uupNJgA8UmgVcNw4V+wqTY= 2022-09-13T07:36:29.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rA405uupNJgA8UmgVcNw4V+wqTY=', 'Content-Length': '0'} 2022-09-13T07:36:29.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e662009dbea88479-00632032fd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:29.783 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.784 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.784 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.785 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.785 DEBUG:boto:Method: GET 2022-09-13T07:36:29.786 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.787 DEBUG:boto:Data: 2022-09-13T07:36:29.788 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.789 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.790 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.790 DEBUG:boto:Params: {} 2022-09-13T07:36:29.791 DEBUG:boto:Token: None 2022-09-13T07:36:29.792 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.793 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S8GFq+d0Mg7deOAas5G24IEZ+Jo= 2022-09-13T07:36:29.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S8GFq+d0Mg7deOAas5G24IEZ+Jo=', 'Content-Length': '0'} 2022-09-13T07:36:29.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060b0541eae0597bc-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.838 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.839 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.840 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.840 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.841 DEBUG:boto:Method: GET 2022-09-13T07:36:29.841 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.842 DEBUG:boto:Data: 2022-09-13T07:36:29.842 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.843 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.843 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.844 DEBUG:boto:Params: {} 2022-09-13T07:36:29.844 DEBUG:boto:Token: None 2022-09-13T07:36:29.845 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.845 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S8GFq+d0Mg7deOAas5G24IEZ+Jo= 2022-09-13T07:36:29.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S8GFq+d0Mg7deOAas5G24IEZ+Jo=', 'Content-Length': '0'} 2022-09-13T07:36:29.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003006ead06c294654-00632032fd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.849 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:29.850 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:29.864 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:29.865 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:29.866 DEBUG:boto:Method: GET 2022-09-13T07:36:29.866 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:29.867 DEBUG:boto:Data: 2022-09-13T07:36:29.867 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.868 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.868 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.869 DEBUG:boto:Params: {} 2022-09-13T07:36:29.870 DEBUG:boto:Token: None 2022-09-13T07:36:29.870 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/ 2022-09-13T07:36:29.871 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HXr9m5PIHfLbybYL+AayokRREds= 2022-09-13T07:36:29.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HXr9m5PIHfLbybYL+AayokRREds=', 'Content-Length': '0'} 2022-09-13T07:36:29.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0294b832d973276-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.875 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:29.876 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:29.876 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:29.877 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:29.877 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:29.878 DEBUG:boto:Method: GET 2022-09-13T07:36:29.878 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:29.879 DEBUG:boto:Data: 2022-09-13T07:36:29.879 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.880 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.880 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.881 DEBUG:boto:Params: {} 2022-09-13T07:36:29.881 DEBUG:boto:Token: None 2022-09-13T07:36:29.882 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/ 2022-09-13T07:36:29.883 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HXr9m5PIHfLbybYL+AayokRREds= 2022-09-13T07:36:29.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HXr9m5PIHfLbybYL+AayokRREds=', 'Content-Length': '0'} 2022-09-13T07:36:29.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010cbae5395157183-00632032fd-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.886 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:29.887 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.888 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.888 DEBUG:boto:Method: HEAD 2022-09-13T07:36:29.889 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.889 DEBUG:boto:Data: 2022-09-13T07:36:29.890 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.890 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.891 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.891 DEBUG:boto:Params: {} 2022-09-13T07:36:29.892 DEBUG:boto:Token: None 2022-09-13T07:36:29.892 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.893 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+EV6VhFFbM9v5pR+hKKDxvs9afU= 2022-09-13T07:36:29.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+EV6VhFFbM9v5pR+hKKDxvs9afU=', 'Content-Length': '0'} 2022-09-13T07:36:29.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000161a1d424085d50b-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.898 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.899 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.900 DEBUG:boto:Method: HEAD 2022-09-13T07:36:29.900 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.901 DEBUG:boto:Data: 2022-09-13T07:36:29.901 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.902 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.902 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.903 DEBUG:boto:Params: {} 2022-09-13T07:36:29.903 DEBUG:boto:Token: None 2022-09-13T07:36:29.904 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.904 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+EV6VhFFbM9v5pR+hKKDxvs9afU= 2022-09-13T07:36:29.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+EV6VhFFbM9v5pR+hKKDxvs9afU=', 'Content-Length': '0'} 2022-09-13T07:36:29.907 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000769c92d5e867b5c4-00632032fd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:29.908 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.909 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.910 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.910 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.911 DEBUG:boto:Method: GET 2022-09-13T07:36:29.911 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.912 DEBUG:boto:Data: 2022-09-13T07:36:29.913 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.914 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.915 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.916 DEBUG:boto:Params: {} 2022-09-13T07:36:29.917 DEBUG:boto:Token: None 2022-09-13T07:36:29.918 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.918 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S8GFq+d0Mg7deOAas5G24IEZ+Jo= 2022-09-13T07:36:29.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S8GFq+d0Mg7deOAas5G24IEZ+Jo=', 'Content-Length': '0'} 2022-09-13T07:36:29.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6129c63a76ae735-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.923 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.923 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:29.924 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.924 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.925 DEBUG:boto:Method: GET 2022-09-13T07:36:29.925 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.926 DEBUG:boto:Data: 2022-09-13T07:36:29.926 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.927 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:29.927 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.928 DEBUG:boto:Params: {} 2022-09-13T07:36:29.929 DEBUG:boto:Token: None 2022-09-13T07:36:29.929 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:29.930 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S8GFq+d0Mg7deOAas5G24IEZ+Jo= 2022-09-13T07:36:29.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S8GFq+d0Mg7deOAas5G24IEZ+Jo=', 'Content-Length': '0'} 2022-09-13T07:36:29.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000287625e8a78d9b81-00632032fd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.974 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:29.975 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:29.976 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:29.977 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:29.978 DEBUG:boto:Method: GET 2022-09-13T07:36:29.979 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:29.980 DEBUG:boto:Data: 2022-09-13T07:36:29.981 DEBUG:boto:Headers: {} 2022-09-13T07:36:29.982 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:29.983 DEBUG:boto:Port: 8000 2022-09-13T07:36:29.984 DEBUG:boto:Params: {} 2022-09-13T07:36:29.985 DEBUG:boto:Token: None 2022-09-13T07:36:29.986 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:29 GMT /yxvvuh-17/ 2022-09-13T07:36:29.987 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HXr9m5PIHfLbybYL+AayokRREds= 2022-09-13T07:36:29.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HXr9m5PIHfLbybYL+AayokRREds=', 'Content-Length': '0'} 2022-09-13T07:36:29.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000345e2121ce2ab2d7-00632032fd-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:29.993 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:29.994 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:29.995 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:29.996 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:29.997 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:29.998 DEBUG:boto:Method: GET 2022-09-13T07:36:29.999 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:30.000 DEBUG:boto:Data: 2022-09-13T07:36:30.001 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.002 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.003 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.004 DEBUG:boto:Params: {} 2022-09-13T07:36:30.005 DEBUG:boto:Token: None 2022-09-13T07:36:30.006 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/ 2022-09-13T07:36:30.007 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI= 2022-09-13T07:36:30.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI=', 'Content-Length': '0'} 2022-09-13T07:36:30.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a58a20cf88de89e-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.012 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:30.015 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.016 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.017 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.018 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.019 DEBUG:boto:Method: GET 2022-09-13T07:36:30.020 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.020 DEBUG:boto:Data: 2022-09-13T07:36:30.021 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.022 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:30.023 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.024 DEBUG:boto:Params: {} 2022-09-13T07:36:30.024 DEBUG:boto:Token: None 2022-09-13T07:36:30.025 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.026 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:koZleZWKoYwKje8EWm2qOq+82qk= 2022-09-13T07:36:30.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:koZleZWKoYwKje8EWm2qOq+82qk=', 'Content-Length': '0'} 2022-09-13T07:36:30.029 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000adef90d4de5f9c2c-00632032fe-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.070 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.071 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.071 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.072 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.072 DEBUG:boto:Method: GET 2022-09-13T07:36:30.073 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.073 DEBUG:boto:Data: 2022-09-13T07:36:30.074 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.076 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.077 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.078 DEBUG:boto:Params: {} 2022-09-13T07:36:30.079 DEBUG:boto:Token: None 2022-09-13T07:36:30.080 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.082 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:koZleZWKoYwKje8EWm2qOq+82qk= 2022-09-13T07:36:30.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:koZleZWKoYwKje8EWm2qOq+82qk=', 'Content-Length': '0'} 2022-09-13T07:36:30.085 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088f375adab9a24ab-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.086 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.087 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.088 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.088 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.089 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.089 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.090 DEBUG:boto:Data: 2022-09-13T07:36:30.090 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.091 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:30.091 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.092 DEBUG:boto:Params: {} 2022-09-13T07:36:30.092 DEBUG:boto:Token: None 2022-09-13T07:36:30.093 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.093 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4gZuJUc0HTm59ka/Mgm1iUWzGx0= 2022-09-13T07:36:30.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4gZuJUc0HTm59ka/Mgm1iUWzGx0=', 'Content-Length': '0'} 2022-09-13T07:36:30.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dcd4dd3342bcbe64-00632032fe-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.096 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.097 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.097 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.098 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.098 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.099 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.099 DEBUG:boto:Data: 2022-09-13T07:36:30.100 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.101 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.101 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.102 DEBUG:boto:Params: {} 2022-09-13T07:36:30.103 DEBUG:boto:Token: None 2022-09-13T07:36:30.103 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.104 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4gZuJUc0HTm59ka/Mgm1iUWzGx0= 2022-09-13T07:36:30.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4gZuJUc0HTm59ka/Mgm1iUWzGx0=', 'Content-Length': '0'} 2022-09-13T07:36:30.107 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f487b53a2121135-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:30.109 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.110 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.111 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.111 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.112 DEBUG:boto:Method: GET 2022-09-13T07:36:30.113 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.113 DEBUG:boto:Data: 2022-09-13T07:36:30.114 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.114 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:30.115 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.116 DEBUG:boto:Params: {} 2022-09-13T07:36:30.116 DEBUG:boto:Token: None 2022-09-13T07:36:30.117 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.118 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:koZleZWKoYwKje8EWm2qOq+82qk= 2022-09-13T07:36:30.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:koZleZWKoYwKje8EWm2qOq+82qk=', 'Content-Length': '0'} 2022-09-13T07:36:30.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001037b530b1d283cd-00632032fe-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.122 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.122 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.123 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.124 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.124 DEBUG:boto:Method: GET 2022-09-13T07:36:30.125 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.125 DEBUG:boto:Data: 2022-09-13T07:36:30.126 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.126 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.127 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.127 DEBUG:boto:Params: {} 2022-09-13T07:36:30.128 DEBUG:boto:Token: None 2022-09-13T07:36:30.128 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.129 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:koZleZWKoYwKje8EWm2qOq+82qk= 2022-09-13T07:36:30.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:koZleZWKoYwKje8EWm2qOq+82qk=', 'Content-Length': '0'} 2022-09-13T07:36:30.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc4c840f55b58d70-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.133 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.133 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.134 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:30.135 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:30.135 DEBUG:boto:Method: GET 2022-09-13T07:36:30.136 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:30.136 DEBUG:boto:Data: 2022-09-13T07:36:30.137 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.137 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:30.138 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.138 DEBUG:boto:Params: {} 2022-09-13T07:36:30.139 DEBUG:boto:Token: None 2022-09-13T07:36:30.140 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/ 2022-09-13T07:36:30.140 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI= 2022-09-13T07:36:30.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI=', 'Content-Length': '0'} 2022-09-13T07:36:30.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf86329128ae3644-00632032fe-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.151 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.153 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.154 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.155 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:30.155 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:30.157 DEBUG:boto:Method: GET 2022-09-13T07:36:30.158 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:30.159 DEBUG:boto:Data: 2022-09-13T07:36:30.160 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.161 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.162 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.163 DEBUG:boto:Params: {} 2022-09-13T07:36:30.164 DEBUG:boto:Token: None 2022-09-13T07:36:30.165 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/ 2022-09-13T07:36:30.167 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI= 2022-09-13T07:36:30.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI=', 'Content-Length': '0'} 2022-09-13T07:36:30.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce52bcb6fa7f9605-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.172 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.173 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.174 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.175 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.176 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.177 DEBUG:boto:Data: 2022-09-13T07:36:30.178 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.178 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:30.179 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.179 DEBUG:boto:Params: {} 2022-09-13T07:36:30.180 DEBUG:boto:Token: None 2022-09-13T07:36:30.180 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.181 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RddmaF0nzp5RhOXFK8BcUMAW+64= 2022-09-13T07:36:30.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RddmaF0nzp5RhOXFK8BcUMAW+64=', 'Content-Length': '0'} 2022-09-13T07:36:30.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042ae2029e0cbd37c-00632032fe-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.185 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.185 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.186 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.186 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.187 DEBUG:boto:Data: 2022-09-13T07:36:30.187 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.188 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.188 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.189 DEBUG:boto:Params: {} 2022-09-13T07:36:30.189 DEBUG:boto:Token: None 2022-09-13T07:36:30.190 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.191 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RddmaF0nzp5RhOXFK8BcUMAW+64= 2022-09-13T07:36:30.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RddmaF0nzp5RhOXFK8BcUMAW+64=', 'Content-Length': '0'} 2022-09-13T07:36:30.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c93797d01cc4b102-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:30.196 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.196 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.197 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.197 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.198 DEBUG:boto:Method: GET 2022-09-13T07:36:30.198 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.199 DEBUG:boto:Data: 2022-09-13T07:36:30.199 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.200 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:30.200 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.201 DEBUG:boto:Params: {} 2022-09-13T07:36:30.202 DEBUG:boto:Token: None 2022-09-13T07:36:30.202 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.203 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:koZleZWKoYwKje8EWm2qOq+82qk= 2022-09-13T07:36:30.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:koZleZWKoYwKje8EWm2qOq+82qk=', 'Content-Length': '0'} 2022-09-13T07:36:30.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095b0c9203c0829fb-00632032fe-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.207 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.207 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:30.208 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.208 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.209 DEBUG:boto:Method: GET 2022-09-13T07:36:30.209 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.210 DEBUG:boto:Data: 2022-09-13T07:36:30.210 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.211 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.211 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.212 DEBUG:boto:Params: {} 2022-09-13T07:36:30.212 DEBUG:boto:Token: None 2022-09-13T07:36:30.213 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:30.213 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:koZleZWKoYwKje8EWm2qOq+82qk= 2022-09-13T07:36:30.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:koZleZWKoYwKje8EWm2qOq+82qk=', 'Content-Length': '0'} 2022-09-13T07:36:30.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037d9663fe42b0b15-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.258 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.259 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.260 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:30.260 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:30.261 DEBUG:boto:Method: GET 2022-09-13T07:36:30.261 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:30.262 DEBUG:boto:Data: 2022-09-13T07:36:30.263 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.263 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:30.264 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.264 DEBUG:boto:Params: {} 2022-09-13T07:36:30.265 DEBUG:boto:Token: None 2022-09-13T07:36:30.266 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/ 2022-09-13T07:36:30.267 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI= 2022-09-13T07:36:30.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI=', 'Content-Length': '0'} 2022-09-13T07:36:30.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eccb460077a9b2c-00632032fe-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.271 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.272 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.273 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.273 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:30.274 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:30.274 DEBUG:boto:Method: GET 2022-09-13T07:36:30.275 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:30.275 DEBUG:boto:Data: 2022-09-13T07:36:30.276 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.276 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.277 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.277 DEBUG:boto:Params: {} 2022-09-13T07:36:30.278 DEBUG:boto:Token: None 2022-09-13T07:36:30.279 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/ 2022-09-13T07:36:30.279 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI= 2022-09-13T07:36:30.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI=', 'Content-Length': '0'} 2022-09-13T07:36:30.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e04998c67e9a554-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.283 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:30.285 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.285 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.286 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.287 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.288 DEBUG:boto:Method: GET 2022-09-13T07:36:30.288 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.289 DEBUG:boto:Data: 2022-09-13T07:36:30.290 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.291 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:30.292 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.293 DEBUG:boto:Params: {} 2022-09-13T07:36:30.294 DEBUG:boto:Token: None 2022-09-13T07:36:30.295 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.296 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LthkYBn8QZfIB1Nfg6GkP1WgHaw= 2022-09-13T07:36:30.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LthkYBn8QZfIB1Nfg6GkP1WgHaw=', 'Content-Length': '0'} 2022-09-13T07:36:30.300 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d5f6c8952201c06-00632032fe-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.346 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.347 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.348 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.349 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.350 DEBUG:boto:Method: GET 2022-09-13T07:36:30.351 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.352 DEBUG:boto:Data: 2022-09-13T07:36:30.353 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.354 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.355 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.356 DEBUG:boto:Params: {} 2022-09-13T07:36:30.357 DEBUG:boto:Token: None 2022-09-13T07:36:30.358 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.359 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LthkYBn8QZfIB1Nfg6GkP1WgHaw= 2022-09-13T07:36:30.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LthkYBn8QZfIB1Nfg6GkP1WgHaw=', 'Content-Length': '0'} 2022-09-13T07:36:30.363 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064d3559549e3abbb-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.364 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.365 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.366 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.367 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.368 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.369 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.370 DEBUG:boto:Data: 2022-09-13T07:36:30.371 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.372 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:30.373 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.374 DEBUG:boto:Params: {} 2022-09-13T07:36:30.375 DEBUG:boto:Token: None 2022-09-13T07:36:30.376 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.377 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j5DFgHbAvJklXiU7RjA2o/B0eag= 2022-09-13T07:36:30.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j5DFgHbAvJklXiU7RjA2o/B0eag=', 'Content-Length': '0'} 2022-09-13T07:36:30.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005267da8af6dd323-00632032fe-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.382 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.383 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.384 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.385 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.386 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.387 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.387 DEBUG:boto:Data: 2022-09-13T07:36:30.388 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.389 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.390 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.391 DEBUG:boto:Params: {} 2022-09-13T07:36:30.392 DEBUG:boto:Token: None 2022-09-13T07:36:30.393 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.394 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:j5DFgHbAvJklXiU7RjA2o/B0eag= 2022-09-13T07:36:30.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:j5DFgHbAvJklXiU7RjA2o/B0eag=', 'Content-Length': '0'} 2022-09-13T07:36:30.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ffd64e529e15fc6-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:30.400 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.401 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.402 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.403 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.404 DEBUG:boto:Method: GET 2022-09-13T07:36:30.406 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.407 DEBUG:boto:Data: 2022-09-13T07:36:30.408 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.409 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:30.410 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.411 DEBUG:boto:Params: {} 2022-09-13T07:36:30.412 DEBUG:boto:Token: None 2022-09-13T07:36:30.413 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.414 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LthkYBn8QZfIB1Nfg6GkP1WgHaw= 2022-09-13T07:36:30.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LthkYBn8QZfIB1Nfg6GkP1WgHaw=', 'Content-Length': '0'} 2022-09-13T07:36:30.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003aa62f83fe5c918-00632032fe-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.418 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.419 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.420 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.420 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.421 DEBUG:boto:Method: GET 2022-09-13T07:36:30.421 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.422 DEBUG:boto:Data: 2022-09-13T07:36:30.422 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.423 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.423 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.424 DEBUG:boto:Params: {} 2022-09-13T07:36:30.424 DEBUG:boto:Token: None 2022-09-13T07:36:30.425 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.425 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LthkYBn8QZfIB1Nfg6GkP1WgHaw= 2022-09-13T07:36:30.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LthkYBn8QZfIB1Nfg6GkP1WgHaw=', 'Content-Length': '0'} 2022-09-13T07:36:30.428 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3aab76f8b3a7e55-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.429 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.430 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.431 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:30.431 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:30.432 DEBUG:boto:Method: GET 2022-09-13T07:36:30.432 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:30.433 DEBUG:boto:Data: 2022-09-13T07:36:30.433 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.434 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:30.435 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.436 DEBUG:boto:Params: {} 2022-09-13T07:36:30.437 DEBUG:boto:Token: None 2022-09-13T07:36:30.437 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/ 2022-09-13T07:36:30.438 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI= 2022-09-13T07:36:30.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI=', 'Content-Length': '0'} 2022-09-13T07:36:30.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c10c6c912928eb6b-00632032fe-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.442 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.443 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.443 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.444 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:30.444 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:30.445 DEBUG:boto:Method: GET 2022-09-13T07:36:30.445 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:30.446 DEBUG:boto:Data: 2022-09-13T07:36:30.446 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.447 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.447 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.448 DEBUG:boto:Params: {} 2022-09-13T07:36:30.449 DEBUG:boto:Token: None 2022-09-13T07:36:30.449 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/ 2022-09-13T07:36:30.450 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI= 2022-09-13T07:36:30.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI=', 'Content-Length': '0'} 2022-09-13T07:36:30.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000647d30719bfbc4a0-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.454 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.455 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.455 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.456 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.456 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.457 DEBUG:boto:Data: 2022-09-13T07:36:30.457 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.458 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:30.458 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.459 DEBUG:boto:Params: {} 2022-09-13T07:36:30.459 DEBUG:boto:Token: None 2022-09-13T07:36:30.460 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.461 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9724jaQwo012TPtyJtMXQw2VNFY= 2022-09-13T07:36:30.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9724jaQwo012TPtyJtMXQw2VNFY=', 'Content-Length': '0'} 2022-09-13T07:36:30.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000160380a527d7f6ec-00632032fe-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.465 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.466 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.466 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.467 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.467 DEBUG:boto:Data: 2022-09-13T07:36:30.468 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.468 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.469 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.469 DEBUG:boto:Params: {} 2022-09-13T07:36:30.470 DEBUG:boto:Token: None 2022-09-13T07:36:30.471 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.471 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9724jaQwo012TPtyJtMXQw2VNFY= 2022-09-13T07:36:30.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9724jaQwo012TPtyJtMXQw2VNFY=', 'Content-Length': '0'} 2022-09-13T07:36:30.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1d3a8e0569cc80d-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:30.476 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.476 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.477 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.477 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.478 DEBUG:boto:Method: GET 2022-09-13T07:36:30.479 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.480 DEBUG:boto:Data: 2022-09-13T07:36:30.481 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.482 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:30.483 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.484 DEBUG:boto:Params: {} 2022-09-13T07:36:30.485 DEBUG:boto:Token: None 2022-09-13T07:36:30.486 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.487 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LthkYBn8QZfIB1Nfg6GkP1WgHaw= 2022-09-13T07:36:30.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LthkYBn8QZfIB1Nfg6GkP1WgHaw=', 'Content-Length': '0'} 2022-09-13T07:36:30.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c62f4c137dad7bc-00632032fe-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.492 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.492 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:30.493 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.493 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.494 DEBUG:boto:Method: GET 2022-09-13T07:36:30.494 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.495 DEBUG:boto:Data: 2022-09-13T07:36:30.496 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.496 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.497 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.497 DEBUG:boto:Params: {} 2022-09-13T07:36:30.498 DEBUG:boto:Token: None 2022-09-13T07:36:30.498 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:30.499 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LthkYBn8QZfIB1Nfg6GkP1WgHaw= 2022-09-13T07:36:30.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LthkYBn8QZfIB1Nfg6GkP1WgHaw=', 'Content-Length': '0'} 2022-09-13T07:36:30.502 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e26fcec02b21bde6-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.542 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.543 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.544 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:30.544 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:30.545 DEBUG:boto:Method: GET 2022-09-13T07:36:30.545 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:30.546 DEBUG:boto:Data: 2022-09-13T07:36:30.546 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.547 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:30.547 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.548 DEBUG:boto:Params: {} 2022-09-13T07:36:30.548 DEBUG:boto:Token: None 2022-09-13T07:36:30.549 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/ 2022-09-13T07:36:30.550 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI= 2022-09-13T07:36:30.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI=', 'Content-Length': '0'} 2022-09-13T07:36:30.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001614337ab82f2ada-00632032fe-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.554 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.555 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.555 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.556 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:30.556 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:30.557 DEBUG:boto:Method: GET 2022-09-13T07:36:30.557 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:30.558 DEBUG:boto:Data: 2022-09-13T07:36:30.558 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.559 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.559 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.560 DEBUG:boto:Params: {} 2022-09-13T07:36:30.560 DEBUG:boto:Token: None 2022-09-13T07:36:30.561 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/ 2022-09-13T07:36:30.561 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI= 2022-09-13T07:36:30.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI=', 'Content-Length': '0'} 2022-09-13T07:36:30.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059c17c1d74725bb0-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.565 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.566 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-17 zones={test-zone2, test-zone1} 2022-09-13T07:36:30.567 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-17 zones={test-zone3, test-zone1} 2022-09-13T07:36:30.567 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.568 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.568 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.569 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:36:30.569 DEBUG:boto:Data: 2022-09-13T07:36:30.569 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.570 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:30.570 DEBUG:boto:Port: 8001 2022-09-13T07:36:30.571 DEBUG:boto:Params: {} 2022-09-13T07:36:30.571 DEBUG:boto:Token: None 2022-09-13T07:36:30.572 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/ 2022-09-13T07:36:30.573 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1S25q17ZCId4vnxWz7CPsU81pXE= 2022-09-13T07:36:30.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1S25q17ZCId4vnxWz7CPsU81pXE=', 'Content-Length': '0'} 2022-09-13T07:36:30.576 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbd520235b114936-00632032fe-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.576 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.577 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.577 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.578 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:36:30.578 DEBUG:boto:Data: 2022-09-13T07:36:30.579 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.579 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.580 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.580 DEBUG:boto:Params: {} 2022-09-13T07:36:30.581 DEBUG:boto:Token: None 2022-09-13T07:36:30.581 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/ 2022-09-13T07:36:30.582 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1S25q17ZCId4vnxWz7CPsU81pXE= 2022-09-13T07:36:30.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1S25q17ZCId4vnxWz7CPsU81pXE=', 'Content-Length': '0'} 2022-09-13T07:36:30.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cddfa21c729cd566-00632032fe-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.586 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:36:30.586 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.587 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.587 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:30.588 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:30.588 DEBUG:boto:Method: GET 2022-09-13T07:36:30.589 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:30.589 DEBUG:boto:Data: 2022-09-13T07:36:30.590 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.590 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:30.591 DEBUG:boto:Port: 8001 2022-09-13T07:36:30.591 DEBUG:boto:Params: {} 2022-09-13T07:36:30.592 DEBUG:boto:Token: None 2022-09-13T07:36:30.592 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/?versions 2022-09-13T07:36:30.593 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iJ9NpznTc6dGlaqao2W4mfBBLKU= 2022-09-13T07:36:30.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iJ9NpznTc6dGlaqao2W4mfBBLKU=', 'Content-Length': '0'} 2022-09-13T07:36:30.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fde1c8d9ed2b673-00632032fe-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.597 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.598 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:36:30.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:36:30.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:36:30.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:36:30.600 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:36:30.600 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.601 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.601 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:30.602 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:30.602 DEBUG:boto:Method: GET 2022-09-13T07:36:30.603 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:30.603 DEBUG:boto:Data: 2022-09-13T07:36:30.604 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.604 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.605 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.605 DEBUG:boto:Params: {} 2022-09-13T07:36:30.606 DEBUG:boto:Token: None 2022-09-13T07:36:30.606 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/?versions 2022-09-13T07:36:30.607 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iJ9NpznTc6dGlaqao2W4mfBBLKU= 2022-09-13T07:36:30.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iJ9NpznTc6dGlaqao2W4mfBBLKU=', 'Content-Length': '0'} 2022-09-13T07:36:30.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d416922cdb59c5d-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.611 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.612 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:36:30.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:36:30.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:36:30.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:36:30.614 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.614 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.615 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:30.615 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:30.616 DEBUG:boto:Method: GET 2022-09-13T07:36:30.616 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:30.617 DEBUG:boto:Data: 2022-09-13T07:36:30.617 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.618 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:30.618 DEBUG:boto:Port: 8001 2022-09-13T07:36:30.619 DEBUG:boto:Params: {} 2022-09-13T07:36:30.619 DEBUG:boto:Token: None 2022-09-13T07:36:30.620 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/?versions 2022-09-13T07:36:30.620 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iJ9NpznTc6dGlaqao2W4mfBBLKU= 2022-09-13T07:36:30.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iJ9NpznTc6dGlaqao2W4mfBBLKU=', 'Content-Length': '0'} 2022-09-13T07:36:30.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b85aee48523ac4bf-00632032fe-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.624 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.625 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.626 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.626 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:30.627 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:30.627 DEBUG:boto:Method: GET 2022-09-13T07:36:30.628 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:30.629 DEBUG:boto:Data: 2022-09-13T07:36:30.629 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.630 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.630 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.631 DEBUG:boto:Params: {} 2022-09-13T07:36:30.631 DEBUG:boto:Token: None 2022-09-13T07:36:30.632 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/?versions 2022-09-13T07:36:30.632 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iJ9NpznTc6dGlaqao2W4mfBBLKU= 2022-09-13T07:36:30.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iJ9NpznTc6dGlaqao2W4mfBBLKU=', 'Content-Length': '0'} 2022-09-13T07:36:30.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f3c6fcfb4b08649-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.636 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:30.637 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:30.638 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:30.639 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.639 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.640 DEBUG:boto:Method: GET 2022-09-13T07:36:30.640 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.641 DEBUG:boto:Data: 2022-09-13T07:36:30.641 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.642 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:30.642 DEBUG:boto:Port: 8001 2022-09-13T07:36:30.643 DEBUG:boto:Params: {} 2022-09-13T07:36:30.643 DEBUG:boto:Token: None 2022-09-13T07:36:30.644 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.644 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VY/2k304bGO/6sHCdsfxjkRzQb0= 2022-09-13T07:36:30.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VY/2k304bGO/6sHCdsfxjkRzQb0=', 'Content-Length': '0'} 2022-09-13T07:36:30.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000086ddc393f7319aa9-00632032fe-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.648 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:30.648 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:30.649 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.649 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.650 DEBUG:boto:Method: GET 2022-09-13T07:36:30.650 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.651 DEBUG:boto:Data: 2022-09-13T07:36:30.651 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.652 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.652 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.653 DEBUG:boto:Params: {} 2022-09-13T07:36:30.653 DEBUG:boto:Token: None 2022-09-13T07:36:30.654 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.654 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VY/2k304bGO/6sHCdsfxjkRzQb0= 2022-09-13T07:36:30.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VY/2k304bGO/6sHCdsfxjkRzQb0=', 'Content-Length': '0'} 2022-09-13T07:36:30.657 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006cec0a44c57dbd44-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.658 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:30.658 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:30.659 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.659 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.660 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.660 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.661 DEBUG:boto:Data: 2022-09-13T07:36:30.661 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.662 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:30.662 DEBUG:boto:Port: 8001 2022-09-13T07:36:30.662 DEBUG:boto:Params: {} 2022-09-13T07:36:30.663 DEBUG:boto:Token: None 2022-09-13T07:36:30.664 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.664 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:E5B1d1m+Un0h+YHOUjEXkbzhV9E= 2022-09-13T07:36:30.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:E5B1d1m+Un0h+YHOUjEXkbzhV9E=', 'Content-Length': '0'} 2022-09-13T07:36:30.666 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000043bd3cbceac95010-00632032fe-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.667 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:30.668 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:30.668 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.669 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.669 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.669 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.670 DEBUG:boto:Data: 2022-09-13T07:36:30.670 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.671 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.671 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.672 DEBUG:boto:Params: {} 2022-09-13T07:36:30.672 DEBUG:boto:Token: None 2022-09-13T07:36:30.673 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.673 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:E5B1d1m+Un0h+YHOUjEXkbzhV9E= 2022-09-13T07:36:30.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:E5B1d1m+Un0h+YHOUjEXkbzhV9E=', 'Content-Length': '0'} 2022-09-13T07:36:30.676 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000576441b3909b389f-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:30.678 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:30.679 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:30.679 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.680 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.680 DEBUG:boto:Method: GET 2022-09-13T07:36:30.681 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.681 DEBUG:boto:Data: 2022-09-13T07:36:30.682 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.683 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:30.683 DEBUG:boto:Port: 8001 2022-09-13T07:36:30.684 DEBUG:boto:Params: {} 2022-09-13T07:36:30.684 DEBUG:boto:Token: None 2022-09-13T07:36:30.685 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.686 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VY/2k304bGO/6sHCdsfxjkRzQb0= 2022-09-13T07:36:30.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VY/2k304bGO/6sHCdsfxjkRzQb0=', 'Content-Length': '0'} 2022-09-13T07:36:30.689 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b47a6532cd2d200e-00632032fe-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.689 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:30.690 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:30.691 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.691 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.692 DEBUG:boto:Method: GET 2022-09-13T07:36:30.692 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.693 DEBUG:boto:Data: 2022-09-13T07:36:30.693 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.694 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.695 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.695 DEBUG:boto:Params: {} 2022-09-13T07:36:30.696 DEBUG:boto:Token: None 2022-09-13T07:36:30.697 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.697 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VY/2k304bGO/6sHCdsfxjkRzQb0= 2022-09-13T07:36:30.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VY/2k304bGO/6sHCdsfxjkRzQb0=', 'Content-Length': '0'} 2022-09-13T07:36:30.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b46ddeaa60b6630c-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.701 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.701 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.702 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:30.703 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:30.703 DEBUG:boto:Method: GET 2022-09-13T07:36:30.704 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:36:30.704 DEBUG:boto:Data: 2022-09-13T07:36:30.705 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.706 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:30.706 DEBUG:boto:Port: 8001 2022-09-13T07:36:30.707 DEBUG:boto:Params: {} 2022-09-13T07:36:30.707 DEBUG:boto:Token: None 2022-09-13T07:36:30.708 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/ 2022-09-13T07:36:30.709 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI= 2022-09-13T07:36:30.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI=', 'Content-Length': '0'} 2022-09-13T07:36:30.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000264328f8ed61ed85-00632032fe-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.755 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.756 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.757 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.758 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:30.758 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:30.759 DEBUG:boto:Method: GET 2022-09-13T07:36:30.759 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:36:30.760 DEBUG:boto:Data: 2022-09-13T07:36:30.761 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.761 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.762 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.762 DEBUG:boto:Params: {} 2022-09-13T07:36:30.763 DEBUG:boto:Token: None 2022-09-13T07:36:30.764 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/ 2022-09-13T07:36:30.764 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI= 2022-09-13T07:36:30.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI=', 'Content-Length': '0'} 2022-09-13T07:36:30.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dca374636be8517-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.775 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.776 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:30.777 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:30.777 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.778 DEBUG:boto:Path: /yxvvuh-17/dummy 2022-09-13T07:36:30.778 DEBUG:boto:Data: 2022-09-13T07:36:30.779 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.779 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:30.780 DEBUG:boto:Port: 8001 2022-09-13T07:36:30.781 DEBUG:boto:Params: {} 2022-09-13T07:36:30.781 DEBUG:boto:Token: None 2022-09-13T07:36:30.782 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/dummy 2022-09-13T07:36:30.782 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:JXrQ/YRs9AJoTDN4IUtkaXrrlOI= 2022-09-13T07:36:30.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JXrQ/YRs9AJoTDN4IUtkaXrrlOI=', 'Content-Length': '0'} 2022-09-13T07:36:30.789 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7ef12866199ee6b-00632032fe-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.790 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:30.791 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:30.791 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.792 DEBUG:boto:Path: /yxvvuh-17/dummy 2022-09-13T07:36:30.792 DEBUG:boto:Data: 2022-09-13T07:36:30.793 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.793 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.794 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.795 DEBUG:boto:Params: {} 2022-09-13T07:36:30.795 DEBUG:boto:Token: None 2022-09-13T07:36:30.796 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/dummy 2022-09-13T07:36:30.796 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:JXrQ/YRs9AJoTDN4IUtkaXrrlOI= 2022-09-13T07:36:30.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JXrQ/YRs9AJoTDN4IUtkaXrrlOI=', 'Content-Length': '0'} 2022-09-13T07:36:30.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e76f9bcd8300f16e-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:30.805 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:30.805 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:30.806 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.807 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.809 DEBUG:boto:Method: GET 2022-09-13T07:36:30.810 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.811 DEBUG:boto:Data: 2022-09-13T07:36:30.811 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.812 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:30.813 DEBUG:boto:Port: 8001 2022-09-13T07:36:30.814 DEBUG:boto:Params: {} 2022-09-13T07:36:30.814 DEBUG:boto:Token: None 2022-09-13T07:36:30.815 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.816 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VY/2k304bGO/6sHCdsfxjkRzQb0= 2022-09-13T07:36:30.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VY/2k304bGO/6sHCdsfxjkRzQb0=', 'Content-Length': '0'} 2022-09-13T07:36:30.820 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000158a3cf677281a7f-00632032fe-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.822 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:30.823 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:30.823 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.824 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.825 DEBUG:boto:Method: GET 2022-09-13T07:36:30.825 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.826 DEBUG:boto:Data: 2022-09-13T07:36:30.827 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.828 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.829 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.829 DEBUG:boto:Params: {} 2022-09-13T07:36:30.830 DEBUG:boto:Token: None 2022-09-13T07:36:30.831 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:30.831 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VY/2k304bGO/6sHCdsfxjkRzQb0= 2022-09-13T07:36:30.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VY/2k304bGO/6sHCdsfxjkRzQb0=', 'Content-Length': '0'} 2022-09-13T07:36:30.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004dbfe0172923e1fe-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.842 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.842 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.843 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:30.844 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:30.844 DEBUG:boto:Method: GET 2022-09-13T07:36:30.845 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:36:30.846 DEBUG:boto:Data: 2022-09-13T07:36:30.846 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.848 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:30.854 DEBUG:boto:Port: 8001 2022-09-13T07:36:30.855 DEBUG:boto:Params: {} 2022-09-13T07:36:30.856 DEBUG:boto:Token: None 2022-09-13T07:36:30.856 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/ 2022-09-13T07:36:30.857 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI= 2022-09-13T07:36:30.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI=', 'Content-Length': '0'} 2022-09-13T07:36:30.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007fe2724dd7f47e5-00632032fe-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.906 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.908 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:30.910 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:30.911 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:30.912 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:30.913 DEBUG:boto:Method: GET 2022-09-13T07:36:30.914 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:36:30.915 DEBUG:boto:Data: 2022-09-13T07:36:30.916 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.917 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.918 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.919 DEBUG:boto:Params: {} 2022-09-13T07:36:30.920 DEBUG:boto:Token: None 2022-09-13T07:36:30.921 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/ 2022-09-13T07:36:30.923 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI= 2022-09-13T07:36:30.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/+P+XmGaXdg1lNTlx6h7/jazhHI=', 'Content-Length': '0'} 2022-09-13T07:36:30.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1799d0268821b65-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.929 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:30.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:30.931 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:30.932 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:30.933 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.934 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.936 DEBUG:boto:Method: GET 2022-09-13T07:36:30.936 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.937 DEBUG:boto:Data: 2022-09-13T07:36:30.938 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.939 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:30.940 DEBUG:boto:Port: 8001 2022-09-13T07:36:30.941 DEBUG:boto:Params: {} 2022-09-13T07:36:30.942 DEBUG:boto:Token: None 2022-09-13T07:36:30.943 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.943 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z4kf1dzbpz1NCVLOkJrW7FAD0UY= 2022-09-13T07:36:30.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z4kf1dzbpz1NCVLOkJrW7FAD0UY=', 'Content-Length': '0'} 2022-09-13T07:36:30.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a7703490d17b815-00632032fe-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.948 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:30.948 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:30.949 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.950 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.950 DEBUG:boto:Method: GET 2022-09-13T07:36:30.951 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.951 DEBUG:boto:Data: 2022-09-13T07:36:30.952 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.953 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.954 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.955 DEBUG:boto:Params: {} 2022-09-13T07:36:30.956 DEBUG:boto:Token: None 2022-09-13T07:36:30.956 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.957 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z4kf1dzbpz1NCVLOkJrW7FAD0UY= 2022-09-13T07:36:30.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z4kf1dzbpz1NCVLOkJrW7FAD0UY=', 'Content-Length': '0'} 2022-09-13T07:36:30.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089d8e39bd52cb662-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.961 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:30.962 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:30.962 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.963 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.964 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.964 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.965 DEBUG:boto:Data: 2022-09-13T07:36:30.966 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.966 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:30.967 DEBUG:boto:Port: 8001 2022-09-13T07:36:30.967 DEBUG:boto:Params: {} 2022-09-13T07:36:30.968 DEBUG:boto:Token: None 2022-09-13T07:36:30.969 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.969 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vjGbt7BBGlo+HDOtUBgAGK1+Tbg= 2022-09-13T07:36:30.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vjGbt7BBGlo+HDOtUBgAGK1+Tbg=', 'Content-Length': '0'} 2022-09-13T07:36:30.973 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047ef615f4a0712d7-00632032fe-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.974 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:30.975 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:30.976 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.977 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.978 DEBUG:boto:Method: HEAD 2022-09-13T07:36:30.978 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.979 DEBUG:boto:Data: 2022-09-13T07:36:30.980 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.981 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:30.982 DEBUG:boto:Port: 8000 2022-09-13T07:36:30.982 DEBUG:boto:Params: {} 2022-09-13T07:36:30.983 DEBUG:boto:Token: None 2022-09-13T07:36:30.984 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.985 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vjGbt7BBGlo+HDOtUBgAGK1+Tbg= 2022-09-13T07:36:30.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vjGbt7BBGlo+HDOtUBgAGK1+Tbg=', 'Content-Length': '0'} 2022-09-13T07:36:30.988 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000032cfc71702f1091-00632032fe-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:30.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:30.990 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:30.991 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:30.991 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.992 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.993 DEBUG:boto:Method: GET 2022-09-13T07:36:30.993 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.994 DEBUG:boto:Data: 2022-09-13T07:36:30.995 DEBUG:boto:Headers: {} 2022-09-13T07:36:30.995 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:30.996 DEBUG:boto:Port: 8001 2022-09-13T07:36:30.997 DEBUG:boto:Params: {} 2022-09-13T07:36:30.997 DEBUG:boto:Token: None 2022-09-13T07:36:30.998 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:30 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:30.999 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z4kf1dzbpz1NCVLOkJrW7FAD0UY= 2022-09-13T07:36:31.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z4kf1dzbpz1NCVLOkJrW7FAD0UY=', 'Content-Length': '0'} 2022-09-13T07:36:31.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0c5b8c5a6b83248-00632032fe-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.004 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:31.004 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:31.005 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:31.005 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:31.006 DEBUG:boto:Method: GET 2022-09-13T07:36:31.007 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:31.007 DEBUG:boto:Data: 2022-09-13T07:36:31.008 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.008 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.009 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.010 DEBUG:boto:Params: {} 2022-09-13T07:36:31.010 DEBUG:boto:Token: None 2022-09-13T07:36:31.011 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:31.012 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gH05HHgWQEzZv4j7R3M5sUCjKrE= 2022-09-13T07:36:31.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gH05HHgWQEzZv4j7R3M5sUCjKrE=', 'Content-Length': '0'} 2022-09-13T07:36:31.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d66cc937a66d3e9f-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.017 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.018 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.019 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:31.020 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:31.021 DEBUG:boto:Method: GET 2022-09-13T07:36:31.022 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:31.023 DEBUG:boto:Data: 2022-09-13T07:36:31.024 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.025 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.026 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.027 DEBUG:boto:Params: {} 2022-09-13T07:36:31.028 DEBUG:boto:Token: None 2022-09-13T07:36:31.028 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.029 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b40eccd99f23b27-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.075 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.076 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.077 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.078 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:31.079 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:31.080 DEBUG:boto:Method: GET 2022-09-13T07:36:31.080 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:31.081 DEBUG:boto:Data: 2022-09-13T07:36:31.081 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.082 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.082 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.083 DEBUG:boto:Params: {} 2022-09-13T07:36:31.084 DEBUG:boto:Token: None 2022-09-13T07:36:31.084 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.085 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000465ad61cd1d48d7-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.090 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.092 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:31.092 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:31.093 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.093 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1 2022-09-13T07:36:31.094 DEBUG:boto:Data: 2022-09-13T07:36:31.095 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.095 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.096 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.097 DEBUG:boto:Params: {} 2022-09-13T07:36:31.098 DEBUG:boto:Token: None 2022-09-13T07:36:31.099 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone1 2022-09-13T07:36:31.100 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XCCmlmz0D/ntSpQmT1Xgf83Me0Y= 2022-09-13T07:36:31.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XCCmlmz0D/ntSpQmT1Xgf83Me0Y=', 'Content-Length': '0'} 2022-09-13T07:36:31.104 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071a40750492a0b94-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.105 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:31.105 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:31.106 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.107 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1 2022-09-13T07:36:31.107 DEBUG:boto:Data: 2022-09-13T07:36:31.108 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.109 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.110 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.111 DEBUG:boto:Params: {} 2022-09-13T07:36:31.112 DEBUG:boto:Token: None 2022-09-13T07:36:31.113 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone1 2022-09-13T07:36:31.114 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XCCmlmz0D/ntSpQmT1Xgf83Me0Y= 2022-09-13T07:36:31.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XCCmlmz0D/ntSpQmT1Xgf83Me0Y=', 'Content-Length': '0'} 2022-09-13T07:36:31.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cf2499ed9b6044d-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:31.120 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:31.120 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:31.121 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:31.121 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:31.122 DEBUG:boto:Method: GET 2022-09-13T07:36:31.123 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:31.123 DEBUG:boto:Data: 2022-09-13T07:36:31.124 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.125 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.125 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.126 DEBUG:boto:Params: {} 2022-09-13T07:36:31.127 DEBUG:boto:Token: None 2022-09-13T07:36:31.127 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:31.128 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gH05HHgWQEzZv4j7R3M5sUCjKrE= 2022-09-13T07:36:31.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gH05HHgWQEzZv4j7R3M5sUCjKrE=', 'Content-Length': '0'} 2022-09-13T07:36:31.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7f3018626096051-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.132 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:31.133 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:31.134 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:31.134 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:31.135 DEBUG:boto:Method: GET 2022-09-13T07:36:31.135 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:31.136 DEBUG:boto:Data: 2022-09-13T07:36:31.136 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.137 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.137 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.138 DEBUG:boto:Params: {} 2022-09-13T07:36:31.139 DEBUG:boto:Token: None 2022-09-13T07:36:31.140 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:31.141 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gH05HHgWQEzZv4j7R3M5sUCjKrE= 2022-09-13T07:36:31.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gH05HHgWQEzZv4j7R3M5sUCjKrE=', 'Content-Length': '0'} 2022-09-13T07:36:31.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000948d4d92bf8d0969-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.187 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.188 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.189 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:31.189 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:31.190 DEBUG:boto:Method: GET 2022-09-13T07:36:31.191 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:31.191 DEBUG:boto:Data: 2022-09-13T07:36:31.192 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.193 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.194 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.194 DEBUG:boto:Params: {} 2022-09-13T07:36:31.195 DEBUG:boto:Token: None 2022-09-13T07:36:31.196 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.197 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e46c065d4e9265f-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.202 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.204 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.205 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.205 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:31.206 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:31.207 DEBUG:boto:Method: GET 2022-09-13T07:36:31.208 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:31.209 DEBUG:boto:Data: 2022-09-13T07:36:31.210 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.211 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.212 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.213 DEBUG:boto:Params: {} 2022-09-13T07:36:31.214 DEBUG:boto:Token: None 2022-09-13T07:36:31.215 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.216 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acb386bd49da67f7-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.222 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:31.224 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.225 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.226 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.227 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.228 DEBUG:boto:Method: GET 2022-09-13T07:36:31.229 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.230 DEBUG:boto:Data: 2022-09-13T07:36:31.231 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.232 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.232 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.233 DEBUG:boto:Params: {} 2022-09-13T07:36:31.235 DEBUG:boto:Token: None 2022-09-13T07:36:31.236 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.237 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5WgkkteHhcUw8b/5aWQu/d5HEYo= 2022-09-13T07:36:31.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5WgkkteHhcUw8b/5aWQu/d5HEYo=', 'Content-Length': '0'} 2022-09-13T07:36:31.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086d873b53dc9672d-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.242 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.243 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.244 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.244 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.245 DEBUG:boto:Method: GET 2022-09-13T07:36:31.246 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.247 DEBUG:boto:Data: 2022-09-13T07:36:31.248 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.249 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.249 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.250 DEBUG:boto:Params: {} 2022-09-13T07:36:31.250 DEBUG:boto:Token: None 2022-09-13T07:36:31.251 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.252 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5WgkkteHhcUw8b/5aWQu/d5HEYo= 2022-09-13T07:36:31.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5WgkkteHhcUw8b/5aWQu/d5HEYo=', 'Content-Length': '0'} 2022-09-13T07:36:31.255 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000280fc66da71a108a-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.255 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.256 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.256 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.257 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.258 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.258 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.259 DEBUG:boto:Data: 2022-09-13T07:36:31.259 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.259 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.260 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.260 DEBUG:boto:Params: {} 2022-09-13T07:36:31.261 DEBUG:boto:Token: None 2022-09-13T07:36:31.262 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.262 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:H62plzBJJ1UXbSW8jFJWnGJmC2g= 2022-09-13T07:36:31.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:H62plzBJJ1UXbSW8jFJWnGJmC2g=', 'Content-Length': '0'} 2022-09-13T07:36:31.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf13126e1df502b0-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.266 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.266 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.267 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.267 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.268 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.268 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.269 DEBUG:boto:Data: 2022-09-13T07:36:31.269 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.270 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.270 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.270 DEBUG:boto:Params: {} 2022-09-13T07:36:31.271 DEBUG:boto:Token: None 2022-09-13T07:36:31.272 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.272 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:H62plzBJJ1UXbSW8jFJWnGJmC2g= 2022-09-13T07:36:31.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:H62plzBJJ1UXbSW8jFJWnGJmC2g=', 'Content-Length': '0'} 2022-09-13T07:36:31.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe5eec629a6ad4ac-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:31.276 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.277 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.277 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.278 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.278 DEBUG:boto:Method: GET 2022-09-13T07:36:31.279 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.279 DEBUG:boto:Data: 2022-09-13T07:36:31.280 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.280 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.281 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.281 DEBUG:boto:Params: {} 2022-09-13T07:36:31.282 DEBUG:boto:Token: None 2022-09-13T07:36:31.283 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.283 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5WgkkteHhcUw8b/5aWQu/d5HEYo= 2022-09-13T07:36:31.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5WgkkteHhcUw8b/5aWQu/d5HEYo=', 'Content-Length': '0'} 2022-09-13T07:36:31.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f678007b26f32526-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.287 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.288 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.288 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.289 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.289 DEBUG:boto:Method: GET 2022-09-13T07:36:31.290 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.290 DEBUG:boto:Data: 2022-09-13T07:36:31.291 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.292 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.292 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.293 DEBUG:boto:Params: {} 2022-09-13T07:36:31.293 DEBUG:boto:Token: None 2022-09-13T07:36:31.294 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.295 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5WgkkteHhcUw8b/5aWQu/d5HEYo= 2022-09-13T07:36:31.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5WgkkteHhcUw8b/5aWQu/d5HEYo=', 'Content-Length': '0'} 2022-09-13T07:36:31.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdc8adf10d08b2c3-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.338 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.339 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.340 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:31.341 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:31.342 DEBUG:boto:Method: GET 2022-09-13T07:36:31.342 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:31.343 DEBUG:boto:Data: 2022-09-13T07:36:31.344 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.345 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.346 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.347 DEBUG:boto:Params: {} 2022-09-13T07:36:31.348 DEBUG:boto:Token: None 2022-09-13T07:36:31.349 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.350 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3b96d52568e9d7b-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.356 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.357 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.359 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.359 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:31.360 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:31.361 DEBUG:boto:Method: GET 2022-09-13T07:36:31.362 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:31.363 DEBUG:boto:Data: 2022-09-13T07:36:31.364 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.365 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.366 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.367 DEBUG:boto:Params: {} 2022-09-13T07:36:31.368 DEBUG:boto:Token: None 2022-09-13T07:36:31.369 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.370 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034fbe9c539725d3c-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.375 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.376 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.377 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.377 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.378 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.378 DEBUG:boto:Data: 2022-09-13T07:36:31.379 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.379 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.380 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.380 DEBUG:boto:Params: {} 2022-09-13T07:36:31.381 DEBUG:boto:Token: None 2022-09-13T07:36:31.382 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.383 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Tuq9LNLY3cmVRKnC7ofQ49gZfOI= 2022-09-13T07:36:31.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Tuq9LNLY3cmVRKnC7ofQ49gZfOI=', 'Content-Length': '0'} 2022-09-13T07:36:31.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c3300b6401cfd6c-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.387 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.387 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.388 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.388 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.389 DEBUG:boto:Data: 2022-09-13T07:36:31.389 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.390 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.390 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.391 DEBUG:boto:Params: {} 2022-09-13T07:36:31.392 DEBUG:boto:Token: None 2022-09-13T07:36:31.393 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.393 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Tuq9LNLY3cmVRKnC7ofQ49gZfOI= 2022-09-13T07:36:31.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Tuq9LNLY3cmVRKnC7ofQ49gZfOI=', 'Content-Length': '0'} 2022-09-13T07:36:31.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad35374dccb1a567-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:31.399 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.400 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.401 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.402 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.402 DEBUG:boto:Method: GET 2022-09-13T07:36:31.403 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.403 DEBUG:boto:Data: 2022-09-13T07:36:31.404 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.404 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.405 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.405 DEBUG:boto:Params: {} 2022-09-13T07:36:31.406 DEBUG:boto:Token: None 2022-09-13T07:36:31.407 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.407 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5WgkkteHhcUw8b/5aWQu/d5HEYo= 2022-09-13T07:36:31.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5WgkkteHhcUw8b/5aWQu/d5HEYo=', 'Content-Length': '0'} 2022-09-13T07:36:31.410 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2c7e9d83a76cac7-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.454 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.455 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:31.455 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.456 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.457 DEBUG:boto:Method: GET 2022-09-13T07:36:31.457 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.458 DEBUG:boto:Data: 2022-09-13T07:36:31.458 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.459 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.459 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.460 DEBUG:boto:Params: {} 2022-09-13T07:36:31.460 DEBUG:boto:Token: None 2022-09-13T07:36:31.461 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:31.461 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5WgkkteHhcUw8b/5aWQu/d5HEYo= 2022-09-13T07:36:31.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5WgkkteHhcUw8b/5aWQu/d5HEYo=', 'Content-Length': '0'} 2022-09-13T07:36:31.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d2c9c4482dcfb58-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.465 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.466 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.466 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:31.467 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:31.467 DEBUG:boto:Method: GET 2022-09-13T07:36:31.468 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:31.468 DEBUG:boto:Data: 2022-09-13T07:36:31.475 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.475 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.476 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.477 DEBUG:boto:Params: {} 2022-09-13T07:36:31.478 DEBUG:boto:Token: None 2022-09-13T07:36:31.478 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.479 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021e85a30b3518998-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.483 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.485 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.486 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.487 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:31.488 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:31.488 DEBUG:boto:Method: GET 2022-09-13T07:36:31.489 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:31.490 DEBUG:boto:Data: 2022-09-13T07:36:31.491 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.492 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.493 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.494 DEBUG:boto:Params: {} 2022-09-13T07:36:31.496 DEBUG:boto:Token: None 2022-09-13T07:36:31.497 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.498 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011d603742ba1af79-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.503 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:31.506 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.507 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.508 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.509 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.510 DEBUG:boto:Method: GET 2022-09-13T07:36:31.511 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.512 DEBUG:boto:Data: 2022-09-13T07:36:31.513 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.514 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.514 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.516 DEBUG:boto:Params: {} 2022-09-13T07:36:31.517 DEBUG:boto:Token: None 2022-09-13T07:36:31.518 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.519 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ygrRsQmF7INhFnBTLk4HmLndfFA= 2022-09-13T07:36:31.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ygrRsQmF7INhFnBTLk4HmLndfFA=', 'Content-Length': '0'} 2022-09-13T07:36:31.523 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d74f577956b5e2f6-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.524 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.526 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.527 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.528 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.529 DEBUG:boto:Method: GET 2022-09-13T07:36:31.530 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.530 DEBUG:boto:Data: 2022-09-13T07:36:31.531 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.532 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.533 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.534 DEBUG:boto:Params: {} 2022-09-13T07:36:31.536 DEBUG:boto:Token: None 2022-09-13T07:36:31.537 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.538 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ygrRsQmF7INhFnBTLk4HmLndfFA= 2022-09-13T07:36:31.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ygrRsQmF7INhFnBTLk4HmLndfFA=', 'Content-Length': '0'} 2022-09-13T07:36:31.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecba593dc98c093b-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.582 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.583 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.583 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.584 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.584 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.585 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.585 DEBUG:boto:Data: 2022-09-13T07:36:31.585 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.586 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.586 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.587 DEBUG:boto:Params: {} 2022-09-13T07:36:31.587 DEBUG:boto:Token: None 2022-09-13T07:36:31.588 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.589 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HkzeDaGVf9U2frKeTW/g51fnrtQ= 2022-09-13T07:36:31.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HkzeDaGVf9U2frKeTW/g51fnrtQ=', 'Content-Length': '0'} 2022-09-13T07:36:31.591 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c69d3a7e5e64bea2-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.592 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.593 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.593 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.594 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.594 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.595 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.595 DEBUG:boto:Data: 2022-09-13T07:36:31.595 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.596 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.596 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.597 DEBUG:boto:Params: {} 2022-09-13T07:36:31.597 DEBUG:boto:Token: None 2022-09-13T07:36:31.598 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.598 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HkzeDaGVf9U2frKeTW/g51fnrtQ= 2022-09-13T07:36:31.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HkzeDaGVf9U2frKeTW/g51fnrtQ=', 'Content-Length': '0'} 2022-09-13T07:36:31.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ec691db620bd9f1-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:31.602 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.603 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.603 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.604 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.604 DEBUG:boto:Method: GET 2022-09-13T07:36:31.605 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.605 DEBUG:boto:Data: 2022-09-13T07:36:31.606 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.606 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.607 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.607 DEBUG:boto:Params: {} 2022-09-13T07:36:31.608 DEBUG:boto:Token: None 2022-09-13T07:36:31.608 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.609 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ygrRsQmF7INhFnBTLk4HmLndfFA= 2022-09-13T07:36:31.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ygrRsQmF7INhFnBTLk4HmLndfFA=', 'Content-Length': '0'} 2022-09-13T07:36:31.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a03def0103374ad8-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.613 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.613 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.614 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.614 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.615 DEBUG:boto:Method: GET 2022-09-13T07:36:31.615 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.616 DEBUG:boto:Data: 2022-09-13T07:36:31.616 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.617 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.617 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.618 DEBUG:boto:Params: {} 2022-09-13T07:36:31.618 DEBUG:boto:Token: None 2022-09-13T07:36:31.619 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.619 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ygrRsQmF7INhFnBTLk4HmLndfFA= 2022-09-13T07:36:31.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ygrRsQmF7INhFnBTLk4HmLndfFA=', 'Content-Length': '0'} 2022-09-13T07:36:31.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070d5b53cf986569d-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.623 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.623 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.624 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:31.624 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:31.625 DEBUG:boto:Method: GET 2022-09-13T07:36:31.625 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:31.626 DEBUG:boto:Data: 2022-09-13T07:36:31.627 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.627 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.628 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.628 DEBUG:boto:Params: {} 2022-09-13T07:36:31.629 DEBUG:boto:Token: None 2022-09-13T07:36:31.629 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.630 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d35217181790e81-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.634 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.635 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.635 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.636 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:31.636 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:31.637 DEBUG:boto:Method: GET 2022-09-13T07:36:31.637 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:31.638 DEBUG:boto:Data: 2022-09-13T07:36:31.638 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.639 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.639 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.640 DEBUG:boto:Params: {} 2022-09-13T07:36:31.641 DEBUG:boto:Token: None 2022-09-13T07:36:31.641 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.642 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3c329e954e0662b-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.646 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.647 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.647 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.648 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.648 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.649 DEBUG:boto:Data: 2022-09-13T07:36:31.649 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.650 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.650 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.651 DEBUG:boto:Params: {} 2022-09-13T07:36:31.651 DEBUG:boto:Token: None 2022-09-13T07:36:31.652 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.653 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:au0t9UuStd6kIuHU/qQCyIEDY1o= 2022-09-13T07:36:31.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:au0t9UuStd6kIuHU/qQCyIEDY1o=', 'Content-Length': '0'} 2022-09-13T07:36:31.656 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af5d87210d31b39b-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.656 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.657 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.658 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.658 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.659 DEBUG:boto:Data: 2022-09-13T07:36:31.659 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.660 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.660 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.661 DEBUG:boto:Params: {} 2022-09-13T07:36:31.661 DEBUG:boto:Token: None 2022-09-13T07:36:31.662 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.663 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:au0t9UuStd6kIuHU/qQCyIEDY1o= 2022-09-13T07:36:31.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:au0t9UuStd6kIuHU/qQCyIEDY1o=', 'Content-Length': '0'} 2022-09-13T07:36:31.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c08411ce3b4ce581-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:31.667 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.667 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.668 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.668 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.669 DEBUG:boto:Method: GET 2022-09-13T07:36:31.670 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.670 DEBUG:boto:Data: 2022-09-13T07:36:31.671 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.672 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.672 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.673 DEBUG:boto:Params: {} 2022-09-13T07:36:31.673 DEBUG:boto:Token: None 2022-09-13T07:36:31.674 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.675 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ygrRsQmF7INhFnBTLk4HmLndfFA= 2022-09-13T07:36:31.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ygrRsQmF7INhFnBTLk4HmLndfFA=', 'Content-Length': '0'} 2022-09-13T07:36:31.677 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082f535eb557b28cc-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.678 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.679 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:31.679 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.680 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.680 DEBUG:boto:Method: GET 2022-09-13T07:36:31.681 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.681 DEBUG:boto:Data: 2022-09-13T07:36:31.682 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.682 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.683 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.683 DEBUG:boto:Params: {} 2022-09-13T07:36:31.684 DEBUG:boto:Token: None 2022-09-13T07:36:31.684 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:31.685 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ygrRsQmF7INhFnBTLk4HmLndfFA= 2022-09-13T07:36:31.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ygrRsQmF7INhFnBTLk4HmLndfFA=', 'Content-Length': '0'} 2022-09-13T07:36:31.687 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4aa519fc103f34a-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.688 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.689 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.689 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:31.690 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:31.690 DEBUG:boto:Method: GET 2022-09-13T07:36:31.691 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:31.691 DEBUG:boto:Data: 2022-09-13T07:36:31.692 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.692 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.693 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.693 DEBUG:boto:Params: {} 2022-09-13T07:36:31.694 DEBUG:boto:Token: None 2022-09-13T07:36:31.695 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.695 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af7d247943d978d1-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.699 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.700 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.701 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.701 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:31.702 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:31.702 DEBUG:boto:Method: GET 2022-09-13T07:36:31.703 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:31.703 DEBUG:boto:Data: 2022-09-13T07:36:31.704 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.704 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:31.705 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.705 DEBUG:boto:Params: {} 2022-09-13T07:36:31.706 DEBUG:boto:Token: None 2022-09-13T07:36:31.706 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.707 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055e749be2f992b47-00632032ff-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.711 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.712 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-17 zones={test-zone3, test-zone1} 2022-09-13T07:36:31.712 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-17 zones={test-zone3, test-zone2} 2022-09-13T07:36:31.713 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.713 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.714 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.714 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:36:31.715 DEBUG:boto:Data: 2022-09-13T07:36:31.715 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.716 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.717 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.717 DEBUG:boto:Params: {} 2022-09-13T07:36:31.718 DEBUG:boto:Token: None 2022-09-13T07:36:31.718 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.719 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:idItWlKcMxj/xp4FQS/puD1YAvA= 2022-09-13T07:36:31.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:idItWlKcMxj/xp4FQS/puD1YAvA=', 'Content-Length': '0'} 2022-09-13T07:36:31.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5fb34968689f50a-00632032ff-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.722 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.723 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.723 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.724 DEBUG:boto:Path: /yxvvuh-17/ 2022-09-13T07:36:31.724 DEBUG:boto:Data: 2022-09-13T07:36:31.725 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.726 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:31.726 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.727 DEBUG:boto:Params: {} 2022-09-13T07:36:31.727 DEBUG:boto:Token: None 2022-09-13T07:36:31.728 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.728 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:idItWlKcMxj/xp4FQS/puD1YAvA= 2022-09-13T07:36:31.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:idItWlKcMxj/xp4FQS/puD1YAvA=', 'Content-Length': '0'} 2022-09-13T07:36:31.732 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a43188264eff8297-00632032ff-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.733 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:36:31.733 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.734 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.734 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:31.735 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:31.735 DEBUG:boto:Method: GET 2022-09-13T07:36:31.736 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:31.736 DEBUG:boto:Data: 2022-09-13T07:36:31.737 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.737 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.738 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.738 DEBUG:boto:Params: {} 2022-09-13T07:36:31.739 DEBUG:boto:Token: None 2022-09-13T07:36:31.740 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/?versions 2022-09-13T07:36:31.740 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oV8hKXoL3j+5MTq0JWJgtY5476c= 2022-09-13T07:36:31.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oV8hKXoL3j+5MTq0JWJgtY5476c=', 'Content-Length': '0'} 2022-09-13T07:36:31.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000399931b30ab7b51e-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.744 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.745 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:36:31.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:36:31.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:36:31.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:36:31.748 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:36:31.748 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.749 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.749 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:31.750 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:31.750 DEBUG:boto:Method: GET 2022-09-13T07:36:31.751 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:31.751 DEBUG:boto:Data: 2022-09-13T07:36:31.752 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.752 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:31.753 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.753 DEBUG:boto:Params: {} 2022-09-13T07:36:31.754 DEBUG:boto:Token: None 2022-09-13T07:36:31.754 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/?versions 2022-09-13T07:36:31.755 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oV8hKXoL3j+5MTq0JWJgtY5476c= 2022-09-13T07:36:31.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oV8hKXoL3j+5MTq0JWJgtY5476c=', 'Content-Length': '0'} 2022-09-13T07:36:31.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001650782920c00f63-00632032ff-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.759 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.760 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:36:31.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:36:31.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:36:31.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:36:31.762 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.762 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.763 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:31.763 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:31.764 DEBUG:boto:Method: GET 2022-09-13T07:36:31.764 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:31.765 DEBUG:boto:Data: 2022-09-13T07:36:31.766 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.766 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.767 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.767 DEBUG:boto:Params: {} 2022-09-13T07:36:31.768 DEBUG:boto:Token: None 2022-09-13T07:36:31.768 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/?versions 2022-09-13T07:36:31.769 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oV8hKXoL3j+5MTq0JWJgtY5476c= 2022-09-13T07:36:31.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oV8hKXoL3j+5MTq0JWJgtY5476c=', 'Content-Length': '0'} 2022-09-13T07:36:31.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004568f4a6c589396b-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.773 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.774 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.774 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.775 DEBUG:boto:path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:31.775 DEBUG:boto:auth_path=/yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:31.776 DEBUG:boto:Method: GET 2022-09-13T07:36:31.776 DEBUG:boto:Path: /yxvvuh-17/?versions&max-keys=999 2022-09-13T07:36:31.777 DEBUG:boto:Data: 2022-09-13T07:36:31.777 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.778 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:31.778 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.779 DEBUG:boto:Params: {} 2022-09-13T07:36:31.779 DEBUG:boto:Token: None 2022-09-13T07:36:31.780 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/?versions 2022-09-13T07:36:31.780 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oV8hKXoL3j+5MTq0JWJgtY5476c= 2022-09-13T07:36:31.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oV8hKXoL3j+5MTq0JWJgtY5476c=', 'Content-Length': '0'} 2022-09-13T07:36:31.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000157d892173761fe0-00632032ff-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.785 DEBUG:boto:b'yxvvuh-17999falsedummydNvJ60bAVxEbh1MfNDsR8pOChcXeHcZtrue2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rwtrue2022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnutrue2022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F24PQwoG4f.rI.lLrfAU7czV9DNK1M3true2022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:31.786 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:31.787 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:31.787 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.788 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.788 DEBUG:boto:Method: GET 2022-09-13T07:36:31.789 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.789 DEBUG:boto:Data: 2022-09-13T07:36:31.790 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.790 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.791 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.791 DEBUG:boto:Params: {} 2022-09-13T07:36:31.792 DEBUG:boto:Token: None 2022-09-13T07:36:31.792 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.793 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3sqxm9D3fqWL9UyG0AyU75Uy/fY= 2022-09-13T07:36:31.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3sqxm9D3fqWL9UyG0AyU75Uy/fY=', 'Content-Length': '0'} 2022-09-13T07:36:31.796 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008932e065b18b261c-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.796 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:31.797 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:31.797 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.798 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.798 DEBUG:boto:Method: GET 2022-09-13T07:36:31.799 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.799 DEBUG:boto:Data: 2022-09-13T07:36:31.800 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.800 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:31.801 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.801 DEBUG:boto:Params: {} 2022-09-13T07:36:31.802 DEBUG:boto:Token: None 2022-09-13T07:36:31.803 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.803 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3sqxm9D3fqWL9UyG0AyU75Uy/fY= 2022-09-13T07:36:31.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3sqxm9D3fqWL9UyG0AyU75Uy/fY=', 'Content-Length': '0'} 2022-09-13T07:36:31.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000070d37961d43ff84d-00632032ff-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.807 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:31.807 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:31.808 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.808 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.809 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.809 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.809 DEBUG:boto:Data: 2022-09-13T07:36:31.810 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.810 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.811 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.811 DEBUG:boto:Params: {} 2022-09-13T07:36:31.812 DEBUG:boto:Token: None 2022-09-13T07:36:31.813 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.813 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O+Y6HpOGV/hzPqVKRloO0Ixle/4= 2022-09-13T07:36:31.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O+Y6HpOGV/hzPqVKRloO0Ixle/4=', 'Content-Length': '0'} 2022-09-13T07:36:31.816 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000709504904c6afefe-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.816 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:31.817 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:31.817 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.818 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.818 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.819 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.819 DEBUG:boto:Data: 2022-09-13T07:36:31.820 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.820 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:31.821 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.821 DEBUG:boto:Params: {} 2022-09-13T07:36:31.822 DEBUG:boto:Token: None 2022-09-13T07:36:31.822 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.823 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O+Y6HpOGV/hzPqVKRloO0Ixle/4= 2022-09-13T07:36:31.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O+Y6HpOGV/hzPqVKRloO0Ixle/4=', 'Content-Length': '0'} 2022-09-13T07:36:31.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c0c16c11b23987c3-00632032ff-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:31.827 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:31.827 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:31.828 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.828 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.829 DEBUG:boto:Method: GET 2022-09-13T07:36:31.829 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.830 DEBUG:boto:Data: 2022-09-13T07:36:31.830 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.831 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.831 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.832 DEBUG:boto:Params: {} 2022-09-13T07:36:31.833 DEBUG:boto:Token: None 2022-09-13T07:36:31.833 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.834 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3sqxm9D3fqWL9UyG0AyU75Uy/fY= 2022-09-13T07:36:31.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3sqxm9D3fqWL9UyG0AyU75Uy/fY=', 'Content-Length': '0'} 2022-09-13T07:36:31.836 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c785d9132ac8d5a3-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.837 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:31.838 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:31.838 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.839 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.839 DEBUG:boto:Method: GET 2022-09-13T07:36:31.840 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.840 DEBUG:boto:Data: 2022-09-13T07:36:31.841 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.841 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:31.842 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.842 DEBUG:boto:Params: {} 2022-09-13T07:36:31.843 DEBUG:boto:Token: None 2022-09-13T07:36:31.843 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.844 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3sqxm9D3fqWL9UyG0AyU75Uy/fY= 2022-09-13T07:36:31.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3sqxm9D3fqWL9UyG0AyU75Uy/fY=', 'Content-Length': '0'} 2022-09-13T07:36:31.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000021b1f76a4f0d687f-00632032ff-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.847 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.848 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.848 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:31.849 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:31.849 DEBUG:boto:Method: GET 2022-09-13T07:36:31.850 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:36:31.850 DEBUG:boto:Data: 2022-09-13T07:36:31.851 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.851 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.852 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.852 DEBUG:boto:Params: {} 2022-09-13T07:36:31.853 DEBUG:boto:Token: None 2022-09-13T07:36:31.854 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.854 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b727fe6968c2500e-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.902 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.904 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.905 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.906 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:31.907 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:31.908 DEBUG:boto:Method: GET 2022-09-13T07:36:31.909 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:36:31.910 DEBUG:boto:Data: 2022-09-13T07:36:31.911 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.912 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:31.913 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.913 DEBUG:boto:Params: {} 2022-09-13T07:36:31.914 DEBUG:boto:Token: None 2022-09-13T07:36:31.915 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.916 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027729cc88d5c9737-00632032ff-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.920 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.921 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:31.922 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:31.923 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.924 DEBUG:boto:Path: /yxvvuh-17/dummy 2022-09-13T07:36:31.925 DEBUG:boto:Data: 2022-09-13T07:36:31.926 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.927 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.928 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.929 DEBUG:boto:Params: {} 2022-09-13T07:36:31.930 DEBUG:boto:Token: None 2022-09-13T07:36:31.931 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/dummy 2022-09-13T07:36:31.932 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WtVooB1fsSVIzmqpPSV3ov5Pq60= 2022-09-13T07:36:31.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WtVooB1fsSVIzmqpPSV3ov5Pq60=', 'Content-Length': '0'} 2022-09-13T07:36:31.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002f2fa9fa87c3059b-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.937 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:31.938 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:31.939 DEBUG:boto:Method: HEAD 2022-09-13T07:36:31.940 DEBUG:boto:Path: /yxvvuh-17/dummy 2022-09-13T07:36:31.941 DEBUG:boto:Data: 2022-09-13T07:36:31.942 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.942 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:31.943 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.943 DEBUG:boto:Params: {} 2022-09-13T07:36:31.944 DEBUG:boto:Token: None 2022-09-13T07:36:31.944 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/dummy 2022-09-13T07:36:31.945 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WtVooB1fsSVIzmqpPSV3ov5Pq60= 2022-09-13T07:36:31.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WtVooB1fsSVIzmqpPSV3ov5Pq60=', 'Content-Length': '0'} 2022-09-13T07:36:31.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009f6af39ecf973806-00632032ff-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:31.949 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:31.950 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:31.950 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.951 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.951 DEBUG:boto:Method: GET 2022-09-13T07:36:31.952 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.952 DEBUG:boto:Data: 2022-09-13T07:36:31.953 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.953 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.954 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.954 DEBUG:boto:Params: {} 2022-09-13T07:36:31.955 DEBUG:boto:Token: None 2022-09-13T07:36:31.956 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.956 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3sqxm9D3fqWL9UyG0AyU75Uy/fY= 2022-09-13T07:36:31.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3sqxm9D3fqWL9UyG0AyU75Uy/fY=', 'Content-Length': '0'} 2022-09-13T07:36:31.959 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c05c9636fba4a50d-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.960 DEBUG:boto:path=/yxvvuh-17/dummy 2022-09-13T07:36:31.960 DEBUG:boto:auth_path=/yxvvuh-17/dummy 2022-09-13T07:36:31.961 DEBUG:boto:path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.961 DEBUG:boto:auth_path=/yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.962 DEBUG:boto:Method: GET 2022-09-13T07:36:31.962 DEBUG:boto:Path: /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.963 DEBUG:boto:Data: 2022-09-13T07:36:31.963 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.964 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:31.964 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.965 DEBUG:boto:Params: {} 2022-09-13T07:36:31.966 DEBUG:boto:Token: None 2022-09-13T07:36:31.966 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/dummy?versionId=dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ 2022-09-13T07:36:31.967 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3sqxm9D3fqWL9UyG0AyU75Uy/fY= 2022-09-13T07:36:31.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3sqxm9D3fqWL9UyG0AyU75Uy/fY=', 'Content-Length': '0'} 2022-09-13T07:36:31.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:30 GMT'), ('x-amz-version-id', 'dNvJ60bAVxEbh1MfNDsR8pOChcXeHcZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000df8d3980e3431815-00632032ff-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.971 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.971 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.972 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:31.972 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:31.973 DEBUG:boto:Method: GET 2022-09-13T07:36:31.973 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:36:31.974 DEBUG:boto:Data: 2022-09-13T07:36:31.974 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.975 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:31.975 DEBUG:boto:Port: 8001 2022-09-13T07:36:31.976 DEBUG:boto:Params: {} 2022-09-13T07:36:31.977 DEBUG:boto:Token: None 2022-09-13T07:36:31.977 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.978 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f87a88ac877a43c-00632032ff-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:31.982 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:31.983 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:31.984 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:31.984 DEBUG:boto:path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:31.985 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=dummy 2022-09-13T07:36:31.985 DEBUG:boto:Method: GET 2022-09-13T07:36:31.986 DEBUG:boto:Path: /yxvvuh-17/?prefix=dummy 2022-09-13T07:36:31.987 DEBUG:boto:Data: 2022-09-13T07:36:31.987 DEBUG:boto:Headers: {} 2022-09-13T07:36:31.988 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:31.988 DEBUG:boto:Port: 8000 2022-09-13T07:36:31.989 DEBUG:boto:Params: {} 2022-09-13T07:36:31.989 DEBUG:boto:Token: None 2022-09-13T07:36:31.990 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:31 GMT /yxvvuh-17/ 2022-09-13T07:36:31.990 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg= 2022-09-13T07:36:31.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7X6rvpzXk+mURxEoB/NhAkPW+Qg=', 'Content-Length': '0'} 2022-09-13T07:36:31.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ad5f87c4002ed42-00632032ff-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.038 DEBUG:boto:b'yxvvuh-17dummy1000falsedummy2022-09-13T07:31:30.940Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:32.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:32.040 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.041 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.042 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.043 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.044 DEBUG:boto:Method: GET 2022-09-13T07:36:32.044 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.045 DEBUG:boto:Data: 2022-09-13T07:36:32.046 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.047 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.048 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.048 DEBUG:boto:Params: {} 2022-09-13T07:36:32.049 DEBUG:boto:Token: None 2022-09-13T07:36:32.050 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.051 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qwKTIzYS+a8Yr6Omu1nFR2J/CwU= 2022-09-13T07:36:32.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qwKTIzYS+a8Yr6Omu1nFR2J/CwU=', 'Content-Length': '0'} 2022-09-13T07:36:32.055 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbd3e36558a00d67-0063203300-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.056 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.057 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.058 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.058 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.059 DEBUG:boto:Method: GET 2022-09-13T07:36:32.060 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.061 DEBUG:boto:Data: 2022-09-13T07:36:32.062 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.062 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.063 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.064 DEBUG:boto:Params: {} 2022-09-13T07:36:32.065 DEBUG:boto:Token: None 2022-09-13T07:36:32.066 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.067 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qwKTIzYS+a8Yr6Omu1nFR2J/CwU= 2022-09-13T07:36:32.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qwKTIzYS+a8Yr6Omu1nFR2J/CwU=', 'Content-Length': '0'} 2022-09-13T07:36:32.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000277dd2264ce8dcc9-0063203300-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.071 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.072 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.073 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.074 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.074 DEBUG:boto:Method: HEAD 2022-09-13T07:36:32.075 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.076 DEBUG:boto:Data: 2022-09-13T07:36:32.077 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.077 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.078 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.079 DEBUG:boto:Params: {} 2022-09-13T07:36:32.080 DEBUG:boto:Token: None 2022-09-13T07:36:32.080 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.081 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2ar4h0MixrvHXz1FJwgEvy+qhO0= 2022-09-13T07:36:32.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2ar4h0MixrvHXz1FJwgEvy+qhO0=', 'Content-Length': '0'} 2022-09-13T07:36:32.083 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000699a3995f20e1bec-0063203300-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.084 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.085 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.085 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.086 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.086 DEBUG:boto:Method: HEAD 2022-09-13T07:36:32.087 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.087 DEBUG:boto:Data: 2022-09-13T07:36:32.088 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.088 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.089 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.089 DEBUG:boto:Params: {} 2022-09-13T07:36:32.090 DEBUG:boto:Token: None 2022-09-13T07:36:32.090 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.091 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2ar4h0MixrvHXz1FJwgEvy+qhO0= 2022-09-13T07:36:32.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2ar4h0MixrvHXz1FJwgEvy+qhO0=', 'Content-Length': '0'} 2022-09-13T07:36:32.094 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e91ab86c0e3c628-0063203300-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:32.095 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.096 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.096 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.097 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.097 DEBUG:boto:Method: GET 2022-09-13T07:36:32.098 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.098 DEBUG:boto:Data: 2022-09-13T07:36:32.099 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.099 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.100 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.100 DEBUG:boto:Params: {} 2022-09-13T07:36:32.101 DEBUG:boto:Token: None 2022-09-13T07:36:32.101 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.102 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qwKTIzYS+a8Yr6Omu1nFR2J/CwU= 2022-09-13T07:36:32.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qwKTIzYS+a8Yr6Omu1nFR2J/CwU=', 'Content-Length': '0'} 2022-09-13T07:36:32.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000727939199f444b88-0063203300-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.106 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.106 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.107 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.107 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.108 DEBUG:boto:Method: GET 2022-09-13T07:36:32.108 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.109 DEBUG:boto:Data: 2022-09-13T07:36:32.109 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.110 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.111 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.111 DEBUG:boto:Params: {} 2022-09-13T07:36:32.112 DEBUG:boto:Token: None 2022-09-13T07:36:32.112 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.113 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qwKTIzYS+a8Yr6Omu1nFR2J/CwU= 2022-09-13T07:36:32.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qwKTIzYS+a8Yr6Omu1nFR2J/CwU=', 'Content-Length': '0'} 2022-09-13T07:36:32.116 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6d482cc18600249-0063203300-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.116 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:32.117 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:32.117 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:32.118 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:32.118 DEBUG:boto:Method: GET 2022-09-13T07:36:32.119 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:32.119 DEBUG:boto:Data: 2022-09-13T07:36:32.120 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.120 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.121 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.122 DEBUG:boto:Params: {} 2022-09-13T07:36:32.122 DEBUG:boto:Token: None 2022-09-13T07:36:32.123 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/ 2022-09-13T07:36:32.123 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ= 2022-09-13T07:36:32.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ=', 'Content-Length': '0'} 2022-09-13T07:36:32.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad3071705a101adb-0063203300-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.128 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:32.128 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:32.129 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:32.130 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:32.130 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:32.131 DEBUG:boto:Method: GET 2022-09-13T07:36:32.131 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:32.132 DEBUG:boto:Data: 2022-09-13T07:36:32.132 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.133 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.133 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.134 DEBUG:boto:Params: {} 2022-09-13T07:36:32.135 DEBUG:boto:Token: None 2022-09-13T07:36:32.135 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/ 2022-09-13T07:36:32.136 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ= 2022-09-13T07:36:32.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ=', 'Content-Length': '0'} 2022-09-13T07:36:32.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031d869fd26cbd08e-0063203300-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.140 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:32.141 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.141 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.142 DEBUG:boto:Method: HEAD 2022-09-13T07:36:32.142 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.143 DEBUG:boto:Data: 2022-09-13T07:36:32.143 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.144 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.144 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.145 DEBUG:boto:Params: {} 2022-09-13T07:36:32.145 DEBUG:boto:Token: None 2022-09-13T07:36:32.146 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.146 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jf4xp1q5Pgxq1TM2Fy1eCOGfecw= 2022-09-13T07:36:32.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jf4xp1q5Pgxq1TM2Fy1eCOGfecw=', 'Content-Length': '0'} 2022-09-13T07:36:32.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059b72310a5e33889-0063203300-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.151 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.151 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.152 DEBUG:boto:Method: HEAD 2022-09-13T07:36:32.152 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.153 DEBUG:boto:Data: 2022-09-13T07:36:32.153 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.154 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.154 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.155 DEBUG:boto:Params: {} 2022-09-13T07:36:32.155 DEBUG:boto:Token: None 2022-09-13T07:36:32.156 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.156 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jf4xp1q5Pgxq1TM2Fy1eCOGfecw= 2022-09-13T07:36:32.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jf4xp1q5Pgxq1TM2Fy1eCOGfecw=', 'Content-Length': '0'} 2022-09-13T07:36:32.159 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0e41550b8d90cfd-0063203300-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:32.160 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.161 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.161 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.162 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.162 DEBUG:boto:Method: GET 2022-09-13T07:36:32.163 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.164 DEBUG:boto:Data: 2022-09-13T07:36:32.164 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.165 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.165 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.166 DEBUG:boto:Params: {} 2022-09-13T07:36:32.166 DEBUG:boto:Token: None 2022-09-13T07:36:32.167 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.168 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qwKTIzYS+a8Yr6Omu1nFR2J/CwU= 2022-09-13T07:36:32.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qwKTIzYS+a8Yr6Omu1nFR2J/CwU=', 'Content-Length': '0'} 2022-09-13T07:36:32.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064cd01278adb3b8e-0063203300-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.214 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.215 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1 2022-09-13T07:36:32.215 DEBUG:boto:path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.216 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.216 DEBUG:boto:Method: GET 2022-09-13T07:36:32.217 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.217 DEBUG:boto:Data: 2022-09-13T07:36:32.218 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.219 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.219 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.220 DEBUG:boto:Params: {} 2022-09-13T07:36:32.220 DEBUG:boto:Token: None 2022-09-13T07:36:32.221 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone1?versionId=pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw 2022-09-13T07:36:32.222 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qwKTIzYS+a8Yr6Omu1nFR2J/CwU= 2022-09-13T07:36:32.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qwKTIzYS+a8Yr6Omu1nFR2J/CwU=', 'Content-Length': '0'} 2022-09-13T07:36:32.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'pUOFwmwcD7Mg4skbdqKhSQ9FnXXx0Rw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cefe39afed06ab9-0063203300-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.226 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:32.226 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:32.227 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:32.228 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:32.228 DEBUG:boto:Method: GET 2022-09-13T07:36:32.229 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:32.229 DEBUG:boto:Data: 2022-09-13T07:36:32.230 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.230 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.231 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.231 DEBUG:boto:Params: {} 2022-09-13T07:36:32.232 DEBUG:boto:Token: None 2022-09-13T07:36:32.233 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/ 2022-09-13T07:36:32.234 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ= 2022-09-13T07:36:32.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ=', 'Content-Length': '0'} 2022-09-13T07:36:32.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d7159cf63b973eb-0063203300-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.238 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:32.239 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:32.239 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:32.240 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:32.240 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:32.241 DEBUG:boto:Method: GET 2022-09-13T07:36:32.241 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone1 2022-09-13T07:36:32.242 DEBUG:boto:Data: 2022-09-13T07:36:32.243 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.243 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.244 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.245 DEBUG:boto:Params: {} 2022-09-13T07:36:32.246 DEBUG:boto:Token: None 2022-09-13T07:36:32.246 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/ 2022-09-13T07:36:32.247 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ= 2022-09-13T07:36:32.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ=', 'Content-Length': '0'} 2022-09-13T07:36:32.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056338f7fcd575a04-0063203300-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.252 DEBUG:boto:b'yxvvuh-17obj-test-zone11000falseobj-test-zone12022-09-13T07:32:06.955Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:32.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:32.253 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.254 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.255 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.256 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.257 DEBUG:boto:Method: GET 2022-09-13T07:36:32.257 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.258 DEBUG:boto:Data: 2022-09-13T07:36:32.258 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.259 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.259 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.260 DEBUG:boto:Params: {} 2022-09-13T07:36:32.261 DEBUG:boto:Token: None 2022-09-13T07:36:32.262 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.262 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:n87apw2qwneq06pygGGiE9mYf8Q= 2022-09-13T07:36:32.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:n87apw2qwneq06pygGGiE9mYf8Q=', 'Content-Length': '0'} 2022-09-13T07:36:32.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cafd6c165b9422b5-0063203300-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.267 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.267 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.268 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.268 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.269 DEBUG:boto:Method: GET 2022-09-13T07:36:32.270 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.271 DEBUG:boto:Data: 2022-09-13T07:36:32.271 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.272 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.273 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.273 DEBUG:boto:Params: {} 2022-09-13T07:36:32.274 DEBUG:boto:Token: None 2022-09-13T07:36:32.275 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.276 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:n87apw2qwneq06pygGGiE9mYf8Q= 2022-09-13T07:36:32.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:n87apw2qwneq06pygGGiE9mYf8Q=', 'Content-Length': '0'} 2022-09-13T07:36:32.278 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4b2eb5fca2df6e2-0063203300-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.280 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.281 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.281 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.282 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.283 DEBUG:boto:Method: HEAD 2022-09-13T07:36:32.283 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.284 DEBUG:boto:Data: 2022-09-13T07:36:32.284 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.285 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.285 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.286 DEBUG:boto:Params: {} 2022-09-13T07:36:32.287 DEBUG:boto:Token: None 2022-09-13T07:36:32.288 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.289 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:U/Yd6pidol5cdKyGqNiYpHLQ1zY= 2022-09-13T07:36:32.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:U/Yd6pidol5cdKyGqNiYpHLQ1zY=', 'Content-Length': '0'} 2022-09-13T07:36:32.293 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000014f57d2fae09efc-0063203300-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.294 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.295 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.296 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.297 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.298 DEBUG:boto:Method: HEAD 2022-09-13T07:36:32.299 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.300 DEBUG:boto:Data: 2022-09-13T07:36:32.300 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.301 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.301 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.302 DEBUG:boto:Params: {} 2022-09-13T07:36:32.303 DEBUG:boto:Token: None 2022-09-13T07:36:32.303 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.304 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:U/Yd6pidol5cdKyGqNiYpHLQ1zY= 2022-09-13T07:36:32.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:U/Yd6pidol5cdKyGqNiYpHLQ1zY=', 'Content-Length': '0'} 2022-09-13T07:36:32.307 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000548aecea8765d899-0063203300-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:32.308 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.308 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.309 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.309 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.310 DEBUG:boto:Method: GET 2022-09-13T07:36:32.311 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.311 DEBUG:boto:Data: 2022-09-13T07:36:32.312 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.312 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.313 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.313 DEBUG:boto:Params: {} 2022-09-13T07:36:32.314 DEBUG:boto:Token: None 2022-09-13T07:36:32.315 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.315 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:n87apw2qwneq06pygGGiE9mYf8Q= 2022-09-13T07:36:32.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:n87apw2qwneq06pygGGiE9mYf8Q=', 'Content-Length': '0'} 2022-09-13T07:36:32.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0b32ecdc4c2ba94-0063203300-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.319 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.319 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.320 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.320 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.321 DEBUG:boto:Method: GET 2022-09-13T07:36:32.321 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.322 DEBUG:boto:Data: 2022-09-13T07:36:32.322 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.323 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.324 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.324 DEBUG:boto:Params: {} 2022-09-13T07:36:32.325 DEBUG:boto:Token: None 2022-09-13T07:36:32.325 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.326 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:n87apw2qwneq06pygGGiE9mYf8Q= 2022-09-13T07:36:32.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:n87apw2qwneq06pygGGiE9mYf8Q=', 'Content-Length': '0'} 2022-09-13T07:36:32.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051bd69cc0de55368-0063203300-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.370 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:32.371 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:32.371 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:32.372 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:32.373 DEBUG:boto:Method: GET 2022-09-13T07:36:32.373 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:32.374 DEBUG:boto:Data: 2022-09-13T07:36:32.375 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.375 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.376 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.376 DEBUG:boto:Params: {} 2022-09-13T07:36:32.377 DEBUG:boto:Token: None 2022-09-13T07:36:32.378 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/ 2022-09-13T07:36:32.378 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ= 2022-09-13T07:36:32.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ=', 'Content-Length': '0'} 2022-09-13T07:36:32.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062728f4a1471f69d-0063203300-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.383 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:32.384 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:32.384 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:32.385 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:32.385 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:32.386 DEBUG:boto:Method: GET 2022-09-13T07:36:32.386 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:32.387 DEBUG:boto:Data: 2022-09-13T07:36:32.387 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.388 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.388 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.389 DEBUG:boto:Params: {} 2022-09-13T07:36:32.390 DEBUG:boto:Token: None 2022-09-13T07:36:32.390 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/ 2022-09-13T07:36:32.391 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ= 2022-09-13T07:36:32.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ=', 'Content-Length': '0'} 2022-09-13T07:36:32.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004971ab2a65738bb7-0063203300-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.395 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:32.395 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.396 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.397 DEBUG:boto:Method: HEAD 2022-09-13T07:36:32.397 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.398 DEBUG:boto:Data: 2022-09-13T07:36:32.398 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.399 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.400 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.401 DEBUG:boto:Params: {} 2022-09-13T07:36:32.402 DEBUG:boto:Token: None 2022-09-13T07:36:32.403 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.403 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:U9G+WNfggUdjGHQjDa3KeKTFMZ8= 2022-09-13T07:36:32.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:U9G+WNfggUdjGHQjDa3KeKTFMZ8=', 'Content-Length': '0'} 2022-09-13T07:36:32.407 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a70863af937bd4b-0063203300-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.408 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.409 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.410 DEBUG:boto:Method: HEAD 2022-09-13T07:36:32.411 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.412 DEBUG:boto:Data: 2022-09-13T07:36:32.413 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.413 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.414 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.414 DEBUG:boto:Params: {} 2022-09-13T07:36:32.415 DEBUG:boto:Token: None 2022-09-13T07:36:32.415 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.416 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:U9G+WNfggUdjGHQjDa3KeKTFMZ8= 2022-09-13T07:36:32.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:U9G+WNfggUdjGHQjDa3KeKTFMZ8=', 'Content-Length': '0'} 2022-09-13T07:36:32.419 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b2d6706f6d8aa23-0063203300-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:32.420 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.421 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.421 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.422 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.423 DEBUG:boto:Method: GET 2022-09-13T07:36:32.424 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.425 DEBUG:boto:Data: 2022-09-13T07:36:32.426 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.427 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.428 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.428 DEBUG:boto:Params: {} 2022-09-13T07:36:32.429 DEBUG:boto:Token: None 2022-09-13T07:36:32.429 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.430 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:n87apw2qwneq06pygGGiE9mYf8Q= 2022-09-13T07:36:32.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:n87apw2qwneq06pygGGiE9mYf8Q=', 'Content-Length': '0'} 2022-09-13T07:36:32.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4680f5945f2b725-0063203300-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.474 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.475 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2 2022-09-13T07:36:32.475 DEBUG:boto:path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.476 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.476 DEBUG:boto:Method: GET 2022-09-13T07:36:32.477 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.477 DEBUG:boto:Data: 2022-09-13T07:36:32.478 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.478 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.479 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.479 DEBUG:boto:Params: {} 2022-09-13T07:36:32.480 DEBUG:boto:Token: None 2022-09-13T07:36:32.481 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone2?versionId=JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu 2022-09-13T07:36:32.481 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:n87apw2qwneq06pygGGiE9mYf8Q= 2022-09-13T07:36:32.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:n87apw2qwneq06pygGGiE9mYf8Q=', 'Content-Length': '0'} 2022-09-13T07:36:32.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'JIvSHKrPzfYfnE-.nZoVNNLKwdMQSnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002255eeed661afe05-0063203300-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.486 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:32.487 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:32.488 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:32.488 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:32.489 DEBUG:boto:Method: GET 2022-09-13T07:36:32.489 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:32.490 DEBUG:boto:Data: 2022-09-13T07:36:32.490 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.491 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.491 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.492 DEBUG:boto:Params: {} 2022-09-13T07:36:32.492 DEBUG:boto:Token: None 2022-09-13T07:36:32.493 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/ 2022-09-13T07:36:32.493 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ= 2022-09-13T07:36:32.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ=', 'Content-Length': '0'} 2022-09-13T07:36:32.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a919f0c8a7b8ee19-0063203300-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.497 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:32.500 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:32.500 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:32.501 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:32.503 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:32.506 DEBUG:boto:Method: GET 2022-09-13T07:36:32.507 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone2 2022-09-13T07:36:32.507 DEBUG:boto:Data: 2022-09-13T07:36:32.508 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.509 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.509 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.510 DEBUG:boto:Params: {} 2022-09-13T07:36:32.510 DEBUG:boto:Token: None 2022-09-13T07:36:32.511 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/ 2022-09-13T07:36:32.512 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ= 2022-09-13T07:36:32.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ=', 'Content-Length': '0'} 2022-09-13T07:36:32.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012e8b0b689b0cd4c-0063203300-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.516 DEBUG:boto:b'yxvvuh-17obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:32.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:32.517 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.518 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.518 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.519 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.519 DEBUG:boto:Method: GET 2022-09-13T07:36:32.520 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.520 DEBUG:boto:Data: 2022-09-13T07:36:32.521 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.521 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.522 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.522 DEBUG:boto:Params: {} 2022-09-13T07:36:32.523 DEBUG:boto:Token: None 2022-09-13T07:36:32.524 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.524 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pcu6Oc0z6Uv49Z6xMC6rIeMQno8= 2022-09-13T07:36:32.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pcu6Oc0z6Uv49Z6xMC6rIeMQno8=', 'Content-Length': '0'} 2022-09-13T07:36:32.527 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c388ff9ea06b585f-0063203300-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.528 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.528 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.529 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.529 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.530 DEBUG:boto:Method: GET 2022-09-13T07:36:32.530 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.531 DEBUG:boto:Data: 2022-09-13T07:36:32.532 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.532 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.533 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.533 DEBUG:boto:Params: {} 2022-09-13T07:36:32.534 DEBUG:boto:Token: None 2022-09-13T07:36:32.535 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.535 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pcu6Oc0z6Uv49Z6xMC6rIeMQno8= 2022-09-13T07:36:32.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pcu6Oc0z6Uv49Z6xMC6rIeMQno8=', 'Content-Length': '0'} 2022-09-13T07:36:32.538 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000806817bd623ee2e2-0063203300-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.582 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.583 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.583 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.584 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.585 DEBUG:boto:Method: HEAD 2022-09-13T07:36:32.585 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.586 DEBUG:boto:Data: 2022-09-13T07:36:32.586 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.587 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.587 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.588 DEBUG:boto:Params: {} 2022-09-13T07:36:32.589 DEBUG:boto:Token: None 2022-09-13T07:36:32.589 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.590 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S3/1r1J3mrB/6dvQ+7Bb+CvKW3c= 2022-09-13T07:36:32.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S3/1r1J3mrB/6dvQ+7Bb+CvKW3c=', 'Content-Length': '0'} 2022-09-13T07:36:32.592 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002079ffc55126628d-0063203300-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.593 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.594 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.594 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.595 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.596 DEBUG:boto:Method: HEAD 2022-09-13T07:36:32.596 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.597 DEBUG:boto:Data: 2022-09-13T07:36:32.597 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.598 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.599 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.599 DEBUG:boto:Params: {} 2022-09-13T07:36:32.600 DEBUG:boto:Token: None 2022-09-13T07:36:32.600 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.601 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S3/1r1J3mrB/6dvQ+7Bb+CvKW3c= 2022-09-13T07:36:32.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S3/1r1J3mrB/6dvQ+7Bb+CvKW3c=', 'Content-Length': '0'} 2022-09-13T07:36:32.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad5b1e34bcce57ae-0063203300-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:32.607 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.607 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.608 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.609 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.609 DEBUG:boto:Method: GET 2022-09-13T07:36:32.610 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.610 DEBUG:boto:Data: 2022-09-13T07:36:32.611 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.612 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.612 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.613 DEBUG:boto:Params: {} 2022-09-13T07:36:32.613 DEBUG:boto:Token: None 2022-09-13T07:36:32.614 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.614 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pcu6Oc0z6Uv49Z6xMC6rIeMQno8= 2022-09-13T07:36:32.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pcu6Oc0z6Uv49Z6xMC6rIeMQno8=', 'Content-Length': '0'} 2022-09-13T07:36:32.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037bb98c19b65e8e5-0063203300-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.618 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.619 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.620 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.621 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.621 DEBUG:boto:Method: GET 2022-09-13T07:36:32.622 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.622 DEBUG:boto:Data: 2022-09-13T07:36:32.623 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.624 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.633 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.634 DEBUG:boto:Params: {} 2022-09-13T07:36:32.635 DEBUG:boto:Token: None 2022-09-13T07:36:32.636 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.637 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pcu6Oc0z6Uv49Z6xMC6rIeMQno8= 2022-09-13T07:36:32.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pcu6Oc0z6Uv49Z6xMC6rIeMQno8=', 'Content-Length': '0'} 2022-09-13T07:36:32.639 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5f76492f1fdb774-0063203300-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.641 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:32.641 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:32.642 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:32.642 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:32.643 DEBUG:boto:Method: GET 2022-09-13T07:36:32.643 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:32.644 DEBUG:boto:Data: 2022-09-13T07:36:32.644 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.645 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.645 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.646 DEBUG:boto:Params: {} 2022-09-13T07:36:32.647 DEBUG:boto:Token: None 2022-09-13T07:36:32.647 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/ 2022-09-13T07:36:32.648 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ= 2022-09-13T07:36:32.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ=', 'Content-Length': '0'} 2022-09-13T07:36:32.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6570a873f90307c-0063203300-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.652 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:32.653 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:32.654 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:32.654 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:32.655 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:32.656 DEBUG:boto:Method: GET 2022-09-13T07:36:32.656 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:32.657 DEBUG:boto:Data: 2022-09-13T07:36:32.657 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.658 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.658 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.659 DEBUG:boto:Params: {} 2022-09-13T07:36:32.659 DEBUG:boto:Token: None 2022-09-13T07:36:32.660 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/ 2022-09-13T07:36:32.661 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ= 2022-09-13T07:36:32.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ=', 'Content-Length': '0'} 2022-09-13T07:36:32.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1e808d5c99b2599-0063203300-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.665 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:32.666 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.666 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.667 DEBUG:boto:Method: HEAD 2022-09-13T07:36:32.667 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.668 DEBUG:boto:Data: 2022-09-13T07:36:32.668 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.669 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.669 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.670 DEBUG:boto:Params: {} 2022-09-13T07:36:32.670 DEBUG:boto:Token: None 2022-09-13T07:36:32.671 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.672 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KNj65/oOPVt9rHCYmxbJxmpDF/0= 2022-09-13T07:36:32.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KNj65/oOPVt9rHCYmxbJxmpDF/0=', 'Content-Length': '0'} 2022-09-13T07:36:32.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e06cfc78a4b65cab-0063203300-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.677 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.677 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.678 DEBUG:boto:Method: HEAD 2022-09-13T07:36:32.679 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.680 DEBUG:boto:Data: 2022-09-13T07:36:32.681 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.681 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.682 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.683 DEBUG:boto:Params: {} 2022-09-13T07:36:32.684 DEBUG:boto:Token: None 2022-09-13T07:36:32.685 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.686 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KNj65/oOPVt9rHCYmxbJxmpDF/0= 2022-09-13T07:36:32.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KNj65/oOPVt9rHCYmxbJxmpDF/0=', 'Content-Length': '0'} 2022-09-13T07:36:32.691 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6d9c10d88bf2345-0063203300-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:32.693 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.695 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.696 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.697 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.697 DEBUG:boto:Method: GET 2022-09-13T07:36:32.698 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.699 DEBUG:boto:Data: 2022-09-13T07:36:32.700 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.701 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.702 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.703 DEBUG:boto:Params: {} 2022-09-13T07:36:32.704 DEBUG:boto:Token: None 2022-09-13T07:36:32.705 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.706 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pcu6Oc0z6Uv49Z6xMC6rIeMQno8= 2022-09-13T07:36:32.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pcu6Oc0z6Uv49Z6xMC6rIeMQno8=', 'Content-Length': '0'} 2022-09-13T07:36:32.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075b26725d05c03f4-0063203300-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.712 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.713 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3 2022-09-13T07:36:32.714 DEBUG:boto:path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.715 DEBUG:boto:auth_path=/yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.716 DEBUG:boto:Method: GET 2022-09-13T07:36:32.717 DEBUG:boto:Path: /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.717 DEBUG:boto:Data: 2022-09-13T07:36:32.718 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.719 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.720 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.721 DEBUG:boto:Params: {} 2022-09-13T07:36:32.722 DEBUG:boto:Token: None 2022-09-13T07:36:32.723 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/obj-test-zone3?versionId=F24PQwoG4f.rI.lLrfAU7czV9DNK1M3 2022-09-13T07:36:32.724 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pcu6Oc0z6Uv49Z6xMC6rIeMQno8= 2022-09-13T07:36:32.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pcu6Oc0z6Uv49Z6xMC6rIeMQno8=', 'Content-Length': '0'} 2022-09-13T07:36:32.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'F24PQwoG4f.rI.lLrfAU7czV9DNK1M3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8c3e1a9c51c55ea-0063203300-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.729 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:32.730 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:32.730 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:32.731 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:32.731 DEBUG:boto:Method: GET 2022-09-13T07:36:32.732 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:32.732 DEBUG:boto:Data: 2022-09-13T07:36:32.733 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.733 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:32.734 DEBUG:boto:Port: 8001 2022-09-13T07:36:32.734 DEBUG:boto:Params: {} 2022-09-13T07:36:32.735 DEBUG:boto:Token: None 2022-09-13T07:36:32.736 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/ 2022-09-13T07:36:32.736 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ= 2022-09-13T07:36:32.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ=', 'Content-Length': '0'} 2022-09-13T07:36:32.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cfe43355de62a21-0063203300-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.782 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:32.784 DEBUG:boto:path=/yxvvuh-17/ 2022-09-13T07:36:32.785 DEBUG:boto:auth_path=/yxvvuh-17/ 2022-09-13T07:36:32.786 DEBUG:boto:path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:32.787 DEBUG:boto:auth_path=/yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:32.788 DEBUG:boto:Method: GET 2022-09-13T07:36:32.788 DEBUG:boto:Path: /yxvvuh-17/?prefix=obj-test-zone3 2022-09-13T07:36:32.789 DEBUG:boto:Data: 2022-09-13T07:36:32.790 DEBUG:boto:Headers: {} 2022-09-13T07:36:32.791 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:32.792 DEBUG:boto:Port: 8000 2022-09-13T07:36:32.792 DEBUG:boto:Params: {} 2022-09-13T07:36:32.793 DEBUG:boto:Token: None 2022-09-13T07:36:32.794 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:32 GMT /yxvvuh-17/ 2022-09-13T07:36:32.795 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ= 2022-09-13T07:36:32.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p5Ev/r8zctdPqM6/xGMoB/YGTlQ=', 'Content-Length': '0'} 2022-09-13T07:36:32.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019a458cf82f4366e-0063203300-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:32.800 DEBUG:boto:b'yxvvuh-17obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.254Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:32.801 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-17 zones={test-zone3, test-zone2} 2022-09-13T07:36:32.801 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-18 2022-09-13T07:36:32.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:32.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:32.804 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:36:32.870 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:36:32.871 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:36:32.921 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:36:32.917+0000 7f0453ba1600 1 bucket sync caught up with source: 2022-09-13T07:36:32.921 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000011.29.5, , 00000000005.20.5, , 00000000010.46.5, , , , , ] 2022-09-13T07:36:32.922 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000011.29.5, , 00000000005.20.5, , 00000000010.46.5, , , , , ] 2022-09-13T07:36:32.922 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:36:32.917+0000 7f0453ba1600 0 bucket checkpoint complete 2022-09-13T07:36:32.934 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-18 2022-09-13T07:36:32.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:32.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:32.937 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:36:33.003 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:36:33.003 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:36:33.052 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:33.048+0000 7f41b70d1600 1 bucket sync caught up with source: 2022-09-13T07:36:33.052 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000011.29.5, , 00000000005.20.5, , 00000000010.46.5, , , , , ] 2022-09-13T07:36:33.052 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000011.29.5, , 00000000005.20.5, , 00000000010.46.5, , , , , ] 2022-09-13T07:36:33.053 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:33.048+0000 7f41b70d1600 0 bucket checkpoint complete 2022-09-13T07:36:33.069 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-18 2022-09-13T07:36:33.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:33.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:33.071 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:36:33.130 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:36:33.131 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:36:33.164 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:33.160+0000 7f83a76a9600 1 bucket sync caught up with source: 2022-09-13T07:36:33.165 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.22.5, , 00000000011.39.5, , 00000000008.55.5, , , , , ] 2022-09-13T07:36:33.166 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.22.5, , 00000000011.39.5, , 00000000008.55.5, , , , , ] 2022-09-13T07:36:33.166 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:33.160+0000 7f83a76a9600 0 bucket checkpoint complete 2022-09-13T07:36:33.177 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-18 2022-09-13T07:36:33.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:33.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:33.180 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:36:33.238 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:36:33.239 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:36:33.272 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:33.268+0000 7f8d23c79600 1 bucket sync caught up with source: 2022-09-13T07:36:33.272 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.22.5, , 00000000011.39.5, , 00000000008.55.5, , , , , ] 2022-09-13T07:36:33.272 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.22.5, , 00000000011.39.5, , 00000000008.55.5, , , , , ] 2022-09-13T07:36:33.273 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:33.268+0000 7f8d23c79600 0 bucket checkpoint complete 2022-09-13T07:36:33.284 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-18 2022-09-13T07:36:33.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:33.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:33.286 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:36:33.344 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:36:33.344 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:36:33.391 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:33.387+0000 7fb15fbcd600 1 bucket sync caught up with source: 2022-09-13T07:36:33.392 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, 00000000004.23.5, , 00000000005.25.5, , 00000000017.77.5, , , , , ] 2022-09-13T07:36:33.392 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, 00000000004.23.5, , 00000000005.25.5, , 00000000017.77.5, , , , , ] 2022-09-13T07:36:33.392 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:33.387+0000 7fb15fbcd600 0 bucket checkpoint complete 2022-09-13T07:36:33.404 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-18 2022-09-13T07:36:33.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:33.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:33.406 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:36:33.465 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:36:33.466 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:36:33.502 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:36:33.498+0000 7f1ab4a38600 1 bucket sync caught up with source: 2022-09-13T07:36:33.502 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, 00000000004.23.5, , 00000000005.25.5, , 00000000017.77.5, , , , , ] 2022-09-13T07:36:33.503 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, 00000000004.23.5, , 00000000005.25.5, , 00000000017.77.5, , , , , ] 2022-09-13T07:36:33.503 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:36:33.498+0000 7f1ab4a38600 0 bucket checkpoint complete 2022-09-13T07:36:33.518 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-18 zones={test-zone2, test-zone1} 2022-09-13T07:36:33.518 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:33.519 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:33.519 DEBUG:boto:Method: HEAD 2022-09-13T07:36:33.520 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:36:33.521 DEBUG:boto:Data: 2022-09-13T07:36:33.521 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.522 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.522 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.523 DEBUG:boto:Params: {} 2022-09-13T07:36:33.523 DEBUG:boto:Token: None 2022-09-13T07:36:33.524 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/ 2022-09-13T07:36:33.525 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TcYzlYTyvJcocO5wwNzPK9cJbfs= 2022-09-13T07:36:33.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TcYzlYTyvJcocO5wwNzPK9cJbfs=', 'Content-Length': '0'} 2022-09-13T07:36:33.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7a6362963c6522f-0063203301-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.528 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:33.529 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:33.529 DEBUG:boto:Method: HEAD 2022-09-13T07:36:33.530 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:36:33.530 DEBUG:boto:Data: 2022-09-13T07:36:33.531 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.531 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.532 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.532 DEBUG:boto:Params: {} 2022-09-13T07:36:33.533 DEBUG:boto:Token: None 2022-09-13T07:36:33.533 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/ 2022-09-13T07:36:33.534 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TcYzlYTyvJcocO5wwNzPK9cJbfs= 2022-09-13T07:36:33.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TcYzlYTyvJcocO5wwNzPK9cJbfs=', 'Content-Length': '0'} 2022-09-13T07:36:33.537 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e7aa3dc114d9427-0063203301-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.538 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:36:33.538 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:33.539 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:33.539 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:33.540 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:33.541 DEBUG:boto:Method: GET 2022-09-13T07:36:33.542 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:33.543 DEBUG:boto:Data: 2022-09-13T07:36:33.543 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.544 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.544 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.545 DEBUG:boto:Params: {} 2022-09-13T07:36:33.545 DEBUG:boto:Token: None 2022-09-13T07:36:33.546 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/?versions 2022-09-13T07:36:33.546 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wcbAfFwcKYCeVvRBGFJv4rOwh/U= 2022-09-13T07:36:33.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wcbAfFwcKYCeVvRBGFJv4rOwh/U=', 'Content-Length': '0'} 2022-09-13T07:36:33.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c8a6b965511de66-0063203301-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.550 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:33.551 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:36:33.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:36:33.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:36:33.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:36:33.553 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:36:33.554 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:33.554 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:33.555 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:33.555 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:33.556 DEBUG:boto:Method: GET 2022-09-13T07:36:33.556 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:33.557 DEBUG:boto:Data: 2022-09-13T07:36:33.557 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.558 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.558 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.559 DEBUG:boto:Params: {} 2022-09-13T07:36:33.559 DEBUG:boto:Token: None 2022-09-13T07:36:33.560 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/?versions 2022-09-13T07:36:33.561 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wcbAfFwcKYCeVvRBGFJv4rOwh/U= 2022-09-13T07:36:33.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wcbAfFwcKYCeVvRBGFJv4rOwh/U=', 'Content-Length': '0'} 2022-09-13T07:36:33.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df4bdc7eddc62f3e-0063203301-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.564 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:33.565 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:36:33.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:36:33.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:36:33.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:36:33.567 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:33.568 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:33.568 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:33.569 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:33.569 DEBUG:boto:Method: GET 2022-09-13T07:36:33.570 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:33.570 DEBUG:boto:Data: 2022-09-13T07:36:33.571 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.571 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.572 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.572 DEBUG:boto:Params: {} 2022-09-13T07:36:33.573 DEBUG:boto:Token: None 2022-09-13T07:36:33.574 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/?versions 2022-09-13T07:36:33.574 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wcbAfFwcKYCeVvRBGFJv4rOwh/U= 2022-09-13T07:36:33.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wcbAfFwcKYCeVvRBGFJv4rOwh/U=', 'Content-Length': '0'} 2022-09-13T07:36:33.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cbeee9ac34c1201-0063203301-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.578 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:33.579 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:33.580 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:33.580 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:33.581 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:33.581 DEBUG:boto:Method: GET 2022-09-13T07:36:33.582 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:33.582 DEBUG:boto:Data: 2022-09-13T07:36:33.583 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.584 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.584 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.585 DEBUG:boto:Params: {} 2022-09-13T07:36:33.585 DEBUG:boto:Token: None 2022-09-13T07:36:33.586 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/?versions 2022-09-13T07:36:33.587 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wcbAfFwcKYCeVvRBGFJv4rOwh/U= 2022-09-13T07:36:33.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wcbAfFwcKYCeVvRBGFJv4rOwh/U=', 'Content-Length': '0'} 2022-09-13T07:36:33.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3cd734c7b662ed2-0063203301-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.590 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:33.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:33.592 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:33.593 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:33.593 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.594 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.595 DEBUG:boto:Method: GET 2022-09-13T07:36:33.595 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.596 DEBUG:boto:Data: 2022-09-13T07:36:33.596 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.597 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.597 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.598 DEBUG:boto:Params: {} 2022-09-13T07:36:33.598 DEBUG:boto:Token: None 2022-09-13T07:36:33.599 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.600 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bUZxbECg0VzVpzkH4q7tNr5GaEQ= 2022-09-13T07:36:33.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bUZxbECg0VzVpzkH4q7tNr5GaEQ=', 'Content-Length': '0'} 2022-09-13T07:36:33.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000026b77d4196236471-0063203301-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.603 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:33.604 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:33.604 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.605 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.606 DEBUG:boto:Method: GET 2022-09-13T07:36:33.606 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.607 DEBUG:boto:Data: 2022-09-13T07:36:33.607 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.608 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.608 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.609 DEBUG:boto:Params: {} 2022-09-13T07:36:33.610 DEBUG:boto:Token: None 2022-09-13T07:36:33.610 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.611 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bUZxbECg0VzVpzkH4q7tNr5GaEQ= 2022-09-13T07:36:33.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bUZxbECg0VzVpzkH4q7tNr5GaEQ=', 'Content-Length': '0'} 2022-09-13T07:36:33.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000471d842d292c8b8c-0063203301-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.614 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:33.615 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:33.615 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.616 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.617 DEBUG:boto:Method: HEAD 2022-09-13T07:36:33.617 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.618 DEBUG:boto:Data: 2022-09-13T07:36:33.618 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.619 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.620 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.620 DEBUG:boto:Params: {} 2022-09-13T07:36:33.621 DEBUG:boto:Token: None 2022-09-13T07:36:33.621 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.622 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UI70skbuiDmjzK5h8Qy72a7LWyM= 2022-09-13T07:36:33.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UI70skbuiDmjzK5h8Qy72a7LWyM=', 'Content-Length': '0'} 2022-09-13T07:36:33.624 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000086ac297506fd3080-0063203301-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.625 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:33.626 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:33.626 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.627 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.628 DEBUG:boto:Method: HEAD 2022-09-13T07:36:33.628 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.629 DEBUG:boto:Data: 2022-09-13T07:36:33.629 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.630 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.630 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.631 DEBUG:boto:Params: {} 2022-09-13T07:36:33.631 DEBUG:boto:Token: None 2022-09-13T07:36:33.632 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.632 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UI70skbuiDmjzK5h8Qy72a7LWyM= 2022-09-13T07:36:33.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UI70skbuiDmjzK5h8Qy72a7LWyM=', 'Content-Length': '0'} 2022-09-13T07:36:33.635 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001ce72d51a7046f00-0063203301-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:33.636 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:33.637 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:33.638 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.638 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.639 DEBUG:boto:Method: GET 2022-09-13T07:36:33.639 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.640 DEBUG:boto:Data: 2022-09-13T07:36:33.640 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.641 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.641 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.642 DEBUG:boto:Params: {} 2022-09-13T07:36:33.643 DEBUG:boto:Token: None 2022-09-13T07:36:33.643 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.644 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bUZxbECg0VzVpzkH4q7tNr5GaEQ= 2022-09-13T07:36:33.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bUZxbECg0VzVpzkH4q7tNr5GaEQ=', 'Content-Length': '0'} 2022-09-13T07:36:33.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001a23ca115ebfba05-0063203301-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.647 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:33.648 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:33.648 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.649 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.650 DEBUG:boto:Method: GET 2022-09-13T07:36:33.650 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.651 DEBUG:boto:Data: 2022-09-13T07:36:33.651 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.652 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.652 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.653 DEBUG:boto:Params: {} 2022-09-13T07:36:33.653 DEBUG:boto:Token: None 2022-09-13T07:36:33.654 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.654 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bUZxbECg0VzVpzkH4q7tNr5GaEQ= 2022-09-13T07:36:33.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bUZxbECg0VzVpzkH4q7tNr5GaEQ=', 'Content-Length': '0'} 2022-09-13T07:36:33.657 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000441ec3ea6f26ac1b-0063203301-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.658 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:33.659 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:33.659 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:33.660 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:33.661 DEBUG:boto:Method: GET 2022-09-13T07:36:33.661 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:36:33.662 DEBUG:boto:Data: 2022-09-13T07:36:33.662 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.663 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.663 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.664 DEBUG:boto:Params: {} 2022-09-13T07:36:33.664 DEBUG:boto:Token: None 2022-09-13T07:36:33.665 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/ 2022-09-13T07:36:33.665 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ= 2022-09-13T07:36:33.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ=', 'Content-Length': '0'} 2022-09-13T07:36:33.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f149baf67a02059-0063203301-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.710 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:33.712 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:33.713 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:33.713 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:33.714 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:33.715 DEBUG:boto:Method: GET 2022-09-13T07:36:33.716 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:36:33.716 DEBUG:boto:Data: 2022-09-13T07:36:33.717 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.717 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.718 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.718 DEBUG:boto:Params: {} 2022-09-13T07:36:33.719 DEBUG:boto:Token: None 2022-09-13T07:36:33.720 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/ 2022-09-13T07:36:33.720 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ= 2022-09-13T07:36:33.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ=', 'Content-Length': '0'} 2022-09-13T07:36:33.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ee7cf028df1c7d0-0063203301-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.724 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:33.725 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:33.726 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:33.726 DEBUG:boto:Method: HEAD 2022-09-13T07:36:33.727 DEBUG:boto:Path: /yxvvuh-18/dummy 2022-09-13T07:36:33.727 DEBUG:boto:Data: 2022-09-13T07:36:33.728 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.728 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.729 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.729 DEBUG:boto:Params: {} 2022-09-13T07:36:33.730 DEBUG:boto:Token: None 2022-09-13T07:36:33.730 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/dummy 2022-09-13T07:36:33.731 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FWTGxYziCcGRjBisisKPji1cXLw= 2022-09-13T07:36:33.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FWTGxYziCcGRjBisisKPji1cXLw=', 'Content-Length': '0'} 2022-09-13T07:36:33.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002d952909811e6b4c-0063203301-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.734 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:33.735 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:33.735 DEBUG:boto:Method: HEAD 2022-09-13T07:36:33.736 DEBUG:boto:Path: /yxvvuh-18/dummy 2022-09-13T07:36:33.736 DEBUG:boto:Data: 2022-09-13T07:36:33.737 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.737 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.737 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.738 DEBUG:boto:Params: {} 2022-09-13T07:36:33.739 DEBUG:boto:Token: None 2022-09-13T07:36:33.739 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/dummy 2022-09-13T07:36:33.740 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FWTGxYziCcGRjBisisKPji1cXLw= 2022-09-13T07:36:33.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FWTGxYziCcGRjBisisKPji1cXLw=', 'Content-Length': '0'} 2022-09-13T07:36:33.742 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ebc9517d2ce8491f-0063203301-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:33.744 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:33.744 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:33.745 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.745 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.746 DEBUG:boto:Method: GET 2022-09-13T07:36:33.746 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.747 DEBUG:boto:Data: 2022-09-13T07:36:33.747 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.748 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.748 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.749 DEBUG:boto:Params: {} 2022-09-13T07:36:33.749 DEBUG:boto:Token: None 2022-09-13T07:36:33.750 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.751 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bUZxbECg0VzVpzkH4q7tNr5GaEQ= 2022-09-13T07:36:33.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bUZxbECg0VzVpzkH4q7tNr5GaEQ=', 'Content-Length': '0'} 2022-09-13T07:36:33.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef7f60d7e9644e3c-0063203301-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.754 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:33.754 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:33.755 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.755 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.756 DEBUG:boto:Method: GET 2022-09-13T07:36:33.757 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.757 DEBUG:boto:Data: 2022-09-13T07:36:33.758 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.758 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.759 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.759 DEBUG:boto:Params: {} 2022-09-13T07:36:33.760 DEBUG:boto:Token: None 2022-09-13T07:36:33.761 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:33.761 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bUZxbECg0VzVpzkH4q7tNr5GaEQ= 2022-09-13T07:36:33.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bUZxbECg0VzVpzkH4q7tNr5GaEQ=', 'Content-Length': '0'} 2022-09-13T07:36:33.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eed113f905abf813-0063203301-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.765 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:33.766 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:33.766 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:33.767 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:33.767 DEBUG:boto:Method: GET 2022-09-13T07:36:33.768 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:36:33.769 DEBUG:boto:Data: 2022-09-13T07:36:33.770 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.771 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.772 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.773 DEBUG:boto:Params: {} 2022-09-13T07:36:33.774 DEBUG:boto:Token: None 2022-09-13T07:36:33.775 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/ 2022-09-13T07:36:33.776 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ= 2022-09-13T07:36:33.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ=', 'Content-Length': '0'} 2022-09-13T07:36:33.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031cebf87b1c66a0d-0063203301-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.780 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:33.781 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:33.781 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:33.782 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:33.783 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:33.784 DEBUG:boto:Method: GET 2022-09-13T07:36:33.785 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:36:33.785 DEBUG:boto:Data: 2022-09-13T07:36:33.786 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.787 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.787 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.788 DEBUG:boto:Params: {} 2022-09-13T07:36:33.788 DEBUG:boto:Token: None 2022-09-13T07:36:33.789 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/ 2022-09-13T07:36:33.790 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ= 2022-09-13T07:36:33.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ=', 'Content-Length': '0'} 2022-09-13T07:36:33.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd95a838e25ba1b9-0063203301-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.794 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:33.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:33.795 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.796 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.797 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.797 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.798 DEBUG:boto:Method: GET 2022-09-13T07:36:33.798 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.799 DEBUG:boto:Data: 2022-09-13T07:36:33.799 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.800 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.800 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.801 DEBUG:boto:Params: {} 2022-09-13T07:36:33.802 DEBUG:boto:Token: None 2022-09-13T07:36:33.802 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.803 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Tq5A+JgLx3xKa3OtKxMWNz0wv+Q= 2022-09-13T07:36:33.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Tq5A+JgLx3xKa3OtKxMWNz0wv+Q=', 'Content-Length': '0'} 2022-09-13T07:36:33.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004aa71fc399918c2e-0063203301-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.806 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.807 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.807 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.808 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.809 DEBUG:boto:Method: GET 2022-09-13T07:36:33.809 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.810 DEBUG:boto:Data: 2022-09-13T07:36:33.810 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.811 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.812 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.812 DEBUG:boto:Params: {} 2022-09-13T07:36:33.813 DEBUG:boto:Token: None 2022-09-13T07:36:33.813 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.814 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Tq5A+JgLx3xKa3OtKxMWNz0wv+Q= 2022-09-13T07:36:33.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Tq5A+JgLx3xKa3OtKxMWNz0wv+Q=', 'Content-Length': '0'} 2022-09-13T07:36:33.817 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082eddeb391ba8c3a-0063203301-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.818 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.818 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.819 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.819 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.820 DEBUG:boto:Method: HEAD 2022-09-13T07:36:33.820 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.821 DEBUG:boto:Data: 2022-09-13T07:36:33.821 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.822 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.822 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.823 DEBUG:boto:Params: {} 2022-09-13T07:36:33.824 DEBUG:boto:Token: None 2022-09-13T07:36:33.824 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.825 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8P8DeboeFZc6V6I+ZUGu8irCJ4s= 2022-09-13T07:36:33.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8P8DeboeFZc6V6I+ZUGu8irCJ4s=', 'Content-Length': '0'} 2022-09-13T07:36:33.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007335822c2f15e322-0063203301-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.828 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.828 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.829 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.829 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.830 DEBUG:boto:Method: HEAD 2022-09-13T07:36:33.830 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.831 DEBUG:boto:Data: 2022-09-13T07:36:33.831 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.832 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.832 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.833 DEBUG:boto:Params: {} 2022-09-13T07:36:33.833 DEBUG:boto:Token: None 2022-09-13T07:36:33.834 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.834 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8P8DeboeFZc6V6I+ZUGu8irCJ4s= 2022-09-13T07:36:33.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8P8DeboeFZc6V6I+ZUGu8irCJ4s=', 'Content-Length': '0'} 2022-09-13T07:36:33.837 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e545c9f77c89c55a-0063203301-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:33.838 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.839 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.839 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.840 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.840 DEBUG:boto:Method: GET 2022-09-13T07:36:33.841 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.841 DEBUG:boto:Data: 2022-09-13T07:36:33.842 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.842 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.843 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.844 DEBUG:boto:Params: {} 2022-09-13T07:36:33.844 DEBUG:boto:Token: None 2022-09-13T07:36:33.845 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.846 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Tq5A+JgLx3xKa3OtKxMWNz0wv+Q= 2022-09-13T07:36:33.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Tq5A+JgLx3xKa3OtKxMWNz0wv+Q=', 'Content-Length': '0'} 2022-09-13T07:36:33.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce8f7dda22e45361-0063203301-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.852 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.853 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.854 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.854 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.855 DEBUG:boto:Method: GET 2022-09-13T07:36:33.855 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.856 DEBUG:boto:Data: 2022-09-13T07:36:33.856 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.857 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.858 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.858 DEBUG:boto:Params: {} 2022-09-13T07:36:33.859 DEBUG:boto:Token: None 2022-09-13T07:36:33.860 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.860 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Tq5A+JgLx3xKa3OtKxMWNz0wv+Q= 2022-09-13T07:36:33.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Tq5A+JgLx3xKa3OtKxMWNz0wv+Q=', 'Content-Length': '0'} 2022-09-13T07:36:33.863 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e506ba52a54dd067-0063203301-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.864 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:33.864 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:33.865 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:33.865 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:33.866 DEBUG:boto:Method: GET 2022-09-13T07:36:33.866 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:33.867 DEBUG:boto:Data: 2022-09-13T07:36:33.867 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.868 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.868 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.869 DEBUG:boto:Params: {} 2022-09-13T07:36:33.869 DEBUG:boto:Token: None 2022-09-13T07:36:33.870 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/ 2022-09-13T07:36:33.871 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ= 2022-09-13T07:36:33.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ=', 'Content-Length': '0'} 2022-09-13T07:36:33.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b65a69ecfa6993f-0063203301-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.878 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:33.879 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:33.880 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:33.881 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:33.895 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:33.896 DEBUG:boto:Method: GET 2022-09-13T07:36:33.897 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:33.898 DEBUG:boto:Data: 2022-09-13T07:36:33.899 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.899 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.900 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.900 DEBUG:boto:Params: {} 2022-09-13T07:36:33.901 DEBUG:boto:Token: None 2022-09-13T07:36:33.902 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/ 2022-09-13T07:36:33.902 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ= 2022-09-13T07:36:33.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ=', 'Content-Length': '0'} 2022-09-13T07:36:33.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c9e0b80c593cc4c-0063203301-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.906 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:33.907 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.908 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.908 DEBUG:boto:Method: HEAD 2022-09-13T07:36:33.909 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.909 DEBUG:boto:Data: 2022-09-13T07:36:33.910 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.910 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.911 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.911 DEBUG:boto:Params: {} 2022-09-13T07:36:33.912 DEBUG:boto:Token: None 2022-09-13T07:36:33.913 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.913 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:B5CuPkb49LeEVv8HGNfBwjkfiOw= 2022-09-13T07:36:33.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:B5CuPkb49LeEVv8HGNfBwjkfiOw=', 'Content-Length': '0'} 2022-09-13T07:36:33.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee5f8e9f4d9e9f40-0063203301-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.917 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.917 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.918 DEBUG:boto:Method: HEAD 2022-09-13T07:36:33.919 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.919 DEBUG:boto:Data: 2022-09-13T07:36:33.920 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.921 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.921 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.922 DEBUG:boto:Params: {} 2022-09-13T07:36:33.922 DEBUG:boto:Token: None 2022-09-13T07:36:33.923 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.923 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:B5CuPkb49LeEVv8HGNfBwjkfiOw= 2022-09-13T07:36:33.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:B5CuPkb49LeEVv8HGNfBwjkfiOw=', 'Content-Length': '0'} 2022-09-13T07:36:33.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046f375ab1797d3cd-0063203301-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:33.929 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.929 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.930 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.931 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.931 DEBUG:boto:Method: GET 2022-09-13T07:36:33.932 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.933 DEBUG:boto:Data: 2022-09-13T07:36:33.934 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.934 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.935 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.935 DEBUG:boto:Params: {} 2022-09-13T07:36:33.936 DEBUG:boto:Token: None 2022-09-13T07:36:33.937 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.937 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Tq5A+JgLx3xKa3OtKxMWNz0wv+Q= 2022-09-13T07:36:33.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Tq5A+JgLx3xKa3OtKxMWNz0wv+Q=', 'Content-Length': '0'} 2022-09-13T07:36:33.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f983112431c71f8-0063203301-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.941 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.941 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:33.942 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.942 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.943 DEBUG:boto:Method: GET 2022-09-13T07:36:33.943 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.944 DEBUG:boto:Data: 2022-09-13T07:36:33.945 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.945 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.946 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.947 DEBUG:boto:Params: {} 2022-09-13T07:36:33.947 DEBUG:boto:Token: None 2022-09-13T07:36:33.948 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:33.949 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Tq5A+JgLx3xKa3OtKxMWNz0wv+Q= 2022-09-13T07:36:33.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Tq5A+JgLx3xKa3OtKxMWNz0wv+Q=', 'Content-Length': '0'} 2022-09-13T07:36:33.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041b09a87dc214ba4-0063203301-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.953 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:33.953 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:33.954 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:33.954 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:33.955 DEBUG:boto:Method: GET 2022-09-13T07:36:33.955 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:33.956 DEBUG:boto:Data: 2022-09-13T07:36:33.956 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.957 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.957 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.958 DEBUG:boto:Params: {} 2022-09-13T07:36:33.958 DEBUG:boto:Token: None 2022-09-13T07:36:33.959 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/ 2022-09-13T07:36:33.960 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ= 2022-09-13T07:36:33.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ=', 'Content-Length': '0'} 2022-09-13T07:36:33.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a532584683bc4e84-0063203301-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.964 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:33.965 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:33.965 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:33.966 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:33.966 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:33.967 DEBUG:boto:Method: GET 2022-09-13T07:36:33.967 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:33.968 DEBUG:boto:Data: 2022-09-13T07:36:33.968 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.969 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:33.969 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.970 DEBUG:boto:Params: {} 2022-09-13T07:36:33.971 DEBUG:boto:Token: None 2022-09-13T07:36:33.971 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/ 2022-09-13T07:36:33.972 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ= 2022-09-13T07:36:33.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fPZfHVU46Sz1di31RhNkyfssfwQ=', 'Content-Length': '0'} 2022-09-13T07:36:33.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a66096dd8526086-0063203301-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:33.982 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:33.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:33.984 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:33.985 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:33.986 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:33.987 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:33.988 DEBUG:boto:Method: GET 2022-09-13T07:36:33.989 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:33.990 DEBUG:boto:Data: 2022-09-13T07:36:33.991 DEBUG:boto:Headers: {} 2022-09-13T07:36:33.992 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:33.992 DEBUG:boto:Port: 8000 2022-09-13T07:36:33.995 DEBUG:boto:Params: {} 2022-09-13T07:36:33.996 DEBUG:boto:Token: None 2022-09-13T07:36:33.997 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:33 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:33.998 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mZM5NrW6vtFh21W2O+mml0qywaw= 2022-09-13T07:36:34.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mZM5NrW6vtFh21W2O+mml0qywaw=', 'Content-Length': '0'} 2022-09-13T07:36:34.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f7d54641c6ae909-0063203301-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.046 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.047 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.048 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.049 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.050 DEBUG:boto:Method: GET 2022-09-13T07:36:34.051 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.052 DEBUG:boto:Data: 2022-09-13T07:36:34.053 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.054 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.055 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.056 DEBUG:boto:Params: {} 2022-09-13T07:36:34.057 DEBUG:boto:Token: None 2022-09-13T07:36:34.058 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.059 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wAvv6zpTTFAhjsZmV7Tmjm8zg+k= 2022-09-13T07:36:34.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wAvv6zpTTFAhjsZmV7Tmjm8zg+k=', 'Content-Length': '0'} 2022-09-13T07:36:34.063 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e157cce5913e641e-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.064 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.066 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.066 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.067 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.067 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.068 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.068 DEBUG:boto:Data: 2022-09-13T07:36:34.069 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.070 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:34.071 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.072 DEBUG:boto:Params: {} 2022-09-13T07:36:34.073 DEBUG:boto:Token: None 2022-09-13T07:36:34.074 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.075 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7qYxnBTq1+6dINsvjz7YFnp0w+8= 2022-09-13T07:36:34.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7qYxnBTq1+6dINsvjz7YFnp0w+8=', 'Content-Length': '0'} 2022-09-13T07:36:34.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007aa29716a65237af-0063203302-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.082 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.083 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.084 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.084 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.085 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.086 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.087 DEBUG:boto:Data: 2022-09-13T07:36:34.088 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.089 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.089 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.090 DEBUG:boto:Params: {} 2022-09-13T07:36:34.091 DEBUG:boto:Token: None 2022-09-13T07:36:34.092 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.093 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7qYxnBTq1+6dINsvjz7YFnp0w+8= 2022-09-13T07:36:34.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7qYxnBTq1+6dINsvjz7YFnp0w+8=', 'Content-Length': '0'} 2022-09-13T07:36:34.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072e072fc0fb9eeeb-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:34.100 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.101 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.101 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.102 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.103 DEBUG:boto:Method: GET 2022-09-13T07:36:34.104 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.105 DEBUG:boto:Data: 2022-09-13T07:36:34.106 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.107 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:34.108 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.109 DEBUG:boto:Params: {} 2022-09-13T07:36:34.110 DEBUG:boto:Token: None 2022-09-13T07:36:34.111 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.112 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wAvv6zpTTFAhjsZmV7Tmjm8zg+k= 2022-09-13T07:36:34.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wAvv6zpTTFAhjsZmV7Tmjm8zg+k=', 'Content-Length': '0'} 2022-09-13T07:36:34.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6fd330c7652f000-0063203302-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.116 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.117 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.117 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.118 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.119 DEBUG:boto:Method: GET 2022-09-13T07:36:34.119 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.120 DEBUG:boto:Data: 2022-09-13T07:36:34.120 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.121 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.121 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.122 DEBUG:boto:Params: {} 2022-09-13T07:36:34.122 DEBUG:boto:Token: None 2022-09-13T07:36:34.123 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.124 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wAvv6zpTTFAhjsZmV7Tmjm8zg+k= 2022-09-13T07:36:34.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wAvv6zpTTFAhjsZmV7Tmjm8zg+k=', 'Content-Length': '0'} 2022-09-13T07:36:34.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a5cf59c708fda7e-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.128 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.128 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.129 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:34.129 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:34.130 DEBUG:boto:Method: GET 2022-09-13T07:36:34.130 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:34.131 DEBUG:boto:Data: 2022-09-13T07:36:34.132 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.132 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:34.133 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.134 DEBUG:boto:Params: {} 2022-09-13T07:36:34.135 DEBUG:boto:Token: None 2022-09-13T07:36:34.135 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/ 2022-09-13T07:36:34.136 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0= 2022-09-13T07:36:34.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0=', 'Content-Length': '0'} 2022-09-13T07:36:34.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e2057b142398e5c-0063203302-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.140 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.141 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.141 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.142 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:34.142 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:34.143 DEBUG:boto:Method: GET 2022-09-13T07:36:34.143 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:34.144 DEBUG:boto:Data: 2022-09-13T07:36:34.144 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.145 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.145 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.146 DEBUG:boto:Params: {} 2022-09-13T07:36:34.146 DEBUG:boto:Token: None 2022-09-13T07:36:34.147 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/ 2022-09-13T07:36:34.148 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0= 2022-09-13T07:36:34.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0=', 'Content-Length': '0'} 2022-09-13T07:36:34.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9bb52e04b924638-0063203302-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.198 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.200 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.201 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.202 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.203 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.203 DEBUG:boto:Data: 2022-09-13T07:36:34.204 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.205 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:34.206 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.207 DEBUG:boto:Params: {} 2022-09-13T07:36:34.208 DEBUG:boto:Token: None 2022-09-13T07:36:34.209 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.210 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RmIPmkzJCLJN34dsh4Ww1E7aVeE= 2022-09-13T07:36:34.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RmIPmkzJCLJN34dsh4Ww1E7aVeE=', 'Content-Length': '0'} 2022-09-13T07:36:34.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc674ee0533d836e-0063203302-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.215 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.216 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.216 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.217 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.218 DEBUG:boto:Data: 2022-09-13T07:36:34.219 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.220 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.220 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.221 DEBUG:boto:Params: {} 2022-09-13T07:36:34.222 DEBUG:boto:Token: None 2022-09-13T07:36:34.223 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.224 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RmIPmkzJCLJN34dsh4Ww1E7aVeE= 2022-09-13T07:36:34.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RmIPmkzJCLJN34dsh4Ww1E7aVeE=', 'Content-Length': '0'} 2022-09-13T07:36:34.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007022df5a42619aed-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:34.230 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.231 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.231 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.232 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.233 DEBUG:boto:Method: GET 2022-09-13T07:36:34.234 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.235 DEBUG:boto:Data: 2022-09-13T07:36:34.236 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.236 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:34.237 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.237 DEBUG:boto:Params: {} 2022-09-13T07:36:34.238 DEBUG:boto:Token: None 2022-09-13T07:36:34.238 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.239 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wAvv6zpTTFAhjsZmV7Tmjm8zg+k= 2022-09-13T07:36:34.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wAvv6zpTTFAhjsZmV7Tmjm8zg+k=', 'Content-Length': '0'} 2022-09-13T07:36:34.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a915fdf5216db12-0063203302-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.242 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.243 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:34.244 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.244 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.245 DEBUG:boto:Method: GET 2022-09-13T07:36:34.245 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.246 DEBUG:boto:Data: 2022-09-13T07:36:34.246 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.247 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.247 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.248 DEBUG:boto:Params: {} 2022-09-13T07:36:34.248 DEBUG:boto:Token: None 2022-09-13T07:36:34.249 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:34.250 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wAvv6zpTTFAhjsZmV7Tmjm8zg+k= 2022-09-13T07:36:34.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wAvv6zpTTFAhjsZmV7Tmjm8zg+k=', 'Content-Length': '0'} 2022-09-13T07:36:34.253 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000344a5782a845cc27-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.254 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.254 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.255 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:34.255 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:34.256 DEBUG:boto:Method: GET 2022-09-13T07:36:34.256 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:34.257 DEBUG:boto:Data: 2022-09-13T07:36:34.258 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.258 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:34.259 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.259 DEBUG:boto:Params: {} 2022-09-13T07:36:34.260 DEBUG:boto:Token: None 2022-09-13T07:36:34.260 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/ 2022-09-13T07:36:34.261 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0= 2022-09-13T07:36:34.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0=', 'Content-Length': '0'} 2022-09-13T07:36:34.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000998b2e4b9aeeaf02-0063203302-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.265 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.265 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.266 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.267 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:34.267 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:34.268 DEBUG:boto:Method: GET 2022-09-13T07:36:34.268 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:34.269 DEBUG:boto:Data: 2022-09-13T07:36:34.269 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.270 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.270 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.271 DEBUG:boto:Params: {} 2022-09-13T07:36:34.272 DEBUG:boto:Token: None 2022-09-13T07:36:34.272 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/ 2022-09-13T07:36:34.273 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0= 2022-09-13T07:36:34.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0=', 'Content-Length': '0'} 2022-09-13T07:36:34.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098e44792029ece92-0063203302-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.277 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:34.279 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.279 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.280 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.280 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.281 DEBUG:boto:Method: GET 2022-09-13T07:36:34.281 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.282 DEBUG:boto:Data: 2022-09-13T07:36:34.282 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.283 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:34.283 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.284 DEBUG:boto:Params: {} 2022-09-13T07:36:34.284 DEBUG:boto:Token: None 2022-09-13T07:36:34.285 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.285 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PTJrInRbOKu33yGvFMnIMVMf/jQ= 2022-09-13T07:36:34.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PTJrInRbOKu33yGvFMnIMVMf/jQ=', 'Content-Length': '0'} 2022-09-13T07:36:34.288 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f152877cfd58968-0063203302-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.289 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.290 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.291 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.292 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.293 DEBUG:boto:Method: GET 2022-09-13T07:36:34.294 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.294 DEBUG:boto:Data: 2022-09-13T07:36:34.295 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.296 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.297 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.297 DEBUG:boto:Params: {} 2022-09-13T07:36:34.298 DEBUG:boto:Token: None 2022-09-13T07:36:34.299 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.300 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PTJrInRbOKu33yGvFMnIMVMf/jQ= 2022-09-13T07:36:34.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PTJrInRbOKu33yGvFMnIMVMf/jQ=', 'Content-Length': '0'} 2022-09-13T07:36:34.303 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014344f189b98bcf4-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.304 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.304 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.305 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.305 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.306 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.306 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.307 DEBUG:boto:Data: 2022-09-13T07:36:34.307 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.308 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:34.308 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.309 DEBUG:boto:Params: {} 2022-09-13T07:36:34.309 DEBUG:boto:Token: None 2022-09-13T07:36:34.310 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.311 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ksfmKEg9VOpuEa8DYC0odWInfvA= 2022-09-13T07:36:34.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ksfmKEg9VOpuEa8DYC0odWInfvA=', 'Content-Length': '0'} 2022-09-13T07:36:34.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6e3768c436df97e-0063203302-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.314 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.314 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.315 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.315 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.316 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.316 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.317 DEBUG:boto:Data: 2022-09-13T07:36:34.318 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.318 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.319 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.320 DEBUG:boto:Params: {} 2022-09-13T07:36:34.321 DEBUG:boto:Token: None 2022-09-13T07:36:34.322 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.323 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ksfmKEg9VOpuEa8DYC0odWInfvA= 2022-09-13T07:36:34.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ksfmKEg9VOpuEa8DYC0odWInfvA=', 'Content-Length': '0'} 2022-09-13T07:36:34.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052698f4b56d86974-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:34.334 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.335 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.336 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.337 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.338 DEBUG:boto:Method: GET 2022-09-13T07:36:34.339 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.340 DEBUG:boto:Data: 2022-09-13T07:36:34.341 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.342 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:34.343 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.344 DEBUG:boto:Params: {} 2022-09-13T07:36:34.345 DEBUG:boto:Token: None 2022-09-13T07:36:34.346 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.347 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PTJrInRbOKu33yGvFMnIMVMf/jQ= 2022-09-13T07:36:34.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PTJrInRbOKu33yGvFMnIMVMf/jQ=', 'Content-Length': '0'} 2022-09-13T07:36:34.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efaba5289d253f2e-0063203302-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.352 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.353 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.354 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.355 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.356 DEBUG:boto:Method: GET 2022-09-13T07:36:34.356 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.357 DEBUG:boto:Data: 2022-09-13T07:36:34.357 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.358 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.358 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.359 DEBUG:boto:Params: {} 2022-09-13T07:36:34.359 DEBUG:boto:Token: None 2022-09-13T07:36:34.360 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.361 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PTJrInRbOKu33yGvFMnIMVMf/jQ= 2022-09-13T07:36:34.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PTJrInRbOKu33yGvFMnIMVMf/jQ=', 'Content-Length': '0'} 2022-09-13T07:36:34.363 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061c571f7cc664492-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.414 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.415 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.416 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:34.417 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:34.418 DEBUG:boto:Method: GET 2022-09-13T07:36:34.419 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:34.420 DEBUG:boto:Data: 2022-09-13T07:36:34.421 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.422 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:34.423 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.424 DEBUG:boto:Params: {} 2022-09-13T07:36:34.425 DEBUG:boto:Token: None 2022-09-13T07:36:34.426 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/ 2022-09-13T07:36:34.427 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0= 2022-09-13T07:36:34.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0=', 'Content-Length': '0'} 2022-09-13T07:36:34.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089346922faf5a565-0063203302-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.433 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.434 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.435 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.436 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:34.437 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:34.438 DEBUG:boto:Method: GET 2022-09-13T07:36:34.439 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:34.439 DEBUG:boto:Data: 2022-09-13T07:36:34.440 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.441 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.442 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.443 DEBUG:boto:Params: {} 2022-09-13T07:36:34.443 DEBUG:boto:Token: None 2022-09-13T07:36:34.444 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/ 2022-09-13T07:36:34.444 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0= 2022-09-13T07:36:34.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0=', 'Content-Length': '0'} 2022-09-13T07:36:34.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2563eba8f0a451d-0063203302-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.451 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.452 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.453 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.453 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.454 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.454 DEBUG:boto:Data: 2022-09-13T07:36:34.455 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.455 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:34.456 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.456 DEBUG:boto:Params: {} 2022-09-13T07:36:34.457 DEBUG:boto:Token: None 2022-09-13T07:36:34.457 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.458 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TBfDNK/LcIC+0+c0QxX+PIOxGBo= 2022-09-13T07:36:34.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TBfDNK/LcIC+0+c0QxX+PIOxGBo=', 'Content-Length': '0'} 2022-09-13T07:36:34.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007955a0fd6424067a-0063203302-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.462 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.463 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.464 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.464 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.465 DEBUG:boto:Data: 2022-09-13T07:36:34.466 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.467 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.467 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.468 DEBUG:boto:Params: {} 2022-09-13T07:36:34.468 DEBUG:boto:Token: None 2022-09-13T07:36:34.469 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.470 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TBfDNK/LcIC+0+c0QxX+PIOxGBo= 2022-09-13T07:36:34.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TBfDNK/LcIC+0+c0QxX+PIOxGBo=', 'Content-Length': '0'} 2022-09-13T07:36:34.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc6163f44a4fd098-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:34.476 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.477 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.478 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.479 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.479 DEBUG:boto:Method: GET 2022-09-13T07:36:34.480 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.481 DEBUG:boto:Data: 2022-09-13T07:36:34.481 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.482 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:34.483 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.484 DEBUG:boto:Params: {} 2022-09-13T07:36:34.485 DEBUG:boto:Token: None 2022-09-13T07:36:34.486 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.487 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PTJrInRbOKu33yGvFMnIMVMf/jQ= 2022-09-13T07:36:34.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PTJrInRbOKu33yGvFMnIMVMf/jQ=', 'Content-Length': '0'} 2022-09-13T07:36:34.490 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4bf702255bb2a20-0063203302-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.492 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.493 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:34.493 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.494 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.495 DEBUG:boto:Method: GET 2022-09-13T07:36:34.496 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.497 DEBUG:boto:Data: 2022-09-13T07:36:34.497 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.498 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.498 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.500 DEBUG:boto:Params: {} 2022-09-13T07:36:34.501 DEBUG:boto:Token: None 2022-09-13T07:36:34.502 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:34.503 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PTJrInRbOKu33yGvFMnIMVMf/jQ= 2022-09-13T07:36:34.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PTJrInRbOKu33yGvFMnIMVMf/jQ=', 'Content-Length': '0'} 2022-09-13T07:36:34.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000429e481e40a97d99-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.506 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.507 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.508 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:34.509 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:34.510 DEBUG:boto:Method: GET 2022-09-13T07:36:34.510 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:34.511 DEBUG:boto:Data: 2022-09-13T07:36:34.512 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.513 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:34.514 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.515 DEBUG:boto:Params: {} 2022-09-13T07:36:34.516 DEBUG:boto:Token: None 2022-09-13T07:36:34.517 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/ 2022-09-13T07:36:34.518 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0= 2022-09-13T07:36:34.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0=', 'Content-Length': '0'} 2022-09-13T07:36:34.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7f433cf231df8f4-0063203302-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.566 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.567 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.568 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.568 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:34.569 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:34.569 DEBUG:boto:Method: GET 2022-09-13T07:36:34.570 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:34.570 DEBUG:boto:Data: 2022-09-13T07:36:34.571 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.571 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.572 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.573 DEBUG:boto:Params: {} 2022-09-13T07:36:34.573 DEBUG:boto:Token: None 2022-09-13T07:36:34.574 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/ 2022-09-13T07:36:34.574 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0= 2022-09-13T07:36:34.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0=', 'Content-Length': '0'} 2022-09-13T07:36:34.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052a4493a74f775c6-0063203302-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.579 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.580 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-18 zones={test-zone2, test-zone1} 2022-09-13T07:36:34.581 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-18 zones={test-zone3, test-zone1} 2022-09-13T07:36:34.581 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.582 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.582 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.583 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:36:34.584 DEBUG:boto:Data: 2022-09-13T07:36:34.584 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.585 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:34.585 DEBUG:boto:Port: 8001 2022-09-13T07:36:34.586 DEBUG:boto:Params: {} 2022-09-13T07:36:34.587 DEBUG:boto:Token: None 2022-09-13T07:36:34.587 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/ 2022-09-13T07:36:34.588 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lTB+SYlxIf4oKC13el8f/lQdrIA= 2022-09-13T07:36:34.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lTB+SYlxIf4oKC13el8f/lQdrIA=', 'Content-Length': '0'} 2022-09-13T07:36:34.592 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009dc907197ac08742-0063203302-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.592 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.593 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.593 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.594 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:36:34.594 DEBUG:boto:Data: 2022-09-13T07:36:34.595 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.596 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.596 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.597 DEBUG:boto:Params: {} 2022-09-13T07:36:34.597 DEBUG:boto:Token: None 2022-09-13T07:36:34.598 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/ 2022-09-13T07:36:34.599 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lTB+SYlxIf4oKC13el8f/lQdrIA= 2022-09-13T07:36:34.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lTB+SYlxIf4oKC13el8f/lQdrIA=', 'Content-Length': '0'} 2022-09-13T07:36:34.602 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c489f67d692b7e9-0063203302-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.603 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:36:34.604 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.604 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.605 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:34.605 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:34.606 DEBUG:boto:Method: GET 2022-09-13T07:36:34.607 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:34.608 DEBUG:boto:Data: 2022-09-13T07:36:34.608 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.609 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:34.609 DEBUG:boto:Port: 8001 2022-09-13T07:36:34.610 DEBUG:boto:Params: {} 2022-09-13T07:36:34.611 DEBUG:boto:Token: None 2022-09-13T07:36:34.612 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/?versions 2022-09-13T07:36:34.612 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:R3fBZDE/dGBVMQzEq+cwlehqLIY= 2022-09-13T07:36:34.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:R3fBZDE/dGBVMQzEq+cwlehqLIY=', 'Content-Length': '0'} 2022-09-13T07:36:34.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097d2753814d3f175-0063203302-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.617 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.617 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:36:34.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:36:34.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:36:34.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:36:34.619 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:36:34.620 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.620 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.621 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:34.622 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:34.622 DEBUG:boto:Method: GET 2022-09-13T07:36:34.623 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:34.623 DEBUG:boto:Data: 2022-09-13T07:36:34.624 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.624 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.625 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.625 DEBUG:boto:Params: {} 2022-09-13T07:36:34.626 DEBUG:boto:Token: None 2022-09-13T07:36:34.627 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/?versions 2022-09-13T07:36:34.627 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:R3fBZDE/dGBVMQzEq+cwlehqLIY= 2022-09-13T07:36:34.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:R3fBZDE/dGBVMQzEq+cwlehqLIY=', 'Content-Length': '0'} 2022-09-13T07:36:34.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f61c30e3112e1932-0063203302-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.632 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.633 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:36:34.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:36:34.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:36:34.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:36:34.636 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.637 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.638 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:34.639 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:34.640 DEBUG:boto:Method: GET 2022-09-13T07:36:34.641 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:34.642 DEBUG:boto:Data: 2022-09-13T07:36:34.643 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.644 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:34.645 DEBUG:boto:Port: 8001 2022-09-13T07:36:34.646 DEBUG:boto:Params: {} 2022-09-13T07:36:34.647 DEBUG:boto:Token: None 2022-09-13T07:36:34.648 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/?versions 2022-09-13T07:36:34.649 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:R3fBZDE/dGBVMQzEq+cwlehqLIY= 2022-09-13T07:36:34.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:R3fBZDE/dGBVMQzEq+cwlehqLIY=', 'Content-Length': '0'} 2022-09-13T07:36:34.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022991a48d8b5593d-0063203302-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.654 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.655 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.656 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.656 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:34.657 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:34.657 DEBUG:boto:Method: GET 2022-09-13T07:36:34.658 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:34.658 DEBUG:boto:Data: 2022-09-13T07:36:34.659 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.659 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.660 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.661 DEBUG:boto:Params: {} 2022-09-13T07:36:34.661 DEBUG:boto:Token: None 2022-09-13T07:36:34.662 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/?versions 2022-09-13T07:36:34.662 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:R3fBZDE/dGBVMQzEq+cwlehqLIY= 2022-09-13T07:36:34.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:R3fBZDE/dGBVMQzEq+cwlehqLIY=', 'Content-Length': '0'} 2022-09-13T07:36:34.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000def41a665c158a4f-0063203302-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.667 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:34.668 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:34.669 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:34.670 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.670 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.671 DEBUG:boto:Method: GET 2022-09-13T07:36:34.671 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.672 DEBUG:boto:Data: 2022-09-13T07:36:34.672 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.673 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:34.674 DEBUG:boto:Port: 8001 2022-09-13T07:36:34.674 DEBUG:boto:Params: {} 2022-09-13T07:36:34.675 DEBUG:boto:Token: None 2022-09-13T07:36:34.676 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.677 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BNEC57p2cd/D8TU3pNVtwdmwXME= 2022-09-13T07:36:34.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BNEC57p2cd/D8TU3pNVtwdmwXME=', 'Content-Length': '0'} 2022-09-13T07:36:34.680 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000760e0eb8d19aadd0-0063203302-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.681 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:34.681 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:34.682 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.682 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.683 DEBUG:boto:Method: GET 2022-09-13T07:36:34.683 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.684 DEBUG:boto:Data: 2022-09-13T07:36:34.685 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.685 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.686 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.686 DEBUG:boto:Params: {} 2022-09-13T07:36:34.687 DEBUG:boto:Token: None 2022-09-13T07:36:34.687 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.688 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BNEC57p2cd/D8TU3pNVtwdmwXME= 2022-09-13T07:36:34.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BNEC57p2cd/D8TU3pNVtwdmwXME=', 'Content-Length': '0'} 2022-09-13T07:36:34.691 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fc3912e71d2fc245-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.692 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:34.692 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:34.693 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.693 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.694 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.695 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.695 DEBUG:boto:Data: 2022-09-13T07:36:34.695 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.696 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:34.696 DEBUG:boto:Port: 8001 2022-09-13T07:36:34.697 DEBUG:boto:Params: {} 2022-09-13T07:36:34.698 DEBUG:boto:Token: None 2022-09-13T07:36:34.698 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.699 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dBulHpQNP4s914/XQ+BLHCfUyTo= 2022-09-13T07:36:34.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dBulHpQNP4s914/XQ+BLHCfUyTo=', 'Content-Length': '0'} 2022-09-13T07:36:34.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a3e93e303c413f31-0063203302-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.703 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:34.704 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:34.705 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.706 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.706 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.707 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.707 DEBUG:boto:Data: 2022-09-13T07:36:34.708 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.708 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.709 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.709 DEBUG:boto:Params: {} 2022-09-13T07:36:34.710 DEBUG:boto:Token: None 2022-09-13T07:36:34.710 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.711 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dBulHpQNP4s914/XQ+BLHCfUyTo= 2022-09-13T07:36:34.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dBulHpQNP4s914/XQ+BLHCfUyTo=', 'Content-Length': '0'} 2022-09-13T07:36:34.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d78eee62736cb13-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:34.715 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:34.716 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:34.716 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.717 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.717 DEBUG:boto:Method: GET 2022-09-13T07:36:34.718 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.718 DEBUG:boto:Data: 2022-09-13T07:36:34.719 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.719 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:34.720 DEBUG:boto:Port: 8001 2022-09-13T07:36:34.720 DEBUG:boto:Params: {} 2022-09-13T07:36:34.721 DEBUG:boto:Token: None 2022-09-13T07:36:34.722 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.723 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BNEC57p2cd/D8TU3pNVtwdmwXME= 2022-09-13T07:36:34.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BNEC57p2cd/D8TU3pNVtwdmwXME=', 'Content-Length': '0'} 2022-09-13T07:36:34.725 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000073551205ded1c4a-0063203302-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.726 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:34.727 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:34.727 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.728 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.728 DEBUG:boto:Method: GET 2022-09-13T07:36:34.729 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.729 DEBUG:boto:Data: 2022-09-13T07:36:34.730 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.731 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.731 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.732 DEBUG:boto:Params: {} 2022-09-13T07:36:34.732 DEBUG:boto:Token: None 2022-09-13T07:36:34.733 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.734 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BNEC57p2cd/D8TU3pNVtwdmwXME= 2022-09-13T07:36:34.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BNEC57p2cd/D8TU3pNVtwdmwXME=', 'Content-Length': '0'} 2022-09-13T07:36:34.736 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab703e9b7e7e687d-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.737 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.738 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.738 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:34.739 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:34.739 DEBUG:boto:Method: GET 2022-09-13T07:36:34.740 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:36:34.740 DEBUG:boto:Data: 2022-09-13T07:36:34.741 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.741 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:34.742 DEBUG:boto:Port: 8001 2022-09-13T07:36:34.742 DEBUG:boto:Params: {} 2022-09-13T07:36:34.743 DEBUG:boto:Token: None 2022-09-13T07:36:34.744 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/ 2022-09-13T07:36:34.744 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0= 2022-09-13T07:36:34.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0=', 'Content-Length': '0'} 2022-09-13T07:36:34.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d28f583f379b6953-0063203302-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.749 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.750 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.751 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.752 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:34.753 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:34.754 DEBUG:boto:Method: GET 2022-09-13T07:36:34.755 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:36:34.755 DEBUG:boto:Data: 2022-09-13T07:36:34.756 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.757 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.758 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.759 DEBUG:boto:Params: {} 2022-09-13T07:36:34.760 DEBUG:boto:Token: None 2022-09-13T07:36:34.761 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/ 2022-09-13T07:36:34.762 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0= 2022-09-13T07:36:34.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0=', 'Content-Length': '0'} 2022-09-13T07:36:34.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084e2a557e421e5a3-0063203302-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.806 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.807 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:34.808 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:34.808 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.809 DEBUG:boto:Path: /yxvvuh-18/dummy 2022-09-13T07:36:34.809 DEBUG:boto:Data: 2022-09-13T07:36:34.810 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.811 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:34.811 DEBUG:boto:Port: 8001 2022-09-13T07:36:34.812 DEBUG:boto:Params: {} 2022-09-13T07:36:34.812 DEBUG:boto:Token: None 2022-09-13T07:36:34.813 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/dummy 2022-09-13T07:36:34.814 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qUy9d63dRORS7a46u2dJsGmINFU= 2022-09-13T07:36:34.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qUy9d63dRORS7a46u2dJsGmINFU=', 'Content-Length': '0'} 2022-09-13T07:36:34.817 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d8f2192098f333c-0063203302-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.818 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:34.818 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:34.819 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.819 DEBUG:boto:Path: /yxvvuh-18/dummy 2022-09-13T07:36:34.820 DEBUG:boto:Data: 2022-09-13T07:36:34.820 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.821 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.821 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.822 DEBUG:boto:Params: {} 2022-09-13T07:36:34.822 DEBUG:boto:Token: None 2022-09-13T07:36:34.823 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/dummy 2022-09-13T07:36:34.824 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qUy9d63dRORS7a46u2dJsGmINFU= 2022-09-13T07:36:34.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qUy9d63dRORS7a46u2dJsGmINFU=', 'Content-Length': '0'} 2022-09-13T07:36:34.827 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000043dd677f69f733ea-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:34.830 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:34.830 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:34.831 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.832 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.833 DEBUG:boto:Method: GET 2022-09-13T07:36:34.834 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.835 DEBUG:boto:Data: 2022-09-13T07:36:34.836 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.837 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:34.838 DEBUG:boto:Port: 8001 2022-09-13T07:36:34.839 DEBUG:boto:Params: {} 2022-09-13T07:36:34.840 DEBUG:boto:Token: None 2022-09-13T07:36:34.841 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.842 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BNEC57p2cd/D8TU3pNVtwdmwXME= 2022-09-13T07:36:34.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BNEC57p2cd/D8TU3pNVtwdmwXME=', 'Content-Length': '0'} 2022-09-13T07:36:34.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d5bebafa62d3a7a5-0063203302-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.847 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:34.848 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:34.849 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.850 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.851 DEBUG:boto:Method: GET 2022-09-13T07:36:34.852 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.852 DEBUG:boto:Data: 2022-09-13T07:36:34.853 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.854 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.855 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.856 DEBUG:boto:Params: {} 2022-09-13T07:36:34.857 DEBUG:boto:Token: None 2022-09-13T07:36:34.858 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:34.859 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BNEC57p2cd/D8TU3pNVtwdmwXME= 2022-09-13T07:36:34.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BNEC57p2cd/D8TU3pNVtwdmwXME=', 'Content-Length': '0'} 2022-09-13T07:36:34.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000642de910738a7f45-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.866 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.867 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.868 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:34.868 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:34.869 DEBUG:boto:Method: GET 2022-09-13T07:36:34.870 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:36:34.871 DEBUG:boto:Data: 2022-09-13T07:36:34.872 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.873 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:34.874 DEBUG:boto:Port: 8001 2022-09-13T07:36:34.875 DEBUG:boto:Params: {} 2022-09-13T07:36:34.876 DEBUG:boto:Token: None 2022-09-13T07:36:34.877 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/ 2022-09-13T07:36:34.878 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0= 2022-09-13T07:36:34.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0=', 'Content-Length': '0'} 2022-09-13T07:36:34.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004902c698066e816c-0063203302-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.884 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.885 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:34.886 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:34.886 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:34.887 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:34.887 DEBUG:boto:Method: GET 2022-09-13T07:36:34.888 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:36:34.888 DEBUG:boto:Data: 2022-09-13T07:36:34.889 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.889 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.890 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.890 DEBUG:boto:Params: {} 2022-09-13T07:36:34.891 DEBUG:boto:Token: None 2022-09-13T07:36:34.891 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/ 2022-09-13T07:36:34.892 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0= 2022-09-13T07:36:34.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2+qZi6v6T+jbiRMNNguikYEj6i0=', 'Content-Length': '0'} 2022-09-13T07:36:34.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caef7b8e3576c1b1-0063203302-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.938 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:34.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:34.940 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:34.940 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:34.941 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.941 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.942 DEBUG:boto:Method: GET 2022-09-13T07:36:34.942 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.943 DEBUG:boto:Data: 2022-09-13T07:36:34.943 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.944 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:34.944 DEBUG:boto:Port: 8001 2022-09-13T07:36:34.945 DEBUG:boto:Params: {} 2022-09-13T07:36:34.946 DEBUG:boto:Token: None 2022-09-13T07:36:34.946 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.947 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xqi/08I/TJmJVj4fDLPpGDoIPQM= 2022-09-13T07:36:34.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xqi/08I/TJmJVj4fDLPpGDoIPQM=', 'Content-Length': '0'} 2022-09-13T07:36:34.950 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb3297528012c925-0063203302-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.951 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:34.951 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:34.952 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.952 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.953 DEBUG:boto:Method: GET 2022-09-13T07:36:34.953 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.954 DEBUG:boto:Data: 2022-09-13T07:36:34.954 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.955 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.955 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.956 DEBUG:boto:Params: {} 2022-09-13T07:36:34.956 DEBUG:boto:Token: None 2022-09-13T07:36:34.957 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.958 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xqi/08I/TJmJVj4fDLPpGDoIPQM= 2022-09-13T07:36:34.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xqi/08I/TJmJVj4fDLPpGDoIPQM=', 'Content-Length': '0'} 2022-09-13T07:36:34.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd99d20a04f4a48f-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.961 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:34.962 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:34.962 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.963 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.964 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.965 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.966 DEBUG:boto:Data: 2022-09-13T07:36:34.967 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.967 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:34.968 DEBUG:boto:Port: 8001 2022-09-13T07:36:34.969 DEBUG:boto:Params: {} 2022-09-13T07:36:34.970 DEBUG:boto:Token: None 2022-09-13T07:36:34.971 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.972 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8jWsmlA+vW43jHHV/iaOhayW4xU= 2022-09-13T07:36:34.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8jWsmlA+vW43jHHV/iaOhayW4xU=', 'Content-Length': '0'} 2022-09-13T07:36:34.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8aaf5645d002729-0063203302-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.977 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:34.978 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:34.979 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.980 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.981 DEBUG:boto:Method: HEAD 2022-09-13T07:36:34.982 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.983 DEBUG:boto:Data: 2022-09-13T07:36:34.984 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.985 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:34.986 DEBUG:boto:Port: 8000 2022-09-13T07:36:34.987 DEBUG:boto:Params: {} 2022-09-13T07:36:34.988 DEBUG:boto:Token: None 2022-09-13T07:36:34.988 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:34 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.989 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8jWsmlA+vW43jHHV/iaOhayW4xU= 2022-09-13T07:36:34.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8jWsmlA+vW43jHHV/iaOhayW4xU=', 'Content-Length': '0'} 2022-09-13T07:36:34.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004be27dfdd5f2b9db-0063203302-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:34.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:34.993 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:34.994 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:34.994 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.995 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.995 DEBUG:boto:Method: GET 2022-09-13T07:36:34.996 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:34.996 DEBUG:boto:Data: 2022-09-13T07:36:34.997 DEBUG:boto:Headers: {} 2022-09-13T07:36:34.997 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:34.998 DEBUG:boto:Port: 8001 2022-09-13T07:36:34.998 DEBUG:boto:Params: {} 2022-09-13T07:36:34.999 DEBUG:boto:Token: None 2022-09-13T07:36:35.000 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:35.000 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mfsCnTunGcLeMRtascVjmCvYiV0= 2022-09-13T07:36:35.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mfsCnTunGcLeMRtascVjmCvYiV0=', 'Content-Length': '0'} 2022-09-13T07:36:35.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e99dd47b4b746db-0063203302-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.046 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:35.047 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:35.048 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:35.049 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:35.050 DEBUG:boto:Method: GET 2022-09-13T07:36:35.051 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:35.051 DEBUG:boto:Data: 2022-09-13T07:36:35.052 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.052 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.053 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.053 DEBUG:boto:Params: {} 2022-09-13T07:36:35.054 DEBUG:boto:Token: None 2022-09-13T07:36:35.055 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:35.055 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mfsCnTunGcLeMRtascVjmCvYiV0= 2022-09-13T07:36:35.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mfsCnTunGcLeMRtascVjmCvYiV0=', 'Content-Length': '0'} 2022-09-13T07:36:35.058 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a905f4eee835382-0063203303-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.060 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.061 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.061 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:35.062 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:35.063 DEBUG:boto:Method: GET 2022-09-13T07:36:35.064 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:35.065 DEBUG:boto:Data: 2022-09-13T07:36:35.066 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.067 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.067 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.068 DEBUG:boto:Params: {} 2022-09-13T07:36:35.069 DEBUG:boto:Token: None 2022-09-13T07:36:35.069 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.070 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU= 2022-09-13T07:36:35.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU=', 'Content-Length': '0'} 2022-09-13T07:36:35.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2884903b8c4996a-0063203303-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.074 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.075 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.076 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.076 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:35.077 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:35.077 DEBUG:boto:Method: GET 2022-09-13T07:36:35.078 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:35.078 DEBUG:boto:Data: 2022-09-13T07:36:35.079 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.079 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.080 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.080 DEBUG:boto:Params: {} 2022-09-13T07:36:35.081 DEBUG:boto:Token: None 2022-09-13T07:36:35.082 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.082 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU= 2022-09-13T07:36:35.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU=', 'Content-Length': '0'} 2022-09-13T07:36:35.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a8bed41a6dd7316-0063203303-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.087 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.088 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:35.089 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:35.090 DEBUG:boto:Method: HEAD 2022-09-13T07:36:35.091 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1 2022-09-13T07:36:35.092 DEBUG:boto:Data: 2022-09-13T07:36:35.092 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.093 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.094 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.094 DEBUG:boto:Params: {} 2022-09-13T07:36:35.095 DEBUG:boto:Token: None 2022-09-13T07:36:35.096 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone1 2022-09-13T07:36:35.097 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:f0O0gPI+vXeAzJEBwEVzNWxOH8g= 2022-09-13T07:36:35.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:f0O0gPI+vXeAzJEBwEVzNWxOH8g=', 'Content-Length': '0'} 2022-09-13T07:36:35.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065f5ea13cfdf54f6-0063203303-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.101 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:35.102 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:35.102 DEBUG:boto:Method: HEAD 2022-09-13T07:36:35.103 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1 2022-09-13T07:36:35.104 DEBUG:boto:Data: 2022-09-13T07:36:35.104 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.105 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.106 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.107 DEBUG:boto:Params: {} 2022-09-13T07:36:35.108 DEBUG:boto:Token: None 2022-09-13T07:36:35.109 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone1 2022-09-13T07:36:35.110 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:f0O0gPI+vXeAzJEBwEVzNWxOH8g= 2022-09-13T07:36:35.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:f0O0gPI+vXeAzJEBwEVzNWxOH8g=', 'Content-Length': '0'} 2022-09-13T07:36:35.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a87e99177d0be7c-0063203303-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:35.117 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:35.118 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:35.119 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:35.120 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:35.120 DEBUG:boto:Method: GET 2022-09-13T07:36:35.121 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:35.121 DEBUG:boto:Data: 2022-09-13T07:36:35.122 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.122 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.123 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.123 DEBUG:boto:Params: {} 2022-09-13T07:36:35.124 DEBUG:boto:Token: None 2022-09-13T07:36:35.125 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:35.125 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mfsCnTunGcLeMRtascVjmCvYiV0= 2022-09-13T07:36:35.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mfsCnTunGcLeMRtascVjmCvYiV0=', 'Content-Length': '0'} 2022-09-13T07:36:35.128 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ddc336277f2751e0-0063203303-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.129 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:35.130 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:35.130 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:35.131 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:35.131 DEBUG:boto:Method: GET 2022-09-13T07:36:35.132 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:35.133 DEBUG:boto:Data: 2022-09-13T07:36:35.133 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.134 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.134 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.135 DEBUG:boto:Params: {} 2022-09-13T07:36:35.135 DEBUG:boto:Token: None 2022-09-13T07:36:35.136 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:35.137 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mfsCnTunGcLeMRtascVjmCvYiV0= 2022-09-13T07:36:35.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mfsCnTunGcLeMRtascVjmCvYiV0=', 'Content-Length': '0'} 2022-09-13T07:36:35.140 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059fa401c11263efb-0063203303-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.183 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.184 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.185 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:35.185 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:35.186 DEBUG:boto:Method: GET 2022-09-13T07:36:35.189 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:35.190 DEBUG:boto:Data: 2022-09-13T07:36:35.191 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.192 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.192 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.193 DEBUG:boto:Params: {} 2022-09-13T07:36:35.194 DEBUG:boto:Token: None 2022-09-13T07:36:35.196 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.197 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU= 2022-09-13T07:36:35.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU=', 'Content-Length': '0'} 2022-09-13T07:36:35.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a11d78c8a758017-0063203303-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.207 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.208 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.209 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.210 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:35.211 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:35.212 DEBUG:boto:Method: GET 2022-09-13T07:36:35.213 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:35.214 DEBUG:boto:Data: 2022-09-13T07:36:35.215 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.216 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.217 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.218 DEBUG:boto:Params: {} 2022-09-13T07:36:35.219 DEBUG:boto:Token: None 2022-09-13T07:36:35.220 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.220 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU= 2022-09-13T07:36:35.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU=', 'Content-Length': '0'} 2022-09-13T07:36:35.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002148f17cf2a928ec-0063203303-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.224 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:35.226 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.226 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.227 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.227 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.228 DEBUG:boto:Method: GET 2022-09-13T07:36:35.228 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.229 DEBUG:boto:Data: 2022-09-13T07:36:35.230 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.230 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.231 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.231 DEBUG:boto:Params: {} 2022-09-13T07:36:35.232 DEBUG:boto:Token: None 2022-09-13T07:36:35.233 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.233 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YJZex47MSDW/OMdfRfTQ9YY8fNM= 2022-09-13T07:36:35.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YJZex47MSDW/OMdfRfTQ9YY8fNM=', 'Content-Length': '0'} 2022-09-13T07:36:35.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a2df56e15e19ee4-0063203303-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.237 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.238 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.238 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.239 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.240 DEBUG:boto:Method: GET 2022-09-13T07:36:35.240 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.241 DEBUG:boto:Data: 2022-09-13T07:36:35.241 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.242 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.242 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.243 DEBUG:boto:Params: {} 2022-09-13T07:36:35.244 DEBUG:boto:Token: None 2022-09-13T07:36:35.244 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.245 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YJZex47MSDW/OMdfRfTQ9YY8fNM= 2022-09-13T07:36:35.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YJZex47MSDW/OMdfRfTQ9YY8fNM=', 'Content-Length': '0'} 2022-09-13T07:36:35.248 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003fe199d968f18a31-0063203303-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.249 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.249 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.250 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.250 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.251 DEBUG:boto:Method: HEAD 2022-09-13T07:36:35.251 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.252 DEBUG:boto:Data: 2022-09-13T07:36:35.252 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.253 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.253 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.254 DEBUG:boto:Params: {} 2022-09-13T07:36:35.254 DEBUG:boto:Token: None 2022-09-13T07:36:35.255 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.256 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WTn0lHq/MpXPRU1kyJjVEVfqTjY= 2022-09-13T07:36:35.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WTn0lHq/MpXPRU1kyJjVEVfqTjY=', 'Content-Length': '0'} 2022-09-13T07:36:35.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3f61fe8d77be813-0063203303-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.259 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.260 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.261 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.262 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.263 DEBUG:boto:Method: HEAD 2022-09-13T07:36:35.264 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.264 DEBUG:boto:Data: 2022-09-13T07:36:35.265 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.265 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.266 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.266 DEBUG:boto:Params: {} 2022-09-13T07:36:35.267 DEBUG:boto:Token: None 2022-09-13T07:36:35.268 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.268 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WTn0lHq/MpXPRU1kyJjVEVfqTjY= 2022-09-13T07:36:35.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WTn0lHq/MpXPRU1kyJjVEVfqTjY=', 'Content-Length': '0'} 2022-09-13T07:36:35.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df516a46b6ee8314-0063203303-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:35.272 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.273 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.273 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.274 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.274 DEBUG:boto:Method: GET 2022-09-13T07:36:35.275 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.276 DEBUG:boto:Data: 2022-09-13T07:36:35.276 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.277 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.277 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.278 DEBUG:boto:Params: {} 2022-09-13T07:36:35.279 DEBUG:boto:Token: None 2022-09-13T07:36:35.279 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.280 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YJZex47MSDW/OMdfRfTQ9YY8fNM= 2022-09-13T07:36:35.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YJZex47MSDW/OMdfRfTQ9YY8fNM=', 'Content-Length': '0'} 2022-09-13T07:36:35.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af8de95f5d74155e-0063203303-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.284 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.285 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.286 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.287 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.288 DEBUG:boto:Method: GET 2022-09-13T07:36:35.289 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.290 DEBUG:boto:Data: 2022-09-13T07:36:35.291 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.292 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.293 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.294 DEBUG:boto:Params: {} 2022-09-13T07:36:35.295 DEBUG:boto:Token: None 2022-09-13T07:36:35.296 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.297 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YJZex47MSDW/OMdfRfTQ9YY8fNM= 2022-09-13T07:36:35.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YJZex47MSDW/OMdfRfTQ9YY8fNM=', 'Content-Length': '0'} 2022-09-13T07:36:35.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c9a3a82d087aa82-0063203303-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.303 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.304 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.304 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:35.305 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:35.307 DEBUG:boto:Method: GET 2022-09-13T07:36:35.307 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:35.308 DEBUG:boto:Data: 2022-09-13T07:36:35.308 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.309 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.309 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.310 DEBUG:boto:Params: {} 2022-09-13T07:36:35.311 DEBUG:boto:Token: None 2022-09-13T07:36:35.311 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.312 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU= 2022-09-13T07:36:35.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU=', 'Content-Length': '0'} 2022-09-13T07:36:35.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007111b9d1c137e8ed-0063203303-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.358 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.359 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.360 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.360 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:35.361 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:35.361 DEBUG:boto:Method: GET 2022-09-13T07:36:35.362 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:35.362 DEBUG:boto:Data: 2022-09-13T07:36:35.363 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.363 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.364 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.364 DEBUG:boto:Params: {} 2022-09-13T07:36:35.365 DEBUG:boto:Token: None 2022-09-13T07:36:35.366 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.366 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU= 2022-09-13T07:36:35.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU=', 'Content-Length': '0'} 2022-09-13T07:36:35.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b7fbb548fc92440-0063203303-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.370 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.371 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.372 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.372 DEBUG:boto:Method: HEAD 2022-09-13T07:36:35.373 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.373 DEBUG:boto:Data: 2022-09-13T07:36:35.373 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.374 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.375 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.375 DEBUG:boto:Params: {} 2022-09-13T07:36:35.376 DEBUG:boto:Token: None 2022-09-13T07:36:35.377 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.377 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RmNeFKjEtHA5FWnrJ0PzZXC3E5I= 2022-09-13T07:36:35.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RmNeFKjEtHA5FWnrJ0PzZXC3E5I=', 'Content-Length': '0'} 2022-09-13T07:36:35.380 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f892026a1174c1be-0063203303-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.381 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.381 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.382 DEBUG:boto:Method: HEAD 2022-09-13T07:36:35.385 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.386 DEBUG:boto:Data: 2022-09-13T07:36:35.386 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.387 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.387 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.388 DEBUG:boto:Params: {} 2022-09-13T07:36:35.388 DEBUG:boto:Token: None 2022-09-13T07:36:35.389 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.389 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RmNeFKjEtHA5FWnrJ0PzZXC3E5I= 2022-09-13T07:36:35.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RmNeFKjEtHA5FWnrJ0PzZXC3E5I=', 'Content-Length': '0'} 2022-09-13T07:36:35.393 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc45924085f61673-0063203303-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:35.394 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.395 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.395 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.396 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.396 DEBUG:boto:Method: GET 2022-09-13T07:36:35.397 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.397 DEBUG:boto:Data: 2022-09-13T07:36:35.398 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.398 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.399 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.399 DEBUG:boto:Params: {} 2022-09-13T07:36:35.400 DEBUG:boto:Token: None 2022-09-13T07:36:35.400 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.401 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YJZex47MSDW/OMdfRfTQ9YY8fNM= 2022-09-13T07:36:35.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YJZex47MSDW/OMdfRfTQ9YY8fNM=', 'Content-Length': '0'} 2022-09-13T07:36:35.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a92ff128704e1d0-0063203303-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.406 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.407 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:35.408 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.409 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.410 DEBUG:boto:Method: GET 2022-09-13T07:36:35.411 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.411 DEBUG:boto:Data: 2022-09-13T07:36:35.412 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.413 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.414 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.415 DEBUG:boto:Params: {} 2022-09-13T07:36:35.416 DEBUG:boto:Token: None 2022-09-13T07:36:35.417 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:35.418 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YJZex47MSDW/OMdfRfTQ9YY8fNM= 2022-09-13T07:36:35.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YJZex47MSDW/OMdfRfTQ9YY8fNM=', 'Content-Length': '0'} 2022-09-13T07:36:35.421 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014b4d79b869b3c56-0063203303-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.422 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.423 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.423 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:35.424 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:35.424 DEBUG:boto:Method: GET 2022-09-13T07:36:35.425 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:35.425 DEBUG:boto:Data: 2022-09-13T07:36:35.426 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.426 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.427 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.427 DEBUG:boto:Params: {} 2022-09-13T07:36:35.428 DEBUG:boto:Token: None 2022-09-13T07:36:35.429 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.429 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU= 2022-09-13T07:36:35.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU=', 'Content-Length': '0'} 2022-09-13T07:36:35.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaf4bd6127aa1d95-0063203303-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.433 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.434 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.435 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.435 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:35.436 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:35.436 DEBUG:boto:Method: GET 2022-09-13T07:36:35.437 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:35.437 DEBUG:boto:Data: 2022-09-13T07:36:35.438 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.438 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.439 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.439 DEBUG:boto:Params: {} 2022-09-13T07:36:35.440 DEBUG:boto:Token: None 2022-09-13T07:36:35.441 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.441 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU= 2022-09-13T07:36:35.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU=', 'Content-Length': '0'} 2022-09-13T07:36:35.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009227bc3094cb98fa-0063203303-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.445 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:35.446 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.447 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.447 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.448 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.448 DEBUG:boto:Method: GET 2022-09-13T07:36:35.449 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.449 DEBUG:boto:Data: 2022-09-13T07:36:35.450 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.451 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.451 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.452 DEBUG:boto:Params: {} 2022-09-13T07:36:35.452 DEBUG:boto:Token: None 2022-09-13T07:36:35.453 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.453 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AoknA2DH4qAwqXV/2oqxPhUDYz0= 2022-09-13T07:36:35.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AoknA2DH4qAwqXV/2oqxPhUDYz0=', 'Content-Length': '0'} 2022-09-13T07:36:35.456 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e93261a5b08babac-0063203303-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.457 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.458 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.458 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.459 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.459 DEBUG:boto:Method: GET 2022-09-13T07:36:35.460 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.460 DEBUG:boto:Data: 2022-09-13T07:36:35.461 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.461 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.462 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.462 DEBUG:boto:Params: {} 2022-09-13T07:36:35.463 DEBUG:boto:Token: None 2022-09-13T07:36:35.463 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.464 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AoknA2DH4qAwqXV/2oqxPhUDYz0= 2022-09-13T07:36:35.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AoknA2DH4qAwqXV/2oqxPhUDYz0=', 'Content-Length': '0'} 2022-09-13T07:36:35.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008dc401f3c7aa87d7-0063203303-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.468 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.468 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.469 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.469 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.470 DEBUG:boto:Method: HEAD 2022-09-13T07:36:35.470 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.471 DEBUG:boto:Data: 2022-09-13T07:36:35.471 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.472 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.472 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.472 DEBUG:boto:Params: {} 2022-09-13T07:36:35.473 DEBUG:boto:Token: None 2022-09-13T07:36:35.474 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.474 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kS5+jcNbP//Mfa8OiPfkCEJgwSk= 2022-09-13T07:36:35.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kS5+jcNbP//Mfa8OiPfkCEJgwSk=', 'Content-Length': '0'} 2022-09-13T07:36:35.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c3dd5efee5a580d-0063203303-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.478 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.478 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.479 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.479 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.479 DEBUG:boto:Method: HEAD 2022-09-13T07:36:35.480 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.480 DEBUG:boto:Data: 2022-09-13T07:36:35.481 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.481 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.482 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.482 DEBUG:boto:Params: {} 2022-09-13T07:36:35.483 DEBUG:boto:Token: None 2022-09-13T07:36:35.483 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.484 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kS5+jcNbP//Mfa8OiPfkCEJgwSk= 2022-09-13T07:36:35.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kS5+jcNbP//Mfa8OiPfkCEJgwSk=', 'Content-Length': '0'} 2022-09-13T07:36:35.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009acbbfff8227a8ad-0063203303-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:35.488 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.489 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.489 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.490 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.490 DEBUG:boto:Method: GET 2022-09-13T07:36:35.491 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.491 DEBUG:boto:Data: 2022-09-13T07:36:35.492 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.492 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.493 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.493 DEBUG:boto:Params: {} 2022-09-13T07:36:35.494 DEBUG:boto:Token: None 2022-09-13T07:36:35.494 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.495 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AoknA2DH4qAwqXV/2oqxPhUDYz0= 2022-09-13T07:36:35.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AoknA2DH4qAwqXV/2oqxPhUDYz0=', 'Content-Length': '0'} 2022-09-13T07:36:35.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abf506c7174e8818-0063203303-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.499 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.500 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.500 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.501 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.501 DEBUG:boto:Method: GET 2022-09-13T07:36:35.502 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.502 DEBUG:boto:Data: 2022-09-13T07:36:35.503 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.504 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.504 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.505 DEBUG:boto:Params: {} 2022-09-13T07:36:35.505 DEBUG:boto:Token: None 2022-09-13T07:36:35.506 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.507 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AoknA2DH4qAwqXV/2oqxPhUDYz0= 2022-09-13T07:36:35.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AoknA2DH4qAwqXV/2oqxPhUDYz0=', 'Content-Length': '0'} 2022-09-13T07:36:35.510 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fbbbbefbd718c2b-0063203303-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.554 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.555 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.555 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:35.556 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:35.556 DEBUG:boto:Method: GET 2022-09-13T07:36:35.557 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:35.557 DEBUG:boto:Data: 2022-09-13T07:36:35.558 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.559 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.560 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.560 DEBUG:boto:Params: {} 2022-09-13T07:36:35.561 DEBUG:boto:Token: None 2022-09-13T07:36:35.561 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.562 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU= 2022-09-13T07:36:35.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU=', 'Content-Length': '0'} 2022-09-13T07:36:35.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b95353cb35fa684-0063203303-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.566 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.567 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.568 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.568 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:35.569 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:35.570 DEBUG:boto:Method: GET 2022-09-13T07:36:35.570 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:35.571 DEBUG:boto:Data: 2022-09-13T07:36:35.572 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.572 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.573 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.573 DEBUG:boto:Params: {} 2022-09-13T07:36:35.574 DEBUG:boto:Token: None 2022-09-13T07:36:35.575 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.575 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU= 2022-09-13T07:36:35.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU=', 'Content-Length': '0'} 2022-09-13T07:36:35.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb663c177c89e59-0063203303-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.580 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.581 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.581 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.582 DEBUG:boto:Method: HEAD 2022-09-13T07:36:35.582 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.583 DEBUG:boto:Data: 2022-09-13T07:36:35.583 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.584 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.584 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.585 DEBUG:boto:Params: {} 2022-09-13T07:36:35.585 DEBUG:boto:Token: None 2022-09-13T07:36:35.586 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.587 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FOjJ9IAu8RZhjaO2jV09bcqD/Do= 2022-09-13T07:36:35.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FOjJ9IAu8RZhjaO2jV09bcqD/Do=', 'Content-Length': '0'} 2022-09-13T07:36:35.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ce345b1e65e03cc-0063203303-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.591 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.591 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.592 DEBUG:boto:Method: HEAD 2022-09-13T07:36:35.592 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.593 DEBUG:boto:Data: 2022-09-13T07:36:35.593 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.594 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.594 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.595 DEBUG:boto:Params: {} 2022-09-13T07:36:35.595 DEBUG:boto:Token: None 2022-09-13T07:36:35.596 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.596 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FOjJ9IAu8RZhjaO2jV09bcqD/Do= 2022-09-13T07:36:35.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FOjJ9IAu8RZhjaO2jV09bcqD/Do=', 'Content-Length': '0'} 2022-09-13T07:36:35.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024acd31909fb7924-0063203303-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:35.601 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.601 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.602 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.602 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.603 DEBUG:boto:Method: GET 2022-09-13T07:36:35.603 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.604 DEBUG:boto:Data: 2022-09-13T07:36:35.604 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.605 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.605 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.606 DEBUG:boto:Params: {} 2022-09-13T07:36:35.607 DEBUG:boto:Token: None 2022-09-13T07:36:35.607 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.608 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AoknA2DH4qAwqXV/2oqxPhUDYz0= 2022-09-13T07:36:35.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AoknA2DH4qAwqXV/2oqxPhUDYz0=', 'Content-Length': '0'} 2022-09-13T07:36:35.611 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a49af1e1de9305b6-0063203303-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.612 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.613 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:35.614 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.614 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.615 DEBUG:boto:Method: GET 2022-09-13T07:36:35.616 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.616 DEBUG:boto:Data: 2022-09-13T07:36:35.617 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.617 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.618 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.618 DEBUG:boto:Params: {} 2022-09-13T07:36:35.619 DEBUG:boto:Token: None 2022-09-13T07:36:35.620 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:35.621 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AoknA2DH4qAwqXV/2oqxPhUDYz0= 2022-09-13T07:36:35.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AoknA2DH4qAwqXV/2oqxPhUDYz0=', 'Content-Length': '0'} 2022-09-13T07:36:35.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e571c6a98bac1c4-0063203303-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.625 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.626 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.627 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:35.627 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:35.628 DEBUG:boto:Method: GET 2022-09-13T07:36:35.628 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:35.629 DEBUG:boto:Data: 2022-09-13T07:36:35.629 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.630 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.630 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.631 DEBUG:boto:Params: {} 2022-09-13T07:36:35.631 DEBUG:boto:Token: None 2022-09-13T07:36:35.632 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.632 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU= 2022-09-13T07:36:35.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU=', 'Content-Length': '0'} 2022-09-13T07:36:35.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c95e862393fa4fcd-0063203303-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.636 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.637 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.638 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.638 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:35.639 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:35.639 DEBUG:boto:Method: GET 2022-09-13T07:36:35.640 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:35.640 DEBUG:boto:Data: 2022-09-13T07:36:35.641 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.641 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:35.642 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.642 DEBUG:boto:Params: {} 2022-09-13T07:36:35.643 DEBUG:boto:Token: None 2022-09-13T07:36:35.643 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.644 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU= 2022-09-13T07:36:35.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU=', 'Content-Length': '0'} 2022-09-13T07:36:35.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b193cf3277c37f6f-0063203303-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.648 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.649 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-18 zones={test-zone3, test-zone1} 2022-09-13T07:36:35.650 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-18 zones={test-zone3, test-zone2} 2022-09-13T07:36:35.650 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.651 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.651 DEBUG:boto:Method: HEAD 2022-09-13T07:36:35.652 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:36:35.652 DEBUG:boto:Data: 2022-09-13T07:36:35.653 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.653 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.654 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.654 DEBUG:boto:Params: {} 2022-09-13T07:36:35.655 DEBUG:boto:Token: None 2022-09-13T07:36:35.656 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.656 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZE4rUgy+btX665wkwBcs9f+whHo= 2022-09-13T07:36:35.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZE4rUgy+btX665wkwBcs9f+whHo=', 'Content-Length': '0'} 2022-09-13T07:36:35.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fb1f317f8d65191-0063203303-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.661 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.662 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.662 DEBUG:boto:Method: HEAD 2022-09-13T07:36:35.663 DEBUG:boto:Path: /yxvvuh-18/ 2022-09-13T07:36:35.663 DEBUG:boto:Data: 2022-09-13T07:36:35.664 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.664 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:35.665 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.665 DEBUG:boto:Params: {} 2022-09-13T07:36:35.666 DEBUG:boto:Token: None 2022-09-13T07:36:35.667 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.667 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZE4rUgy+btX665wkwBcs9f+whHo= 2022-09-13T07:36:35.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZE4rUgy+btX665wkwBcs9f+whHo=', 'Content-Length': '0'} 2022-09-13T07:36:35.671 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a1365eef6c62713-0063203303-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.672 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:36:35.672 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.673 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.673 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:35.674 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:35.674 DEBUG:boto:Method: GET 2022-09-13T07:36:35.675 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:35.675 DEBUG:boto:Data: 2022-09-13T07:36:35.676 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.676 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.677 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.678 DEBUG:boto:Params: {} 2022-09-13T07:36:35.678 DEBUG:boto:Token: None 2022-09-13T07:36:35.679 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/?versions 2022-09-13T07:36:35.679 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xrinqoGkATYbd3LF/V1sFokxlUw= 2022-09-13T07:36:35.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xrinqoGkATYbd3LF/V1sFokxlUw=', 'Content-Length': '0'} 2022-09-13T07:36:35.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090da7ceead188890-0063203303-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.683 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.684 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:36:35.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:36:35.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:36:35.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:36:35.686 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:36:35.687 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.687 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.688 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:35.689 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:35.690 DEBUG:boto:Method: GET 2022-09-13T07:36:35.691 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:35.692 DEBUG:boto:Data: 2022-09-13T07:36:35.693 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.694 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:35.695 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.695 DEBUG:boto:Params: {} 2022-09-13T07:36:35.697 DEBUG:boto:Token: None 2022-09-13T07:36:35.698 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/?versions 2022-09-13T07:36:35.699 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xrinqoGkATYbd3LF/V1sFokxlUw= 2022-09-13T07:36:35.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xrinqoGkATYbd3LF/V1sFokxlUw=', 'Content-Length': '0'} 2022-09-13T07:36:35.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0458dcd8bade7e6-0063203303-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.704 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.705 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:36:35.706 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-13T07:36:35.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-13T07:36:35.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-13T07:36:35.708 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.709 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.709 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:35.710 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:35.710 DEBUG:boto:Method: GET 2022-09-13T07:36:35.711 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:35.711 DEBUG:boto:Data: 2022-09-13T07:36:35.712 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.712 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.713 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.713 DEBUG:boto:Params: {} 2022-09-13T07:36:35.714 DEBUG:boto:Token: None 2022-09-13T07:36:35.715 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/?versions 2022-09-13T07:36:35.715 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xrinqoGkATYbd3LF/V1sFokxlUw= 2022-09-13T07:36:35.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xrinqoGkATYbd3LF/V1sFokxlUw=', 'Content-Length': '0'} 2022-09-13T07:36:35.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021423b0a015bd44d-0063203303-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.719 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.721 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.721 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.722 DEBUG:boto:path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:35.722 DEBUG:boto:auth_path=/yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:35.723 DEBUG:boto:Method: GET 2022-09-13T07:36:35.723 DEBUG:boto:Path: /yxvvuh-18/?versions&max-keys=999 2022-09-13T07:36:35.724 DEBUG:boto:Data: 2022-09-13T07:36:35.725 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.725 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:35.726 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.726 DEBUG:boto:Params: {} 2022-09-13T07:36:35.727 DEBUG:boto:Token: None 2022-09-13T07:36:35.727 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/?versions 2022-09-13T07:36:35.728 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xrinqoGkATYbd3LF/V1sFokxlUw= 2022-09-13T07:36:35.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xrinqoGkATYbd3LF/V1sFokxlUw=', 'Content-Length': '0'} 2022-09-13T07:36:35.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cde2bcec8a3b0589-0063203303-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.732 DEBUG:boto:b'yxvvuh-18999falsedummyC7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0true2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1peYXnjgsVROpSRMqu7BDytKSaBCENfftrue2022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone24MFZ9p0FICPqR5Wxj87ukarF6wECv5Ftrue2022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jGr2ahP3JH9MIikQrDceF1PJUmw6xXhtrue2022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:35.734 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:35.734 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:35.735 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.735 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.736 DEBUG:boto:Method: GET 2022-09-13T07:36:35.736 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.737 DEBUG:boto:Data: 2022-09-13T07:36:35.738 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.738 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.739 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.740 DEBUG:boto:Params: {} 2022-09-13T07:36:35.741 DEBUG:boto:Token: None 2022-09-13T07:36:35.742 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.743 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ek0I4+pB8RlvIuscLIu3PgNsz6o= 2022-09-13T07:36:35.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ek0I4+pB8RlvIuscLIu3PgNsz6o=', 'Content-Length': '0'} 2022-09-13T07:36:35.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f1aa80cc1b3568fd-0063203303-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.749 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:35.749 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:35.750 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.751 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.752 DEBUG:boto:Method: GET 2022-09-13T07:36:35.753 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.754 DEBUG:boto:Data: 2022-09-13T07:36:35.755 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.756 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:35.757 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.758 DEBUG:boto:Params: {} 2022-09-13T07:36:35.759 DEBUG:boto:Token: None 2022-09-13T07:36:35.760 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.761 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ek0I4+pB8RlvIuscLIu3PgNsz6o= 2022-09-13T07:36:35.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ek0I4+pB8RlvIuscLIu3PgNsz6o=', 'Content-Length': '0'} 2022-09-13T07:36:35.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007212dd7ca0cc78bd-0063203303-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.767 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:35.768 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:35.768 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.769 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.770 DEBUG:boto:Method: HEAD 2022-09-13T07:36:35.771 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.772 DEBUG:boto:Data: 2022-09-13T07:36:35.773 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.774 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.775 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.776 DEBUG:boto:Params: {} 2022-09-13T07:36:35.777 DEBUG:boto:Token: None 2022-09-13T07:36:35.778 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.779 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9GT5/NZEcFiM/euJfJOq/mcl4i0= 2022-09-13T07:36:35.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9GT5/NZEcFiM/euJfJOq/mcl4i0=', 'Content-Length': '0'} 2022-09-13T07:36:35.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bdb3de8c60e309e3-0063203303-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.784 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:35.785 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:35.785 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.786 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.787 DEBUG:boto:Method: HEAD 2022-09-13T07:36:35.788 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.789 DEBUG:boto:Data: 2022-09-13T07:36:35.790 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.791 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:35.792 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.793 DEBUG:boto:Params: {} 2022-09-13T07:36:35.794 DEBUG:boto:Token: None 2022-09-13T07:36:35.795 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.796 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9GT5/NZEcFiM/euJfJOq/mcl4i0= 2022-09-13T07:36:35.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9GT5/NZEcFiM/euJfJOq/mcl4i0=', 'Content-Length': '0'} 2022-09-13T07:36:35.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000079e186f323416226-0063203303-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:35.801 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:35.802 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:35.803 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.804 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.805 DEBUG:boto:Method: GET 2022-09-13T07:36:35.806 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.807 DEBUG:boto:Data: 2022-09-13T07:36:35.808 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.809 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.810 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.811 DEBUG:boto:Params: {} 2022-09-13T07:36:35.812 DEBUG:boto:Token: None 2022-09-13T07:36:35.813 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.814 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ek0I4+pB8RlvIuscLIu3PgNsz6o= 2022-09-13T07:36:35.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ek0I4+pB8RlvIuscLIu3PgNsz6o=', 'Content-Length': '0'} 2022-09-13T07:36:35.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005c2664eb48bf13b1-0063203303-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.819 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:35.820 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:35.821 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.822 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.823 DEBUG:boto:Method: GET 2022-09-13T07:36:35.824 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.825 DEBUG:boto:Data: 2022-09-13T07:36:35.826 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.827 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:35.828 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.829 DEBUG:boto:Params: {} 2022-09-13T07:36:35.830 DEBUG:boto:Token: None 2022-09-13T07:36:35.831 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:35.832 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ek0I4+pB8RlvIuscLIu3PgNsz6o= 2022-09-13T07:36:35.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ek0I4+pB8RlvIuscLIu3PgNsz6o=', 'Content-Length': '0'} 2022-09-13T07:36:35.836 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c306bb64d1d9f115-0063203303-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.942 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.943 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.945 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:35.946 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:35.946 DEBUG:boto:Method: GET 2022-09-13T07:36:35.947 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:36:35.948 DEBUG:boto:Data: 2022-09-13T07:36:35.949 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.949 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:35.950 DEBUG:boto:Port: 8001 2022-09-13T07:36:35.951 DEBUG:boto:Params: {} 2022-09-13T07:36:35.952 DEBUG:boto:Token: None 2022-09-13T07:36:35.953 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.954 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU= 2022-09-13T07:36:35.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU=', 'Content-Length': '0'} 2022-09-13T07:36:35.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab70e78fd5245cab-0063203303-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.963 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.965 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:35.966 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:35.967 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:35.967 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:35.968 DEBUG:boto:Method: GET 2022-09-13T07:36:35.969 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:36:35.970 DEBUG:boto:Data: 2022-09-13T07:36:35.971 DEBUG:boto:Headers: {} 2022-09-13T07:36:35.972 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:35.973 DEBUG:boto:Port: 8000 2022-09-13T07:36:35.974 DEBUG:boto:Params: {} 2022-09-13T07:36:35.974 DEBUG:boto:Token: None 2022-09-13T07:36:35.975 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:35 GMT /yxvvuh-18/ 2022-09-13T07:36:35.976 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU= 2022-09-13T07:36:35.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZJvm1Bv2cutL/NrgTJ9ZbNeiAjU=', 'Content-Length': '0'} 2022-09-13T07:36:35.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8c7db564f05c74b-0063203303-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:35.994 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:35.996 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:35.998 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:35.999 DEBUG:boto:Method: HEAD 2022-09-13T07:36:36.000 DEBUG:boto:Path: /yxvvuh-18/dummy 2022-09-13T07:36:36.001 DEBUG:boto:Data: 2022-09-13T07:36:36.001 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.002 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.003 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.004 DEBUG:boto:Params: {} 2022-09-13T07:36:36.005 DEBUG:boto:Token: None 2022-09-13T07:36:36.006 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/dummy 2022-09-13T07:36:36.007 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:l7D65FkHe5BBsFTQNap3C3/ef+Q= 2022-09-13T07:36:36.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:l7D65FkHe5BBsFTQNap3C3/ef+Q=', 'Content-Length': '0'} 2022-09-13T07:36:36.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dc6127eaeb863d78-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.021 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:36.022 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:36.023 DEBUG:boto:Method: HEAD 2022-09-13T07:36:36.023 DEBUG:boto:Path: /yxvvuh-18/dummy 2022-09-13T07:36:36.024 DEBUG:boto:Data: 2022-09-13T07:36:36.025 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.025 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.026 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.026 DEBUG:boto:Params: {} 2022-09-13T07:36:36.027 DEBUG:boto:Token: None 2022-09-13T07:36:36.028 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/dummy 2022-09-13T07:36:36.028 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:l7D65FkHe5BBsFTQNap3C3/ef+Q= 2022-09-13T07:36:36.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:l7D65FkHe5BBsFTQNap3C3/ef+Q=', 'Content-Length': '0'} 2022-09-13T07:36:36.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f8b942d832c5ed58-0063203304-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:36:36.041 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:36.043 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:36.043 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:36.044 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:36.045 DEBUG:boto:Method: GET 2022-09-13T07:36:36.048 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:36.051 DEBUG:boto:Data: 2022-09-13T07:36:36.052 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.052 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.053 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.054 DEBUG:boto:Params: {} 2022-09-13T07:36:36.055 DEBUG:boto:Token: None 2022-09-13T07:36:36.055 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:36.056 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:udLMp1wEdXbIrNsSeXxDxlUI8gU= 2022-09-13T07:36:36.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:udLMp1wEdXbIrNsSeXxDxlUI8gU=', 'Content-Length': '0'} 2022-09-13T07:36:36.065 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009eb57985bda72112-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.067 DEBUG:boto:path=/yxvvuh-18/dummy 2022-09-13T07:36:36.068 DEBUG:boto:auth_path=/yxvvuh-18/dummy 2022-09-13T07:36:36.069 DEBUG:boto:path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:36.070 DEBUG:boto:auth_path=/yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:36.071 DEBUG:boto:Method: GET 2022-09-13T07:36:36.071 DEBUG:boto:Path: /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:36.072 DEBUG:boto:Data: 2022-09-13T07:36:36.073 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.073 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.074 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.075 DEBUG:boto:Params: {} 2022-09-13T07:36:36.075 DEBUG:boto:Token: None 2022-09-13T07:36:36.076 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/dummy?versionId=C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0 2022-09-13T07:36:36.077 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:udLMp1wEdXbIrNsSeXxDxlUI8gU= 2022-09-13T07:36:36.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:udLMp1wEdXbIrNsSeXxDxlUI8gU=', 'Content-Length': '0'} 2022-09-13T07:36:36.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:31:56 GMT'), ('x-amz-version-id', 'C7AZwX4cTQgb-BxQ8N3vj4voy5QrRL0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af00d2e5022ae5e4-0063203304-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.092 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:36.093 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:36.094 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:36.095 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:36.096 DEBUG:boto:Method: GET 2022-09-13T07:36:36.096 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:36:36.097 DEBUG:boto:Data: 2022-09-13T07:36:36.097 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.098 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.098 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.099 DEBUG:boto:Params: {} 2022-09-13T07:36:36.100 DEBUG:boto:Token: None 2022-09-13T07:36:36.100 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/ 2022-09-13T07:36:36.103 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68= 2022-09-13T07:36:36.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68=', 'Content-Length': '0'} 2022-09-13T07:36:36.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039430084d6a2f37a-0063203304-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.119 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:36.120 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:36.120 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:36.121 DEBUG:boto:path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:36.121 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=dummy 2022-09-13T07:36:36.122 DEBUG:boto:Method: GET 2022-09-13T07:36:36.122 DEBUG:boto:Path: /yxvvuh-18/?prefix=dummy 2022-09-13T07:36:36.123 DEBUG:boto:Data: 2022-09-13T07:36:36.123 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.124 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.124 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.125 DEBUG:boto:Params: {} 2022-09-13T07:36:36.125 DEBUG:boto:Token: None 2022-09-13T07:36:36.126 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/ 2022-09-13T07:36:36.127 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68= 2022-09-13T07:36:36.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68=', 'Content-Length': '0'} 2022-09-13T07:36:36.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db1af95723674b07-0063203304-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.131 DEBUG:boto:b'yxvvuh-18dummy1000falsedummy2022-09-13T07:31:56.224Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:36.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:36.133 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.134 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.134 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.135 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.135 DEBUG:boto:Method: GET 2022-09-13T07:36:36.136 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.137 DEBUG:boto:Data: 2022-09-13T07:36:36.137 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.138 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.139 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.139 DEBUG:boto:Params: {} 2022-09-13T07:36:36.140 DEBUG:boto:Token: None 2022-09-13T07:36:36.141 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.141 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mlSY4/4yEGEMmjS8MFa690boMD8= 2022-09-13T07:36:36.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mlSY4/4yEGEMmjS8MFa690boMD8=', 'Content-Length': '0'} 2022-09-13T07:36:36.147 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed9a847440756b62-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.148 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.148 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.149 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.149 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.150 DEBUG:boto:Method: GET 2022-09-13T07:36:36.151 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.151 DEBUG:boto:Data: 2022-09-13T07:36:36.152 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.152 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.153 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.153 DEBUG:boto:Params: {} 2022-09-13T07:36:36.154 DEBUG:boto:Token: None 2022-09-13T07:36:36.154 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.155 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mlSY4/4yEGEMmjS8MFa690boMD8= 2022-09-13T07:36:36.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mlSY4/4yEGEMmjS8MFa690boMD8=', 'Content-Length': '0'} 2022-09-13T07:36:36.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000878b08f7d3c9949d-0063203304-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.165 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.166 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.167 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.167 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.168 DEBUG:boto:Method: HEAD 2022-09-13T07:36:36.169 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.170 DEBUG:boto:Data: 2022-09-13T07:36:36.171 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.172 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.172 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.173 DEBUG:boto:Params: {} 2022-09-13T07:36:36.173 DEBUG:boto:Token: None 2022-09-13T07:36:36.174 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.175 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jkJgK8suTMvF6E8CaLrY4jNOipI= 2022-09-13T07:36:36.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jkJgK8suTMvF6E8CaLrY4jNOipI=', 'Content-Length': '0'} 2022-09-13T07:36:36.182 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000116b3655e5cc0a60-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.184 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.185 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.186 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.187 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.188 DEBUG:boto:Method: HEAD 2022-09-13T07:36:36.189 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.192 DEBUG:boto:Data: 2022-09-13T07:36:36.193 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.193 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.194 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.195 DEBUG:boto:Params: {} 2022-09-13T07:36:36.196 DEBUG:boto:Token: None 2022-09-13T07:36:36.197 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.197 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jkJgK8suTMvF6E8CaLrY4jNOipI= 2022-09-13T07:36:36.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jkJgK8suTMvF6E8CaLrY4jNOipI=', 'Content-Length': '0'} 2022-09-13T07:36:36.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022f0fd836b63f800-0063203304-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:36.205 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.206 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.207 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.207 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.208 DEBUG:boto:Method: GET 2022-09-13T07:36:36.208 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.209 DEBUG:boto:Data: 2022-09-13T07:36:36.210 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.210 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.219 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.219 DEBUG:boto:Params: {} 2022-09-13T07:36:36.220 DEBUG:boto:Token: None 2022-09-13T07:36:36.221 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.221 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mlSY4/4yEGEMmjS8MFa690boMD8= 2022-09-13T07:36:36.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mlSY4/4yEGEMmjS8MFa690boMD8=', 'Content-Length': '0'} 2022-09-13T07:36:36.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bec3c41152064bb-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.276 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.277 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.278 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.280 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.281 DEBUG:boto:Method: GET 2022-09-13T07:36:36.282 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.283 DEBUG:boto:Data: 2022-09-13T07:36:36.284 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.285 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.286 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.287 DEBUG:boto:Params: {} 2022-09-13T07:36:36.288 DEBUG:boto:Token: None 2022-09-13T07:36:36.289 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.290 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mlSY4/4yEGEMmjS8MFa690boMD8= 2022-09-13T07:36:36.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mlSY4/4yEGEMmjS8MFa690boMD8=', 'Content-Length': '0'} 2022-09-13T07:36:36.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010ca63d8eb993a99-0063203304-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.299 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:36.300 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:36.302 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:36.303 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:36.304 DEBUG:boto:Method: GET 2022-09-13T07:36:36.306 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:36.308 DEBUG:boto:Data: 2022-09-13T07:36:36.310 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.311 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.313 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.317 DEBUG:boto:Params: {} 2022-09-13T07:36:36.318 DEBUG:boto:Token: None 2022-09-13T07:36:36.318 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/ 2022-09-13T07:36:36.319 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68= 2022-09-13T07:36:36.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68=', 'Content-Length': '0'} 2022-09-13T07:36:36.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f64ae87186c38b5-0063203304-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.328 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:36.332 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:36.333 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:36.335 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:36.336 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:36.337 DEBUG:boto:Method: GET 2022-09-13T07:36:36.338 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:36.339 DEBUG:boto:Data: 2022-09-13T07:36:36.340 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.341 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.342 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.343 DEBUG:boto:Params: {} 2022-09-13T07:36:36.344 DEBUG:boto:Token: None 2022-09-13T07:36:36.345 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/ 2022-09-13T07:36:36.346 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68= 2022-09-13T07:36:36.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68=', 'Content-Length': '0'} 2022-09-13T07:36:36.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7f86afdec924507-0063203304-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.356 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:36.358 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.359 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.360 DEBUG:boto:Method: HEAD 2022-09-13T07:36:36.361 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.362 DEBUG:boto:Data: 2022-09-13T07:36:36.362 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.363 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.363 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.364 DEBUG:boto:Params: {} 2022-09-13T07:36:36.365 DEBUG:boto:Token: None 2022-09-13T07:36:36.365 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.366 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:db4XfqLdaAJMwaLaXyf21kn2QK4= 2022-09-13T07:36:36.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:db4XfqLdaAJMwaLaXyf21kn2QK4=', 'Content-Length': '0'} 2022-09-13T07:36:36.369 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ffb085733d698d7-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.370 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.371 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.372 DEBUG:boto:Method: HEAD 2022-09-13T07:36:36.373 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.373 DEBUG:boto:Data: 2022-09-13T07:36:36.374 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.375 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.375 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.376 DEBUG:boto:Params: {} 2022-09-13T07:36:36.376 DEBUG:boto:Token: None 2022-09-13T07:36:36.377 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.378 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:db4XfqLdaAJMwaLaXyf21kn2QK4= 2022-09-13T07:36:36.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:db4XfqLdaAJMwaLaXyf21kn2QK4=', 'Content-Length': '0'} 2022-09-13T07:36:36.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012a7ec3f526f1da4-0063203304-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-13T07:36:36.386 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.387 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.388 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.388 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.389 DEBUG:boto:Method: GET 2022-09-13T07:36:36.390 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.390 DEBUG:boto:Data: 2022-09-13T07:36:36.391 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.391 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.392 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.393 DEBUG:boto:Params: {} 2022-09-13T07:36:36.395 DEBUG:boto:Token: None 2022-09-13T07:36:36.396 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.397 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mlSY4/4yEGEMmjS8MFa690boMD8= 2022-09-13T07:36:36.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mlSY4/4yEGEMmjS8MFa690boMD8=', 'Content-Length': '0'} 2022-09-13T07:36:36.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b52fee13e6b1d4d2-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.401 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.402 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1 2022-09-13T07:36:36.403 DEBUG:boto:path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.403 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.404 DEBUG:boto:Method: GET 2022-09-13T07:36:36.404 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.405 DEBUG:boto:Data: 2022-09-13T07:36:36.405 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.406 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.406 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.407 DEBUG:boto:Params: {} 2022-09-13T07:36:36.408 DEBUG:boto:Token: None 2022-09-13T07:36:36.408 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone1?versionId=peYXnjgsVROpSRMqu7BDytKSaBCENff 2022-09-13T07:36:36.409 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mlSY4/4yEGEMmjS8MFa690boMD8= 2022-09-13T07:36:36.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mlSY4/4yEGEMmjS8MFa690boMD8=', 'Content-Length': '0'} 2022-09-13T07:36:36.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'peYXnjgsVROpSRMqu7BDytKSaBCENff'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a05623aded2aca4d-0063203304-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.413 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:36.413 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:36.414 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:36.415 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:36.416 DEBUG:boto:Method: GET 2022-09-13T07:36:36.417 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:36.417 DEBUG:boto:Data: 2022-09-13T07:36:36.418 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.419 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.420 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.420 DEBUG:boto:Params: {} 2022-09-13T07:36:36.421 DEBUG:boto:Token: None 2022-09-13T07:36:36.421 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/ 2022-09-13T07:36:36.422 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68= 2022-09-13T07:36:36.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68=', 'Content-Length': '0'} 2022-09-13T07:36:36.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041865aafb13f4f09-0063203304-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.435 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:36.436 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:36.437 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:36.437 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:36.438 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:36.439 DEBUG:boto:Method: GET 2022-09-13T07:36:36.440 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone1 2022-09-13T07:36:36.440 DEBUG:boto:Data: 2022-09-13T07:36:36.441 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.442 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.442 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.445 DEBUG:boto:Params: {} 2022-09-13T07:36:36.446 DEBUG:boto:Token: None 2022-09-13T07:36:36.446 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/ 2022-09-13T07:36:36.447 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68= 2022-09-13T07:36:36.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68=', 'Content-Length': '0'} 2022-09-13T07:36:36.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fff8543610e413f3-0063203304-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.457 DEBUG:boto:b'yxvvuh-18obj-test-zone11000falseobj-test-zone12022-09-13T07:32:07.408Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:36.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:36.461 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.462 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.462 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.463 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.464 DEBUG:boto:Method: GET 2022-09-13T07:36:36.465 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.466 DEBUG:boto:Data: 2022-09-13T07:36:36.467 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.468 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.469 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.470 DEBUG:boto:Params: {} 2022-09-13T07:36:36.471 DEBUG:boto:Token: None 2022-09-13T07:36:36.472 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.474 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:95KfFgziw4A7/TvbbHAaOGukAm4= 2022-09-13T07:36:36.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:95KfFgziw4A7/TvbbHAaOGukAm4=', 'Content-Length': '0'} 2022-09-13T07:36:36.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000dc5d99b398b8794-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.484 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.485 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.485 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.486 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.487 DEBUG:boto:Method: GET 2022-09-13T07:36:36.489 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.489 DEBUG:boto:Data: 2022-09-13T07:36:36.490 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.491 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.492 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.493 DEBUG:boto:Params: {} 2022-09-13T07:36:36.493 DEBUG:boto:Token: None 2022-09-13T07:36:36.495 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.495 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:95KfFgziw4A7/TvbbHAaOGukAm4= 2022-09-13T07:36:36.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:95KfFgziw4A7/TvbbHAaOGukAm4=', 'Content-Length': '0'} 2022-09-13T07:36:36.499 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008873b15725331fa7-0063203304-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.501 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.502 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.503 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.504 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.505 DEBUG:boto:Method: HEAD 2022-09-13T07:36:36.505 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.506 DEBUG:boto:Data: 2022-09-13T07:36:36.506 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.507 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.507 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.508 DEBUG:boto:Params: {} 2022-09-13T07:36:36.509 DEBUG:boto:Token: None 2022-09-13T07:36:36.509 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.510 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pV1YokfF/c9TujUzkSf7x8X/Les= 2022-09-13T07:36:36.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pV1YokfF/c9TujUzkSf7x8X/Les=', 'Content-Length': '0'} 2022-09-13T07:36:36.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000413f8539dd8a38a4-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.522 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.522 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.523 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.524 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.524 DEBUG:boto:Method: HEAD 2022-09-13T07:36:36.525 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.525 DEBUG:boto:Data: 2022-09-13T07:36:36.526 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.526 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.527 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.528 DEBUG:boto:Params: {} 2022-09-13T07:36:36.528 DEBUG:boto:Token: None 2022-09-13T07:36:36.529 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.529 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pV1YokfF/c9TujUzkSf7x8X/Les= 2022-09-13T07:36:36.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pV1YokfF/c9TujUzkSf7x8X/Les=', 'Content-Length': '0'} 2022-09-13T07:36:36.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c6c21200c3658a5-0063203304-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:36.536 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.536 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.537 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.537 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.538 DEBUG:boto:Method: GET 2022-09-13T07:36:36.539 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.539 DEBUG:boto:Data: 2022-09-13T07:36:36.540 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.540 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.541 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.542 DEBUG:boto:Params: {} 2022-09-13T07:36:36.542 DEBUG:boto:Token: None 2022-09-13T07:36:36.543 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.544 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:95KfFgziw4A7/TvbbHAaOGukAm4= 2022-09-13T07:36:36.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:95KfFgziw4A7/TvbbHAaOGukAm4=', 'Content-Length': '0'} 2022-09-13T07:36:36.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009859d2e280e344c9-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.597 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.599 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.600 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.601 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.602 DEBUG:boto:Method: GET 2022-09-13T07:36:36.603 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.604 DEBUG:boto:Data: 2022-09-13T07:36:36.605 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.606 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.607 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.608 DEBUG:boto:Params: {} 2022-09-13T07:36:36.610 DEBUG:boto:Token: None 2022-09-13T07:36:36.611 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.612 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:95KfFgziw4A7/TvbbHAaOGukAm4= 2022-09-13T07:36:36.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:95KfFgziw4A7/TvbbHAaOGukAm4=', 'Content-Length': '0'} 2022-09-13T07:36:36.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f752100a7c08636-0063203304-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.620 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:36.623 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:36.624 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:36.625 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:36.625 DEBUG:boto:Method: GET 2022-09-13T07:36:36.626 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:36.627 DEBUG:boto:Data: 2022-09-13T07:36:36.628 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.629 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.630 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.631 DEBUG:boto:Params: {} 2022-09-13T07:36:36.632 DEBUG:boto:Token: None 2022-09-13T07:36:36.633 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/ 2022-09-13T07:36:36.635 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68= 2022-09-13T07:36:36.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68=', 'Content-Length': '0'} 2022-09-13T07:36:36.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d553f32b1fb6b6bc-0063203304-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.641 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:36.642 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:36.643 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:36.644 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:36.645 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:36.646 DEBUG:boto:Method: GET 2022-09-13T07:36:36.647 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:36.648 DEBUG:boto:Data: 2022-09-13T07:36:36.649 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.650 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.651 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.652 DEBUG:boto:Params: {} 2022-09-13T07:36:36.653 DEBUG:boto:Token: None 2022-09-13T07:36:36.655 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/ 2022-09-13T07:36:36.656 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68= 2022-09-13T07:36:36.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68=', 'Content-Length': '0'} 2022-09-13T07:36:36.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000024f596e45b367d0-0063203304-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.662 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:36.664 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.665 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.665 DEBUG:boto:Method: HEAD 2022-09-13T07:36:36.667 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.667 DEBUG:boto:Data: 2022-09-13T07:36:36.668 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.669 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.670 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.671 DEBUG:boto:Params: {} 2022-09-13T07:36:36.672 DEBUG:boto:Token: None 2022-09-13T07:36:36.673 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.674 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4NSupT+xaPjkdr1JTQOeTEPt6vg= 2022-09-13T07:36:36.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4NSupT+xaPjkdr1JTQOeTEPt6vg=', 'Content-Length': '0'} 2022-09-13T07:36:36.684 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002545a05a672f118a-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.686 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.688 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.689 DEBUG:boto:Method: HEAD 2022-09-13T07:36:36.690 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.691 DEBUG:boto:Data: 2022-09-13T07:36:36.692 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.693 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.694 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.695 DEBUG:boto:Params: {} 2022-09-13T07:36:36.697 DEBUG:boto:Token: None 2022-09-13T07:36:36.698 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.699 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4NSupT+xaPjkdr1JTQOeTEPt6vg= 2022-09-13T07:36:36.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4NSupT+xaPjkdr1JTQOeTEPt6vg=', 'Content-Length': '0'} 2022-09-13T07:36:36.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012d4f66e6baa1c68-0063203304-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-13T07:36:36.713 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.715 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.716 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.716 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.717 DEBUG:boto:Method: GET 2022-09-13T07:36:36.717 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.718 DEBUG:boto:Data: 2022-09-13T07:36:36.719 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.720 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.720 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.722 DEBUG:boto:Params: {} 2022-09-13T07:36:36.724 DEBUG:boto:Token: None 2022-09-13T07:36:36.725 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.727 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:95KfFgziw4A7/TvbbHAaOGukAm4= 2022-09-13T07:36:36.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:95KfFgziw4A7/TvbbHAaOGukAm4=', 'Content-Length': '0'} 2022-09-13T07:36:36.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021008bb351a59fe3-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.734 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.735 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2 2022-09-13T07:36:36.736 DEBUG:boto:path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.737 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.738 DEBUG:boto:Method: GET 2022-09-13T07:36:36.739 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.740 DEBUG:boto:Data: 2022-09-13T07:36:36.740 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.741 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.742 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.744 DEBUG:boto:Params: {} 2022-09-13T07:36:36.745 DEBUG:boto:Token: None 2022-09-13T07:36:36.746 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone2?versionId=4MFZ9p0FICPqR5Wxj87ukarF6wECv5F 2022-09-13T07:36:36.747 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:95KfFgziw4A7/TvbbHAaOGukAm4= 2022-09-13T07:36:36.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:95KfFgziw4A7/TvbbHAaOGukAm4=', 'Content-Length': '0'} 2022-09-13T07:36:36.752 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', '4MFZ9p0FICPqR5Wxj87ukarF6wECv5F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bc226243fa3c01d-0063203304-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.754 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:36.755 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:36.756 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:36.757 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:36.758 DEBUG:boto:Method: GET 2022-09-13T07:36:36.760 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:36.761 DEBUG:boto:Data: 2022-09-13T07:36:36.762 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.763 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.764 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.765 DEBUG:boto:Params: {} 2022-09-13T07:36:36.767 DEBUG:boto:Token: None 2022-09-13T07:36:36.768 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/ 2022-09-13T07:36:36.769 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68= 2022-09-13T07:36:36.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68=', 'Content-Length': '0'} 2022-09-13T07:36:36.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aa9cecbf9ea84be-0063203304-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.775 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:36.777 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:36.778 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:36.779 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:36.780 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:36.781 DEBUG:boto:Method: GET 2022-09-13T07:36:36.782 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone2 2022-09-13T07:36:36.784 DEBUG:boto:Data: 2022-09-13T07:36:36.785 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.786 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.787 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.789 DEBUG:boto:Params: {} 2022-09-13T07:36:36.790 DEBUG:boto:Token: None 2022-09-13T07:36:36.791 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/ 2022-09-13T07:36:36.793 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68= 2022-09-13T07:36:36.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68=', 'Content-Length': '0'} 2022-09-13T07:36:36.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040a7cc9c8e3683a7-0063203304-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.800 DEBUG:boto:b'yxvvuh-18obj-test-zone21000falseobj-test-zone22022-09-13T07:32:07.574Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:36.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:36.802 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.803 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.804 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.805 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.806 DEBUG:boto:Method: GET 2022-09-13T07:36:36.807 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.807 DEBUG:boto:Data: 2022-09-13T07:36:36.808 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.809 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.809 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.810 DEBUG:boto:Params: {} 2022-09-13T07:36:36.811 DEBUG:boto:Token: None 2022-09-13T07:36:36.812 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.813 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8h4QvJXqzRN1vXQ1DCHZCBYraZ4= 2022-09-13T07:36:36.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8h4QvJXqzRN1vXQ1DCHZCBYraZ4=', 'Content-Length': '0'} 2022-09-13T07:36:36.818 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccabdb4e1ae9b0f9-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.819 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.820 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.821 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.821 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.822 DEBUG:boto:Method: GET 2022-09-13T07:36:36.822 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.823 DEBUG:boto:Data: 2022-09-13T07:36:36.824 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.825 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.825 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.826 DEBUG:boto:Params: {} 2022-09-13T07:36:36.827 DEBUG:boto:Token: None 2022-09-13T07:36:36.827 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.828 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8h4QvJXqzRN1vXQ1DCHZCBYraZ4= 2022-09-13T07:36:36.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8h4QvJXqzRN1vXQ1DCHZCBYraZ4=', 'Content-Length': '0'} 2022-09-13T07:36:36.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb20cda800bbcb68-0063203304-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.879 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.880 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.880 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.881 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.882 DEBUG:boto:Method: HEAD 2022-09-13T07:36:36.882 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.883 DEBUG:boto:Data: 2022-09-13T07:36:36.883 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.884 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.884 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.884 DEBUG:boto:Params: {} 2022-09-13T07:36:36.885 DEBUG:boto:Token: None 2022-09-13T07:36:36.886 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.886 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uIc7jLAKrwh3K7zU6QLVny8mK4U= 2022-09-13T07:36:36.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uIc7jLAKrwh3K7zU6QLVny8mK4U=', 'Content-Length': '0'} 2022-09-13T07:36:36.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ced51838d51e48b-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.890 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.891 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.891 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.892 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.893 DEBUG:boto:Method: HEAD 2022-09-13T07:36:36.893 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.894 DEBUG:boto:Data: 2022-09-13T07:36:36.895 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.895 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.896 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.896 DEBUG:boto:Params: {} 2022-09-13T07:36:36.897 DEBUG:boto:Token: None 2022-09-13T07:36:36.897 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.898 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uIc7jLAKrwh3K7zU6QLVny8mK4U= 2022-09-13T07:36:36.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uIc7jLAKrwh3K7zU6QLVny8mK4U=', 'Content-Length': '0'} 2022-09-13T07:36:36.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b5ba1edcc59f682-0063203304-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:36.904 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.905 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.906 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.906 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.907 DEBUG:boto:Method: GET 2022-09-13T07:36:36.908 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.908 DEBUG:boto:Data: 2022-09-13T07:36:36.910 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.911 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.912 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.912 DEBUG:boto:Params: {} 2022-09-13T07:36:36.913 DEBUG:boto:Token: None 2022-09-13T07:36:36.914 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.915 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8h4QvJXqzRN1vXQ1DCHZCBYraZ4= 2022-09-13T07:36:36.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8h4QvJXqzRN1vXQ1DCHZCBYraZ4=', 'Content-Length': '0'} 2022-09-13T07:36:36.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b5778b34a0d2c7e-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.926 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.927 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.928 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.929 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.931 DEBUG:boto:Method: GET 2022-09-13T07:36:36.932 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.933 DEBUG:boto:Data: 2022-09-13T07:36:36.934 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.935 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.936 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.936 DEBUG:boto:Params: {} 2022-09-13T07:36:36.938 DEBUG:boto:Token: None 2022-09-13T07:36:36.939 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:36.941 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8h4QvJXqzRN1vXQ1DCHZCBYraZ4= 2022-09-13T07:36:36.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8h4QvJXqzRN1vXQ1DCHZCBYraZ4=', 'Content-Length': '0'} 2022-09-13T07:36:36.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000330b3e58822806bc-0063203304-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.947 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:36.948 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:36.950 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:36.952 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:36.953 DEBUG:boto:Method: GET 2022-09-13T07:36:36.954 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:36.956 DEBUG:boto:Data: 2022-09-13T07:36:36.957 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.958 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.959 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.961 DEBUG:boto:Params: {} 2022-09-13T07:36:36.963 DEBUG:boto:Token: None 2022-09-13T07:36:36.964 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/ 2022-09-13T07:36:36.966 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68= 2022-09-13T07:36:36.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68=', 'Content-Length': '0'} 2022-09-13T07:36:36.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8d173e71ade8721-0063203304-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.973 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:36.975 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:36.976 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:36.976 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:36.977 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:36.977 DEBUG:boto:Method: GET 2022-09-13T07:36:36.978 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:36.978 DEBUG:boto:Data: 2022-09-13T07:36:36.979 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.980 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:36.980 DEBUG:boto:Port: 8000 2022-09-13T07:36:36.981 DEBUG:boto:Params: {} 2022-09-13T07:36:36.981 DEBUG:boto:Token: None 2022-09-13T07:36:36.982 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/ 2022-09-13T07:36:36.983 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68= 2022-09-13T07:36:36.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vxOJFm1cvc1x0iOppMhRpVbhQ68=', 'Content-Length': '0'} 2022-09-13T07:36:36.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb03e73b6e780ce6-0063203304-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:36.987 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:36.988 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.988 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.989 DEBUG:boto:Method: HEAD 2022-09-13T07:36:36.989 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.990 DEBUG:boto:Data: 2022-09-13T07:36:36.990 DEBUG:boto:Headers: {} 2022-09-13T07:36:36.991 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:36.991 DEBUG:boto:Port: 8001 2022-09-13T07:36:36.992 DEBUG:boto:Params: {} 2022-09-13T07:36:36.992 DEBUG:boto:Token: None 2022-09-13T07:36:36.993 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:36 GMT /yxvvuh-18/obj-test-zone3 2022-09-13T07:36:36.993 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Dzs2J6V1ZpzlUJoPAsBGD4WnVeg= 2022-09-13T07:36:36.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Dzs2J6V1ZpzlUJoPAsBGD4WnVeg=', 'Content-Length': '0'} 2022-09-13T07:36:37.000 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1a9a3d5e9e34568-0063203304-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:37.001 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:37.002 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:37.002 DEBUG:boto:Method: HEAD 2022-09-13T07:36:37.003 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3 2022-09-13T07:36:37.003 DEBUG:boto:Data: 2022-09-13T07:36:37.004 DEBUG:boto:Headers: {} 2022-09-13T07:36:37.005 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:37.006 DEBUG:boto:Port: 8000 2022-09-13T07:36:37.007 DEBUG:boto:Params: {} 2022-09-13T07:36:37.008 DEBUG:boto:Token: None 2022-09-13T07:36:37.009 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:37 GMT /yxvvuh-18/obj-test-zone3 2022-09-13T07:36:37.009 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F87t72+3JsFRvTxuEhDPFqh9xGU= 2022-09-13T07:36:37.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F87t72+3JsFRvTxuEhDPFqh9xGU=', 'Content-Length': '0'} 2022-09-13T07:36:37.013 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004636613d1f7ba13f-0063203305-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:37.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-13T07:36:37.014 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:37.015 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:37.015 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:37.016 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:37.016 DEBUG:boto:Method: GET 2022-09-13T07:36:37.017 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:37.017 DEBUG:boto:Data: 2022-09-13T07:36:37.018 DEBUG:boto:Headers: {} 2022-09-13T07:36:37.018 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:37.019 DEBUG:boto:Port: 8001 2022-09-13T07:36:37.020 DEBUG:boto:Params: {} 2022-09-13T07:36:37.020 DEBUG:boto:Token: None 2022-09-13T07:36:37.021 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:37 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:37.022 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0UWzzVTNJBnf/hNr+pIDk4ivBGo= 2022-09-13T07:36:37.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0UWzzVTNJBnf/hNr+pIDk4ivBGo=', 'Content-Length': '0'} 2022-09-13T07:36:37.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c98222bbd77dab4-0063203305-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:37.072 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:37.073 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3 2022-09-13T07:36:37.074 DEBUG:boto:path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:37.075 DEBUG:boto:auth_path=/yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:37.076 DEBUG:boto:Method: GET 2022-09-13T07:36:37.077 DEBUG:boto:Path: /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:37.077 DEBUG:boto:Data: 2022-09-13T07:36:37.078 DEBUG:boto:Headers: {} 2022-09-13T07:36:37.079 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:37.080 DEBUG:boto:Port: 8000 2022-09-13T07:36:37.081 DEBUG:boto:Params: {} 2022-09-13T07:36:37.082 DEBUG:boto:Token: None 2022-09-13T07:36:37.084 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:37 GMT /yxvvuh-18/obj-test-zone3?versionId=jGr2ahP3JH9MIikQrDceF1PJUmw6xXh 2022-09-13T07:36:37.085 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0UWzzVTNJBnf/hNr+pIDk4ivBGo= 2022-09-13T07:36:37.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0UWzzVTNJBnf/hNr+pIDk4ivBGo=', 'Content-Length': '0'} 2022-09-13T07:36:37.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:35:14 GMT'), ('x-amz-version-id', 'jGr2ahP3JH9MIikQrDceF1PJUmw6xXh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f85310c5499bba71-0063203305-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:37.093 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:37.094 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:37.094 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:37.095 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:37.096 DEBUG:boto:Method: GET 2022-09-13T07:36:37.096 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:37.097 DEBUG:boto:Data: 2022-09-13T07:36:37.099 DEBUG:boto:Headers: {} 2022-09-13T07:36:37.099 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:36:37.101 DEBUG:boto:Port: 8001 2022-09-13T07:36:37.101 DEBUG:boto:Params: {} 2022-09-13T07:36:37.102 DEBUG:boto:Token: None 2022-09-13T07:36:37.102 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:37 GMT /yxvvuh-18/ 2022-09-13T07:36:37.103 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:14XPBV1JRxNj/yrhI20kTUbUdQ8= 2022-09-13T07:36:37.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:14XPBV1JRxNj/yrhI20kTUbUdQ8=', 'Content-Length': '0'} 2022-09-13T07:36:37.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b11c102c8e765d1-0063203305-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:37.107 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:37.110 DEBUG:boto:path=/yxvvuh-18/ 2022-09-13T07:36:37.111 DEBUG:boto:auth_path=/yxvvuh-18/ 2022-09-13T07:36:37.112 DEBUG:boto:path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:37.113 DEBUG:boto:auth_path=/yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:37.114 DEBUG:boto:Method: GET 2022-09-13T07:36:37.115 DEBUG:boto:Path: /yxvvuh-18/?prefix=obj-test-zone3 2022-09-13T07:36:37.116 DEBUG:boto:Data: 2022-09-13T07:36:37.117 DEBUG:boto:Headers: {} 2022-09-13T07:36:37.118 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:36:37.119 DEBUG:boto:Port: 8000 2022-09-13T07:36:37.120 DEBUG:boto:Params: {} 2022-09-13T07:36:37.121 DEBUG:boto:Token: None 2022-09-13T07:36:37.122 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:36:37 GMT /yxvvuh-18/ 2022-09-13T07:36:37.123 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:14XPBV1JRxNj/yrhI20kTUbUdQ8= 2022-09-13T07:36:37.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:14XPBV1JRxNj/yrhI20kTUbUdQ8=', 'Content-Length': '0'} 2022-09-13T07:36:37.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ede439310d406150-0063203305-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:37.129 DEBUG:boto:b'yxvvuh-18obj-test-zone31000falseobj-test-zone32022-09-13T07:32:07.718Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:36:37.131 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-18 zones={test-zone3, test-zone2} 2022-09-13T07:36:37.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2022-09-13T07:36:37.138 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:36:37.139 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:36:37.140 DEBUG:boto:Method: PUT 2022-09-13T07:36:37.141 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:36:37.141 DEBUG:boto:Data: 2022-09-13T07:36:37.142 DEBUG:boto:Headers: {} 2022-09-13T07:36:37.143 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:37.143 DEBUG:boto:Port: 8000 2022-09-13T07:36:37.144 DEBUG:boto:Params: {} 2022-09-13T07:36:37.144 DEBUG:boto:Token: None 2022-09-13T07:36:37.145 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:36:37 GMT /yxvvuh-19/ 2022-09-13T07:36:37.146 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bJbUMGBfu2o65BaWD4NrDZzZ5oo= 2022-09-13T07:36:37.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bJbUMGBfu2o65BaWD4NrDZzZ5oo=', 'Content-Length': '0'} 2022-09-13T07:36:37.185 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001250ef5eb5e2e140-0063203305-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:37.186 DEBUG:rgw_multi.tests:created bucket=yxvvuh-19 2022-09-13T07:36:37.186 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:36:37.187 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:36:37.187 DEBUG:boto:path=/yxvvuh-19/?versioning 2022-09-13T07:36:37.188 DEBUG:boto:auth_path=/yxvvuh-19/?versioning 2022-09-13T07:36:37.188 DEBUG:boto:Method: PUT 2022-09-13T07:36:37.189 DEBUG:boto:Path: /yxvvuh-19/?versioning 2022-09-13T07:36:37.189 DEBUG:boto:Data: Enabled Disabled 2022-09-13T07:36:37.190 DEBUG:boto:Headers: {} 2022-09-13T07:36:37.190 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:37.191 DEBUG:boto:Port: 8000 2022-09-13T07:36:37.191 DEBUG:boto:Params: {} 2022-09-13T07:36:37.192 DEBUG:boto:Token: None 2022-09-13T07:36:37.193 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:36:37 GMT /yxvvuh-19/?versioning 2022-09-13T07:36:37.194 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6c7+c4UpTbk1YzLvLBnRZ2+jHB0= 2022-09-13T07:36:37.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6c7+c4UpTbk1YzLvLBnRZ2+jHB0=', 'Content-Length': '228'} 2022-09-13T07:36:37.206 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000407230d28fff1d0c-0063203305-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:37.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:36:37.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:36:37.209 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:36:37.278 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:36:37.279 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:36:37.311 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:36:37.328 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:36:37.329 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:36:37.329 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:36:37.330 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:36:37.331 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:36:37.331 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:36:37.332 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:36:37.332 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:36:37.333 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:36:37.334 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:36:37.334 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054597.202925_140.1", 2022-09-13T07:36:37.335 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:36:37.202925Z" 2022-09-13T07:36:37.335 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:36:37.337 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:36:37.337 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:36:37.337 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T07:36:37.338 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:36:37.338 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:36:37.338 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054597.202925_140.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:36:37.339 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:36:37.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:36:37.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:36:37.343 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:36:37.404 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:36:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:36:37.435 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:36:37.430+0000 7f8647b5f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:36:37.436 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:36:37.431+0000 7f8647b5f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:36:37.448 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:36:37.448 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:36:37.449 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:36:37.449 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:36:37.449 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:36:37.450 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:36:37.450 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:36:37.451 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:36:37.451 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:36:37.452 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:36:37.452 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:36:37.453 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:36:37.453 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:36:37.454 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:36:37.454 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:36:37.455 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:36:37.455 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:36:37.456 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:36:37.456 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:36:37.457 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:36:37.457 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:36:37.458 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:36:37.458 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:36:37.459 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:36:37.459 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:36:37.460 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:36:37.460 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:36:37.460 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:36:37.461 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:36:37.461 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:36:37.462 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:36:37.462 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:36:37.463 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:36:37.463 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:36:37.463 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:36:37.464 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:36:37.464 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:36:37.464 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054597.202925_140.1", 2022-09-13T07:36:37.464 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:36:37.465 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:36:37.465 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:36:37.465 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:36:37.202925Z", 2022-09-13T07:36:37.466 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:36:37.466 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:36:37.466 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:36:37.466 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:36:37.467 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:36:37.467 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:36:37.467 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:36:37.467 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:36:37.468 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:36:37.468 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:36:37.468 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:36:37.468 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:36:37.469 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:36:37.469 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:36:37.469 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:36:37.469 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:36:37.470 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:36:37.470 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:36:37.471 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:36:37.471 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:36:37.471 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:36:37.472 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:36:37Z" 2022-09-13T07:36:37.472 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:36:37.473 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054597.202925_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:36:37.202925Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:36:37Z" } 2022-09-13T07:36:37.474 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054597.202925_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:36:37.202925Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:36:37.475 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054597.202925_140.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:36:37.475 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054597.202925_140.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:36:37.476 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:36:37.476 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:36:37.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:36:37.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:36:37.478 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:36:37.533 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:36:37.534 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:36:37.561 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:37.557+0000 7f0f4f98a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:36:37.562 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:37.558+0000 7f0f4f98a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:36:37.578 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:36:37.579 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:36:37.579 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:36:37.580 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:36:37.580 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:36:37.580 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:36:37.581 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:36:37.581 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:36:37.582 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:36:37.582 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:36:37.583 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:36:37.583 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:36:37.583 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:36:37.584 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:36:37.584 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:36:37.585 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:36:37.585 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:36:37.585 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:36:37.586 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:36:37.586 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:36:37.587 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:36:37.587 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:36:37.588 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:36:37.588 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:36:37.589 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:36:37.589 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:36:37.590 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:36:37.590 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:36:37.591 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:36:37.591 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:36:37.592 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:36:37.592 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:36:37.592 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:36:37.593 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:36:37.593 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:36:37.594 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:36:37.594 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:36:37.595 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054597.202925_140.1", 2022-09-13T07:36:37.595 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:36:37.596 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:36:37.596 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:36:37.597 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:36:37.202925Z", 2022-09-13T07:36:37.597 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:36:37.598 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:36:37.598 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:36:37.599 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:36:37.599 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:36:37.599 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:36:37.600 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:36:37.600 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:36:37.601 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:36:37.601 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:36:37.601 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:36:37.602 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:36:37.602 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:36:37.603 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:36:37.603 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:36:37.604 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:36:37.604 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:36:37.605 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:36:37.605 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:36:37.606 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:36:37.606 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:36:37.607 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:36:37Z" 2022-09-13T07:36:37.607 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:36:37.609 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054597.202925_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:36:37.202925Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:36:37Z" } 2022-09-13T07:36:37.610 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054597.202925_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:36:37.202925Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:36:37.611 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054597.202925_140.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:36:37.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054597.202925_140.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:36:37.613 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:36:37.614 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:36:37.615 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:36:37.616 DEBUG:boto:Method: HEAD 2022-09-13T07:36:37.616 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:36:37.617 DEBUG:boto:Data: 2022-09-13T07:36:37.618 DEBUG:boto:Headers: {} 2022-09-13T07:36:37.619 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:37.620 DEBUG:boto:Port: 8000 2022-09-13T07:36:37.621 DEBUG:boto:Params: {} 2022-09-13T07:36:37.622 DEBUG:boto:Token: None 2022-09-13T07:36:37.623 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:36:37 GMT /yxvvuh-19/ 2022-09-13T07:36:37.624 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:I39YfpxJRO2FaldcSPcQRfU9sVo= 2022-09-13T07:36:37.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:36:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:I39YfpxJRO2FaldcSPcQRfU9sVo=', 'Content-Length': '0'} 2022-09-13T07:36:37.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f09af781597698de-0063203305-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:37.647 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:36:37.649 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:36:37.650 DEBUG:boto:Method: PUT 2022-09-13T07:36:37.651 DEBUG:boto:Path: /yxvvuh-19/dummy 2022-09-13T07:36:37.652 DEBUG:boto:Data: 2022-09-13T07:36:37.654 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T07:36:37.655 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:36:37.656 DEBUG:boto:Port: 8000 2022-09-13T07:36:37.658 DEBUG:boto:Params: {} 2022-09-13T07:36:37.659 DEBUG:boto:Token: None 2022-09-13T07:36:37.661 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 13 Sep 2022 07:36:37 GMT /yxvvuh-19/dummy 2022-09-13T07:36:37.662 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0o1owUK/5p3ljnxIS8WkVZAsOqw= 2022-09-13T07:36:37.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:36:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0o1owUK/5p3ljnxIS8WkVZAsOqw='} 2022-09-13T07:36:37.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-amz-request-id', 'tx0000065aaa012b33d6d5f-0063203305-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:36:37.679 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-19 2022-09-13T07:36:37.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:37.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:37.682 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:36:37.755 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:36:37.756 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:36:37.802 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:36:37.797+0000 7fa7023fc600 1 waiting to reach incremental sync.. 2022-09-13T07:36:42.802 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:36:42.798+0000 7fa7023fc600 1 waiting to reach incremental sync.. 2022-09-13T07:36:47.804 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:36:47.800+0000 7fa7023fc600 1 bucket sync caught up with source: 2022-09-13T07:36:47.804 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , 00000000002.6.10, , , , , ] 2022-09-13T07:36:47.805 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , 00000000002.6.10, , , , , ] 2022-09-13T07:36:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:36:47.800+0000 7fa7023fc600 0 bucket checkpoint complete 2022-09-13T07:36:47.826 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-19 2022-09-13T07:36:47.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:47.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:47.828 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:36:47.893 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:36:47.894 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:36:47.935 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:47.931+0000 7ff219de0600 1 waiting to reach incremental sync.. 2022-09-13T07:36:52.937 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:52.933+0000 7ff219de0600 1 bucket sync caught up with source: 2022-09-13T07:36:52.938 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000002.6.10, , , , , ] 2022-09-13T07:36:52.938 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000002.6.10, , , , , ] 2022-09-13T07:36:52.938 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:52.933+0000 7ff219de0600 0 bucket checkpoint complete 2022-09-13T07:36:52.961 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-19 2022-09-13T07:36:52.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:52.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:36:52.963 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:36:53.027 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:36:53.027 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:36:53.062 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:53.058+0000 7f93fdbf1600 1 waiting to reach incremental sync.. 2022-09-13T07:36:58.063 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:36:58.059+0000 7f93fdbf1600 1 waiting to reach incremental sync.. 2022-09-13T07:37:03.063 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:37:03.059+0000 7f93fdbf1600 1 waiting to reach incremental sync.. 2022-09-13T07:37:08.067 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:37:08.061+0000 7f93fdbf1600 1 bucket sync caught up with source: 2022-09-13T07:37:08.067 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000002.12.10, , , , , ] 2022-09-13T07:37:08.069 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000002.12.10, , , , , ] 2022-09-13T07:37:08.069 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:37:08.061+0000 7f93fdbf1600 0 bucket checkpoint complete 2022-09-13T07:37:08.088 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-19 2022-09-13T07:37:08.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:37:08.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:37:08.091 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:37:08.156 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:37:08.157 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:37:08.209 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:37:08.204+0000 7f62cd8d9600 1 waiting for incremental sync to catch up: 2022-09-13T07:37:08.210 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000002.12.10, , , , , ] 2022-09-13T07:37:08.210 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000003.14.9, , , , , ] 2022-09-13T07:37:13.210 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:37:13.206+0000 7f62cd8d9600 1 bucket sync caught up with source: 2022-09-13T07:37:13.210 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000003.14.9, , , , , ] 2022-09-13T07:37:13.210 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000003.14.9, , , , , ] 2022-09-13T07:37:13.211 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:37:13.206+0000 7f62cd8d9600 0 bucket checkpoint complete 2022-09-13T07:37:13.226 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-19 2022-09-13T07:37:13.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:37:13.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:37:13.228 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:37:13.293 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:37:13.293 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:37:13.329 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:37:13.325+0000 7f93b9448600 1 waiting for incremental sync to catch up: 2022-09-13T07:37:13.330 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000003.21.9, , , , , ] 2022-09-13T07:37:13.330 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2022-09-13T07:37:18.331 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:37:18.327+0000 7f93b9448600 1 waiting for incremental sync to catch up: 2022-09-13T07:37:18.331 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000003.21.9, , , , , ] 2022-09-13T07:37:18.332 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2022-09-13T07:37:23.332 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:37:23.328+0000 7f93b9448600 1 waiting for incremental sync to catch up: 2022-09-13T07:37:23.333 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000003.21.9, , , , , ] 2022-09-13T07:37:23.333 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2022-09-13T07:37:28.334 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:37:28.331+0000 7f93b9448600 1 bucket sync caught up with source: 2022-09-13T07:37:28.334 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.23.9, , , , , ] 2022-09-13T07:37:28.335 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2022-09-13T07:37:28.335 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:37:28.331+0000 7f93b9448600 0 bucket checkpoint complete 2022-09-13T07:37:28.354 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-19 2022-09-13T07:37:28.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:37:28.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:37:28.357 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:37:28.421 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:37:28.421 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:37:28.472 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:37:28.468+0000 7fe795d9a600 1 bucket sync caught up with source: 2022-09-13T07:37:28.473 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , 00000000004.23.9, , , , , ] 2022-09-13T07:37:28.473 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2022-09-13T07:37:28.473 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:37:28.468+0000 7fe795d9a600 0 bucket checkpoint complete 2022-09-13T07:37:28.496 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-19 zones={test-zone2, test-zone1} 2022-09-13T07:37:28.498 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.499 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.500 DEBUG:boto:Method: HEAD 2022-09-13T07:37:28.501 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:28.502 DEBUG:boto:Data: 2022-09-13T07:37:28.503 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.504 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:28.505 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.506 DEBUG:boto:Params: {} 2022-09-13T07:37:28.507 DEBUG:boto:Token: None 2022-09-13T07:37:28.508 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/ 2022-09-13T07:37:28.510 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uxKLnOZ+6k9NEoRhXNDbO8anS24= 2022-09-13T07:37:28.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uxKLnOZ+6k9NEoRhXNDbO8anS24=', 'Content-Length': '0'} 2022-09-13T07:37:28.514 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1b2d30d1fb41057-0063203338-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.515 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.516 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.517 DEBUG:boto:Method: HEAD 2022-09-13T07:37:28.518 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:28.519 DEBUG:boto:Data: 2022-09-13T07:37:28.520 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.521 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.522 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.523 DEBUG:boto:Params: {} 2022-09-13T07:37:28.524 DEBUG:boto:Token: None 2022-09-13T07:37:28.525 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/ 2022-09-13T07:37:28.526 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uxKLnOZ+6k9NEoRhXNDbO8anS24= 2022-09-13T07:37:28.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uxKLnOZ+6k9NEoRhXNDbO8anS24=', 'Content-Length': '0'} 2022-09-13T07:37:28.530 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8881d523e5d1ff1-0063203338-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.531 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:37:28.532 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.533 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.534 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.535 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.535 DEBUG:boto:Method: GET 2022-09-13T07:37:28.536 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.537 DEBUG:boto:Data: 2022-09-13T07:37:28.538 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.539 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:28.540 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.541 DEBUG:boto:Params: {} 2022-09-13T07:37:28.542 DEBUG:boto:Token: None 2022-09-13T07:37:28.543 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/?versions 2022-09-13T07:37:28.544 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk= 2022-09-13T07:37:28.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk=', 'Content-Length': '0'} 2022-09-13T07:37:28.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c406365f57a1ae04-0063203338-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.549 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:28.550 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:37:28.550 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:37:28.551 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.551 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.552 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.553 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.553 DEBUG:boto:Method: GET 2022-09-13T07:37:28.554 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.554 DEBUG:boto:Data: 2022-09-13T07:37:28.555 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.555 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.556 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.556 DEBUG:boto:Params: {} 2022-09-13T07:37:28.557 DEBUG:boto:Token: None 2022-09-13T07:37:28.557 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/?versions 2022-09-13T07:37:28.558 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk= 2022-09-13T07:37:28.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk=', 'Content-Length': '0'} 2022-09-13T07:37:28.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009120af86ad46f0c8-0063203338-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.562 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:28.563 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:37:28.564 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.565 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.565 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.566 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.566 DEBUG:boto:Method: GET 2022-09-13T07:37:28.567 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.567 DEBUG:boto:Data: 2022-09-13T07:37:28.568 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.569 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:28.569 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.570 DEBUG:boto:Params: {} 2022-09-13T07:37:28.570 DEBUG:boto:Token: None 2022-09-13T07:37:28.571 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/?versions 2022-09-13T07:37:28.572 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk= 2022-09-13T07:37:28.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk=', 'Content-Length': '0'} 2022-09-13T07:37:28.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004702c9e2f41ed40d-0063203338-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.575 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:28.576 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.577 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.577 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.577 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.578 DEBUG:boto:Method: GET 2022-09-13T07:37:28.578 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.579 DEBUG:boto:Data: 2022-09-13T07:37:28.579 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.580 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.580 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.581 DEBUG:boto:Params: {} 2022-09-13T07:37:28.581 DEBUG:boto:Token: None 2022-09-13T07:37:28.582 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/?versions 2022-09-13T07:37:28.583 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk= 2022-09-13T07:37:28.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk=', 'Content-Length': '0'} 2022-09-13T07:37:28.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076d5ac62b0fc3fc1-0063203338-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.587 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:28.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:37:28.588 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.589 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.590 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.590 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.591 DEBUG:boto:Method: GET 2022-09-13T07:37:28.592 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.592 DEBUG:boto:Data: 2022-09-13T07:37:28.593 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.593 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:28.594 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.594 DEBUG:boto:Params: {} 2022-09-13T07:37:28.595 DEBUG:boto:Token: None 2022-09-13T07:37:28.596 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.596 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI= 2022-09-13T07:37:28.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI=', 'Content-Length': '0'} 2022-09-13T07:37:28.601 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c5d0416abfd0d85-0063203338-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.602 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.602 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.603 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.603 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.604 DEBUG:boto:Method: GET 2022-09-13T07:37:28.604 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.605 DEBUG:boto:Data: 2022-09-13T07:37:28.605 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.606 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.607 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.607 DEBUG:boto:Params: {} 2022-09-13T07:37:28.608 DEBUG:boto:Token: None 2022-09-13T07:37:28.609 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.609 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI= 2022-09-13T07:37:28.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI=', 'Content-Length': '0'} 2022-09-13T07:37:28.612 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004dec6b2e81311d77-0063203338-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.612 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.613 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.613 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.614 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.614 DEBUG:boto:Method: HEAD 2022-09-13T07:37:28.615 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.615 DEBUG:boto:Data: 2022-09-13T07:37:28.616 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.617 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:28.617 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.618 DEBUG:boto:Params: {} 2022-09-13T07:37:28.618 DEBUG:boto:Token: None 2022-09-13T07:37:28.619 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.620 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EEc4pfFjeoMKVM22rTG9ZYkV1GA= 2022-09-13T07:37:28.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EEc4pfFjeoMKVM22rTG9ZYkV1GA=', 'Content-Length': '0'} 2022-09-13T07:37:28.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000034bc492a17de5d87-0063203338-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.623 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.623 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.624 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.624 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.625 DEBUG:boto:Method: HEAD 2022-09-13T07:37:28.625 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.626 DEBUG:boto:Data: 2022-09-13T07:37:28.626 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.627 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.627 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.628 DEBUG:boto:Params: {} 2022-09-13T07:37:28.628 DEBUG:boto:Token: None 2022-09-13T07:37:28.629 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.629 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EEc4pfFjeoMKVM22rTG9ZYkV1GA= 2022-09-13T07:37:28.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EEc4pfFjeoMKVM22rTG9ZYkV1GA=', 'Content-Length': '0'} 2022-09-13T07:37:28.632 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db12ff3f8c1eb80e-0063203338-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:37:28.634 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.635 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.636 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.637 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.637 DEBUG:boto:Method: GET 2022-09-13T07:37:28.638 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.639 DEBUG:boto:Data: 2022-09-13T07:37:28.640 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.641 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:28.641 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.642 DEBUG:boto:Params: {} 2022-09-13T07:37:28.643 DEBUG:boto:Token: None 2022-09-13T07:37:28.644 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.645 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI= 2022-09-13T07:37:28.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI=', 'Content-Length': '0'} 2022-09-13T07:37:28.648 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000055a9b2bd1985c018-0063203338-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.649 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.649 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.650 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.650 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.651 DEBUG:boto:Method: GET 2022-09-13T07:37:28.652 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.652 DEBUG:boto:Data: 2022-09-13T07:37:28.653 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.653 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.654 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.654 DEBUG:boto:Params: {} 2022-09-13T07:37:28.655 DEBUG:boto:Token: None 2022-09-13T07:37:28.656 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.656 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI= 2022-09-13T07:37:28.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI=', 'Content-Length': '0'} 2022-09-13T07:37:28.659 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d4b1138798316652-0063203338-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.660 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.660 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.661 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:28.662 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:28.662 DEBUG:boto:Method: GET 2022-09-13T07:37:28.663 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:37:28.664 DEBUG:boto:Data: 2022-09-13T07:37:28.664 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.665 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:28.665 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.666 DEBUG:boto:Params: {} 2022-09-13T07:37:28.667 DEBUG:boto:Token: None 2022-09-13T07:37:28.668 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/ 2022-09-13T07:37:28.670 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:u2J54gHviIo1RUNGAYwL/2fQFFM= 2022-09-13T07:37:28.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:u2J54gHviIo1RUNGAYwL/2fQFFM=', 'Content-Length': '0'} 2022-09-13T07:37:28.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e26e4a4d63ea9ba6-0063203338-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.714 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:28.715 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.716 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.716 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:28.717 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:28.718 DEBUG:boto:Method: GET 2022-09-13T07:37:28.718 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:37:28.719 DEBUG:boto:Data: 2022-09-13T07:37:28.719 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.720 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.721 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.721 DEBUG:boto:Params: {} 2022-09-13T07:37:28.722 DEBUG:boto:Token: None 2022-09-13T07:37:28.723 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/ 2022-09-13T07:37:28.724 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:u2J54gHviIo1RUNGAYwL/2fQFFM= 2022-09-13T07:37:28.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:u2J54gHviIo1RUNGAYwL/2fQFFM=', 'Content-Length': '0'} 2022-09-13T07:37:28.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000538952a96afffdc7-0063203338-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.728 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:28.729 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.729 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.730 DEBUG:boto:Method: HEAD 2022-09-13T07:37:28.731 DEBUG:boto:Path: /yxvvuh-19/dummy 2022-09-13T07:37:28.731 DEBUG:boto:Data: 2022-09-13T07:37:28.732 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.732 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:28.733 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.733 DEBUG:boto:Params: {} 2022-09-13T07:37:28.734 DEBUG:boto:Token: None 2022-09-13T07:37:28.734 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy 2022-09-13T07:37:28.735 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:n8uZtFHUs9EjIg34caTxn3YtVlU= 2022-09-13T07:37:28.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:n8uZtFHUs9EjIg34caTxn3YtVlU=', 'Content-Length': '0'} 2022-09-13T07:37:28.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000045f172ea459a43fa-0063203338-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.739 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.739 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.740 DEBUG:boto:Method: HEAD 2022-09-13T07:37:28.741 DEBUG:boto:Path: /yxvvuh-19/dummy 2022-09-13T07:37:28.741 DEBUG:boto:Data: 2022-09-13T07:37:28.742 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.742 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.743 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.743 DEBUG:boto:Params: {} 2022-09-13T07:37:28.744 DEBUG:boto:Token: None 2022-09-13T07:37:28.745 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy 2022-09-13T07:37:28.745 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:n8uZtFHUs9EjIg34caTxn3YtVlU= 2022-09-13T07:37:28.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:n8uZtFHUs9EjIg34caTxn3YtVlU=', 'Content-Length': '0'} 2022-09-13T07:37:28.749 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa18b5e4f03c95dd-0063203338-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:37:28.750 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.751 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.751 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.752 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.752 DEBUG:boto:Method: GET 2022-09-13T07:37:28.753 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.753 DEBUG:boto:Data: 2022-09-13T07:37:28.754 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.755 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:28.755 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.756 DEBUG:boto:Params: {} 2022-09-13T07:37:28.757 DEBUG:boto:Token: None 2022-09-13T07:37:28.757 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.758 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI= 2022-09-13T07:37:28.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI=', 'Content-Length': '0'} 2022-09-13T07:37:28.761 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c3f59f76d59d153e-0063203338-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.761 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.762 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.763 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.763 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.764 DEBUG:boto:Method: GET 2022-09-13T07:37:28.765 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.765 DEBUG:boto:Data: 2022-09-13T07:37:28.766 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.767 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.768 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.768 DEBUG:boto:Params: {} 2022-09-13T07:37:28.769 DEBUG:boto:Token: None 2022-09-13T07:37:28.770 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.770 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI= 2022-09-13T07:37:28.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI=', 'Content-Length': '0'} 2022-09-13T07:37:28.773 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c21872e71de0c35a-0063203338-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.774 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.775 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.776 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:28.777 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:28.777 DEBUG:boto:Method: GET 2022-09-13T07:37:28.778 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:37:28.778 DEBUG:boto:Data: 2022-09-13T07:37:28.779 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.780 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:28.780 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.781 DEBUG:boto:Params: {} 2022-09-13T07:37:28.781 DEBUG:boto:Token: None 2022-09-13T07:37:28.782 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/ 2022-09-13T07:37:28.783 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:u2J54gHviIo1RUNGAYwL/2fQFFM= 2022-09-13T07:37:28.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:u2J54gHviIo1RUNGAYwL/2fQFFM=', 'Content-Length': '0'} 2022-09-13T07:37:28.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044fec104ce6a662a-0063203338-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.788 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:28.789 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.790 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.791 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:28.792 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:28.793 DEBUG:boto:Method: GET 2022-09-13T07:37:28.794 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:37:28.795 DEBUG:boto:Data: 2022-09-13T07:37:28.795 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.796 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.796 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.797 DEBUG:boto:Params: {} 2022-09-13T07:37:28.797 DEBUG:boto:Token: None 2022-09-13T07:37:28.798 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/ 2022-09-13T07:37:28.799 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:u2J54gHviIo1RUNGAYwL/2fQFFM= 2022-09-13T07:37:28.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:u2J54gHviIo1RUNGAYwL/2fQFFM=', 'Content-Length': '0'} 2022-09-13T07:37:28.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cf4ee61d4c9bfbd-0063203338-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.850 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:28.851 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-19 zones={test-zone2, test-zone1} 2022-09-13T07:37:28.852 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-19 zones={test-zone3, test-zone1} 2022-09-13T07:37:28.852 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.852 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.853 DEBUG:boto:Method: HEAD 2022-09-13T07:37:28.853 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:28.854 DEBUG:boto:Data: 2022-09-13T07:37:28.855 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.855 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:28.855 DEBUG:boto:Port: 8001 2022-09-13T07:37:28.856 DEBUG:boto:Params: {} 2022-09-13T07:37:28.857 DEBUG:boto:Token: None 2022-09-13T07:37:28.857 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/ 2022-09-13T07:37:28.858 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uxKLnOZ+6k9NEoRhXNDbO8anS24= 2022-09-13T07:37:28.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uxKLnOZ+6k9NEoRhXNDbO8anS24=', 'Content-Length': '0'} 2022-09-13T07:37:28.862 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004db3e788057cdb9c-0063203338-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.864 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.864 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.865 DEBUG:boto:Method: HEAD 2022-09-13T07:37:28.865 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:28.866 DEBUG:boto:Data: 2022-09-13T07:37:28.866 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.867 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.867 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.868 DEBUG:boto:Params: {} 2022-09-13T07:37:28.868 DEBUG:boto:Token: None 2022-09-13T07:37:28.869 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/ 2022-09-13T07:37:28.869 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uxKLnOZ+6k9NEoRhXNDbO8anS24= 2022-09-13T07:37:28.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uxKLnOZ+6k9NEoRhXNDbO8anS24=', 'Content-Length': '0'} 2022-09-13T07:37:28.873 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019988555f251e8c3-0063203338-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.874 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:37:28.874 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.875 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.875 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.876 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.876 DEBUG:boto:Method: GET 2022-09-13T07:37:28.877 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.877 DEBUG:boto:Data: 2022-09-13T07:37:28.877 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.878 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:28.878 DEBUG:boto:Port: 8001 2022-09-13T07:37:28.879 DEBUG:boto:Params: {} 2022-09-13T07:37:28.879 DEBUG:boto:Token: None 2022-09-13T07:37:28.880 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/?versions 2022-09-13T07:37:28.881 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk= 2022-09-13T07:37:28.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk=', 'Content-Length': '0'} 2022-09-13T07:37:28.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c37a95743bece887-0063203338-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.885 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:28.885 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:37:28.886 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:37:28.886 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.887 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.888 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.888 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.889 DEBUG:boto:Method: GET 2022-09-13T07:37:28.889 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.890 DEBUG:boto:Data: 2022-09-13T07:37:28.890 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.891 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.891 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.892 DEBUG:boto:Params: {} 2022-09-13T07:37:28.892 DEBUG:boto:Token: None 2022-09-13T07:37:28.893 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/?versions 2022-09-13T07:37:28.893 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk= 2022-09-13T07:37:28.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk=', 'Content-Length': '0'} 2022-09-13T07:37:28.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9a7e873367f6993-0063203338-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.897 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:28.898 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:37:28.899 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.899 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.900 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.900 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.901 DEBUG:boto:Method: GET 2022-09-13T07:37:28.901 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.902 DEBUG:boto:Data: 2022-09-13T07:37:28.902 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.903 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:28.903 DEBUG:boto:Port: 8001 2022-09-13T07:37:28.904 DEBUG:boto:Params: {} 2022-09-13T07:37:28.904 DEBUG:boto:Token: None 2022-09-13T07:37:28.905 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/?versions 2022-09-13T07:37:28.906 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk= 2022-09-13T07:37:28.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk=', 'Content-Length': '0'} 2022-09-13T07:37:28.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ef46a0524e7e7e7-0063203338-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.909 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:28.910 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.911 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.911 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.912 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.912 DEBUG:boto:Method: GET 2022-09-13T07:37:28.913 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:28.913 DEBUG:boto:Data: 2022-09-13T07:37:28.914 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.914 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.915 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.915 DEBUG:boto:Params: {} 2022-09-13T07:37:28.916 DEBUG:boto:Token: None 2022-09-13T07:37:28.916 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/?versions 2022-09-13T07:37:28.917 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk= 2022-09-13T07:37:28.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lgFcXUlJrh3fmLAL8wS5/jAu5Dk=', 'Content-Length': '0'} 2022-09-13T07:37:28.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087e093de3297af16-0063203338-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.921 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:28.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:37:28.922 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.923 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.924 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.924 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.925 DEBUG:boto:Method: GET 2022-09-13T07:37:28.925 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.926 DEBUG:boto:Data: 2022-09-13T07:37:28.926 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.927 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:28.927 DEBUG:boto:Port: 8001 2022-09-13T07:37:28.928 DEBUG:boto:Params: {} 2022-09-13T07:37:28.928 DEBUG:boto:Token: None 2022-09-13T07:37:28.929 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.929 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI= 2022-09-13T07:37:28.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI=', 'Content-Length': '0'} 2022-09-13T07:37:28.932 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000058a65b7dc0e94df3-0063203338-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.933 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.933 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.934 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.934 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.935 DEBUG:boto:Method: GET 2022-09-13T07:37:28.935 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.936 DEBUG:boto:Data: 2022-09-13T07:37:28.936 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.937 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.937 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.938 DEBUG:boto:Params: {} 2022-09-13T07:37:28.938 DEBUG:boto:Token: None 2022-09-13T07:37:28.939 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.940 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI= 2022-09-13T07:37:28.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI=', 'Content-Length': '0'} 2022-09-13T07:37:28.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d2907291c8653cad-0063203338-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.943 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.943 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.944 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.945 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.945 DEBUG:boto:Method: HEAD 2022-09-13T07:37:28.946 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.947 DEBUG:boto:Data: 2022-09-13T07:37:28.948 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.949 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:28.949 DEBUG:boto:Port: 8001 2022-09-13T07:37:28.950 DEBUG:boto:Params: {} 2022-09-13T07:37:28.951 DEBUG:boto:Token: None 2022-09-13T07:37:28.952 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.952 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EEc4pfFjeoMKVM22rTG9ZYkV1GA= 2022-09-13T07:37:28.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EEc4pfFjeoMKVM22rTG9ZYkV1GA=', 'Content-Length': '0'} 2022-09-13T07:37:28.955 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001d9211fecb6f10b1-0063203338-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.956 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.956 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.957 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.957 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.958 DEBUG:boto:Method: HEAD 2022-09-13T07:37:28.958 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.959 DEBUG:boto:Data: 2022-09-13T07:37:28.959 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.960 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.960 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.961 DEBUG:boto:Params: {} 2022-09-13T07:37:28.961 DEBUG:boto:Token: None 2022-09-13T07:37:28.962 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.962 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EEc4pfFjeoMKVM22rTG9ZYkV1GA= 2022-09-13T07:37:28.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EEc4pfFjeoMKVM22rTG9ZYkV1GA=', 'Content-Length': '0'} 2022-09-13T07:37:28.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000089202bf319cd2a6-0063203338-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:37:28.966 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.966 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.967 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.967 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.968 DEBUG:boto:Method: GET 2022-09-13T07:37:28.968 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.969 DEBUG:boto:Data: 2022-09-13T07:37:28.969 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.970 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:28.970 DEBUG:boto:Port: 8001 2022-09-13T07:37:28.971 DEBUG:boto:Params: {} 2022-09-13T07:37:28.972 DEBUG:boto:Token: None 2022-09-13T07:37:28.972 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.973 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI= 2022-09-13T07:37:28.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI=', 'Content-Length': '0'} 2022-09-13T07:37:28.975 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d930f49455521fb2-0063203338-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.976 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:28.977 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:28.977 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.978 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.978 DEBUG:boto:Method: GET 2022-09-13T07:37:28.979 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.979 DEBUG:boto:Data: 2022-09-13T07:37:28.980 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.980 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:28.981 DEBUG:boto:Port: 8000 2022-09-13T07:37:28.981 DEBUG:boto:Params: {} 2022-09-13T07:37:28.982 DEBUG:boto:Token: None 2022-09-13T07:37:28.982 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:28.983 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI= 2022-09-13T07:37:28.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf73SpdTyozhlbgSkOFW7qjKGTI=', 'Content-Length': '0'} 2022-09-13T07:37:28.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004bf4e1acc445d5de-0063203338-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:28.986 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:28.987 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:28.987 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:28.988 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:28.988 DEBUG:boto:Method: GET 2022-09-13T07:37:28.989 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:37:28.989 DEBUG:boto:Data: 2022-09-13T07:37:28.990 DEBUG:boto:Headers: {} 2022-09-13T07:37:28.990 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:28.991 DEBUG:boto:Port: 8001 2022-09-13T07:37:28.991 DEBUG:boto:Params: {} 2022-09-13T07:37:28.992 DEBUG:boto:Token: None 2022-09-13T07:37:28.993 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:28 GMT /yxvvuh-19/ 2022-09-13T07:37:28.993 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:u2J54gHviIo1RUNGAYwL/2fQFFM= 2022-09-13T07:37:28.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:u2J54gHviIo1RUNGAYwL/2fQFFM=', 'Content-Length': '0'} 2022-09-13T07:37:28.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000452db92364f5baab-0063203338-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.038 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:29.039 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.040 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.040 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.041 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.041 DEBUG:boto:Method: GET 2022-09-13T07:37:29.042 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.042 DEBUG:boto:Data: 2022-09-13T07:37:29.043 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.043 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:29.044 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.044 DEBUG:boto:Params: {} 2022-09-13T07:37:29.045 DEBUG:boto:Token: None 2022-09-13T07:37:29.045 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.046 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b1c1/cBCeSpkeVSCh7uOKHmpj1M= 2022-09-13T07:37:29.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b1c1/cBCeSpkeVSCh7uOKHmpj1M=', 'Content-Length': '0'} 2022-09-13T07:37:29.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035b3f2d71b48f187-0063203339-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.051 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:29.052 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:29.053 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:29.053 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.054 DEBUG:boto:Path: /yxvvuh-19/dummy 2022-09-13T07:37:29.055 DEBUG:boto:Data: 2022-09-13T07:37:29.056 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.056 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.057 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.058 DEBUG:boto:Params: {} 2022-09-13T07:37:29.059 DEBUG:boto:Token: None 2022-09-13T07:37:29.059 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/dummy 2022-09-13T07:37:29.060 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EDRBuhl2DQpFaxXaZapft7dcqTE= 2022-09-13T07:37:29.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EDRBuhl2DQpFaxXaZapft7dcqTE=', 'Content-Length': '0'} 2022-09-13T07:37:29.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001de1f4b043876cd2-0063203339-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.064 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:29.064 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:29.065 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.065 DEBUG:boto:Path: /yxvvuh-19/dummy 2022-09-13T07:37:29.066 DEBUG:boto:Data: 2022-09-13T07:37:29.067 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.067 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:29.068 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.068 DEBUG:boto:Params: {} 2022-09-13T07:37:29.069 DEBUG:boto:Token: None 2022-09-13T07:37:29.070 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/dummy 2022-09-13T07:37:29.070 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EDRBuhl2DQpFaxXaZapft7dcqTE= 2022-09-13T07:37:29.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EDRBuhl2DQpFaxXaZapft7dcqTE=', 'Content-Length': '0'} 2022-09-13T07:37:29.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d114288bc096c36b-0063203339-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:37:29.075 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:29.076 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:29.077 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.077 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.078 DEBUG:boto:Method: GET 2022-09-13T07:37:29.079 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.079 DEBUG:boto:Data: 2022-09-13T07:37:29.080 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.080 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.081 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.082 DEBUG:boto:Params: {} 2022-09-13T07:37:29.082 DEBUG:boto:Token: None 2022-09-13T07:37:29.083 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.084 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc= 2022-09-13T07:37:29.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc=', 'Content-Length': '0'} 2022-09-13T07:37:29.087 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000484e89e99747aac2-0063203339-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.088 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:29.088 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:29.089 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.089 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.090 DEBUG:boto:Method: GET 2022-09-13T07:37:29.090 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.091 DEBUG:boto:Data: 2022-09-13T07:37:29.091 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.092 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:29.092 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.093 DEBUG:boto:Params: {} 2022-09-13T07:37:29.094 DEBUG:boto:Token: None 2022-09-13T07:37:29.095 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.095 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc= 2022-09-13T07:37:29.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc=', 'Content-Length': '0'} 2022-09-13T07:37:29.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004178398c72349037-0063203339-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.099 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.099 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.100 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.100 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.101 DEBUG:boto:Method: GET 2022-09-13T07:37:29.101 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.102 DEBUG:boto:Data: 2022-09-13T07:37:29.103 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.103 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.104 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.104 DEBUG:boto:Params: {} 2022-09-13T07:37:29.107 DEBUG:boto:Token: None 2022-09-13T07:37:29.107 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.108 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b1c1/cBCeSpkeVSCh7uOKHmpj1M= 2022-09-13T07:37:29.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b1c1/cBCeSpkeVSCh7uOKHmpj1M=', 'Content-Length': '0'} 2022-09-13T07:37:29.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b677ba7e0332182-0063203339-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.154 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:29.155 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.156 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.157 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.158 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.159 DEBUG:boto:Method: GET 2022-09-13T07:37:29.159 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.160 DEBUG:boto:Data: 2022-09-13T07:37:29.160 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.161 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:29.162 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.162 DEBUG:boto:Params: {} 2022-09-13T07:37:29.163 DEBUG:boto:Token: None 2022-09-13T07:37:29.164 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.165 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b1c1/cBCeSpkeVSCh7uOKHmpj1M= 2022-09-13T07:37:29.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b1c1/cBCeSpkeVSCh7uOKHmpj1M=', 'Content-Length': '0'} 2022-09-13T07:37:29.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e9ab6de844e1f73-0063203339-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.170 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:29.171 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-19 zones={test-zone3, test-zone1} 2022-09-13T07:37:29.171 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-19 zones={test-zone3, test-zone2} 2022-09-13T07:37:29.172 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.172 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.173 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.173 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:29.174 DEBUG:boto:Data: 2022-09-13T07:37:29.174 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.175 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.175 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.176 DEBUG:boto:Params: {} 2022-09-13T07:37:29.176 DEBUG:boto:Token: None 2022-09-13T07:37:29.177 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.178 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U= 2022-09-13T07:37:29.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U=', 'Content-Length': '0'} 2022-09-13T07:37:29.181 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000156aec45e7d075da-0063203339-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.182 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.183 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.183 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.184 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:29.184 DEBUG:boto:Data: 2022-09-13T07:37:29.185 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.185 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.186 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.187 DEBUG:boto:Params: {} 2022-09-13T07:37:29.187 DEBUG:boto:Token: None 2022-09-13T07:37:29.188 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.189 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U= 2022-09-13T07:37:29.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U=', 'Content-Length': '0'} 2022-09-13T07:37:29.192 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e7ebde18bce9ecb-0063203339-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.192 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:37:29.193 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.194 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.195 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:29.195 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:29.196 DEBUG:boto:Method: GET 2022-09-13T07:37:29.196 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:29.197 DEBUG:boto:Data: 2022-09-13T07:37:29.198 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.198 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.199 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.199 DEBUG:boto:Params: {} 2022-09-13T07:37:29.200 DEBUG:boto:Token: None 2022-09-13T07:37:29.200 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/?versions 2022-09-13T07:37:29.201 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XJBPili0iEgu1/hoTVTkbcjA1dY= 2022-09-13T07:37:29.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XJBPili0iEgu1/hoTVTkbcjA1dY=', 'Content-Length': '0'} 2022-09-13T07:37:29.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9eda6f2bb231516-0063203339-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.205 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:29.206 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:37:29.207 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:37:29.207 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.208 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.208 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:29.209 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:29.209 DEBUG:boto:Method: GET 2022-09-13T07:37:29.210 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:29.211 DEBUG:boto:Data: 2022-09-13T07:37:29.211 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.212 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.212 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.213 DEBUG:boto:Params: {} 2022-09-13T07:37:29.214 DEBUG:boto:Token: None 2022-09-13T07:37:29.214 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/?versions 2022-09-13T07:37:29.215 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XJBPili0iEgu1/hoTVTkbcjA1dY= 2022-09-13T07:37:29.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XJBPili0iEgu1/hoTVTkbcjA1dY=', 'Content-Length': '0'} 2022-09-13T07:37:29.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6b9dc75ed8ce4ba-0063203339-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.219 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:29.220 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:37:29.221 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.221 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.222 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:29.222 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:29.223 DEBUG:boto:Method: GET 2022-09-13T07:37:29.223 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:29.224 DEBUG:boto:Data: 2022-09-13T07:37:29.224 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.225 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.225 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.226 DEBUG:boto:Params: {} 2022-09-13T07:37:29.226 DEBUG:boto:Token: None 2022-09-13T07:37:29.227 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/?versions 2022-09-13T07:37:29.228 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XJBPili0iEgu1/hoTVTkbcjA1dY= 2022-09-13T07:37:29.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XJBPili0iEgu1/hoTVTkbcjA1dY=', 'Content-Length': '0'} 2022-09-13T07:37:29.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000055cbc6081341c8b-0063203339-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.232 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:29.233 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.233 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.234 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:29.234 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:29.235 DEBUG:boto:Method: GET 2022-09-13T07:37:29.235 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:37:29.236 DEBUG:boto:Data: 2022-09-13T07:37:29.236 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.237 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.238 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.239 DEBUG:boto:Params: {} 2022-09-13T07:37:29.240 DEBUG:boto:Token: None 2022-09-13T07:37:29.241 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/?versions 2022-09-13T07:37:29.243 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XJBPili0iEgu1/hoTVTkbcjA1dY= 2022-09-13T07:37:29.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XJBPili0iEgu1/hoTVTkbcjA1dY=', 'Content-Length': '0'} 2022-09-13T07:37:29.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf178805676b8700-0063203339-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.290 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:29.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:37:29.292 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:29.293 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:29.294 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.294 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.295 DEBUG:boto:Method: GET 2022-09-13T07:37:29.295 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.296 DEBUG:boto:Data: 2022-09-13T07:37:29.296 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.297 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.297 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.298 DEBUG:boto:Params: {} 2022-09-13T07:37:29.298 DEBUG:boto:Token: None 2022-09-13T07:37:29.299 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.300 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc= 2022-09-13T07:37:29.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc=', 'Content-Length': '0'} 2022-09-13T07:37:29.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f08f168ec20e19c-0063203339-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.304 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:29.305 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:29.305 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.306 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.306 DEBUG:boto:Method: GET 2022-09-13T07:37:29.307 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.307 DEBUG:boto:Data: 2022-09-13T07:37:29.308 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.308 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.309 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.309 DEBUG:boto:Params: {} 2022-09-13T07:37:29.310 DEBUG:boto:Token: None 2022-09-13T07:37:29.311 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.311 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc= 2022-09-13T07:37:29.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc=', 'Content-Length': '0'} 2022-09-13T07:37:29.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f07f80d043c9b698-0063203339-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.315 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:29.315 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:29.316 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.316 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.317 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.317 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.318 DEBUG:boto:Data: 2022-09-13T07:37:29.318 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.319 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.319 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.320 DEBUG:boto:Params: {} 2022-09-13T07:37:29.320 DEBUG:boto:Token: None 2022-09-13T07:37:29.321 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.322 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lIbBuma5GzPG3UlJv7fAvQvSGq4= 2022-09-13T07:37:29.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lIbBuma5GzPG3UlJv7fAvQvSGq4=', 'Content-Length': '0'} 2022-09-13T07:37:29.325 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000034580ba02f6a8c5-0063203339-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.325 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:29.326 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:29.326 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.327 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.328 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.328 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.329 DEBUG:boto:Data: 2022-09-13T07:37:29.330 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.331 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.331 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.332 DEBUG:boto:Params: {} 2022-09-13T07:37:29.332 DEBUG:boto:Token: None 2022-09-13T07:37:29.333 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.334 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lIbBuma5GzPG3UlJv7fAvQvSGq4= 2022-09-13T07:37:29.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lIbBuma5GzPG3UlJv7fAvQvSGq4=', 'Content-Length': '0'} 2022-09-13T07:37:29.336 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ae523d58f92d06b9-0063203339-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:37:29.338 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:29.339 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:29.340 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.340 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.341 DEBUG:boto:Method: GET 2022-09-13T07:37:29.341 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.342 DEBUG:boto:Data: 2022-09-13T07:37:29.343 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.343 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.344 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.345 DEBUG:boto:Params: {} 2022-09-13T07:37:29.345 DEBUG:boto:Token: None 2022-09-13T07:37:29.346 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.347 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc= 2022-09-13T07:37:29.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc=', 'Content-Length': '0'} 2022-09-13T07:37:29.351 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000015952b8592643ce0-0063203339-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.351 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:29.352 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:29.353 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.353 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.354 DEBUG:boto:Method: GET 2022-09-13T07:37:29.354 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.355 DEBUG:boto:Data: 2022-09-13T07:37:29.355 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.356 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.356 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.357 DEBUG:boto:Params: {} 2022-09-13T07:37:29.358 DEBUG:boto:Token: None 2022-09-13T07:37:29.359 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.359 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc= 2022-09-13T07:37:29.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc=', 'Content-Length': '0'} 2022-09-13T07:37:29.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000390d031e265c4022-0063203339-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.363 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.363 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.364 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.365 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.365 DEBUG:boto:Method: GET 2022-09-13T07:37:29.366 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.366 DEBUG:boto:Data: 2022-09-13T07:37:29.367 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.367 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.368 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.368 DEBUG:boto:Params: {} 2022-09-13T07:37:29.369 DEBUG:boto:Token: None 2022-09-13T07:37:29.370 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.370 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b1c1/cBCeSpkeVSCh7uOKHmpj1M= 2022-09-13T07:37:29.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b1c1/cBCeSpkeVSCh7uOKHmpj1M=', 'Content-Length': '0'} 2022-09-13T07:37:29.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006901a3423105ab07-0063203339-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.414 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:29.415 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.416 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.416 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.417 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.417 DEBUG:boto:Method: GET 2022-09-13T07:37:29.418 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.419 DEBUG:boto:Data: 2022-09-13T07:37:29.419 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.420 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.420 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.421 DEBUG:boto:Params: {} 2022-09-13T07:37:29.422 DEBUG:boto:Token: None 2022-09-13T07:37:29.422 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.423 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b1c1/cBCeSpkeVSCh7uOKHmpj1M= 2022-09-13T07:37:29.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b1c1/cBCeSpkeVSCh7uOKHmpj1M=', 'Content-Length': '0'} 2022-09-13T07:37:29.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be4902bf5d859a3a-0063203339-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.429 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:29.430 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:29.431 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:29.432 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.433 DEBUG:boto:Path: /yxvvuh-19/dummy 2022-09-13T07:37:29.434 DEBUG:boto:Data: 2022-09-13T07:37:29.435 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.436 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.436 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.437 DEBUG:boto:Params: {} 2022-09-13T07:37:29.438 DEBUG:boto:Token: None 2022-09-13T07:37:29.439 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/dummy 2022-09-13T07:37:29.440 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EDRBuhl2DQpFaxXaZapft7dcqTE= 2022-09-13T07:37:29.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EDRBuhl2DQpFaxXaZapft7dcqTE=', 'Content-Length': '0'} 2022-09-13T07:37:29.444 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006842219e7a538680-0063203339-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.444 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:29.445 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:29.445 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.446 DEBUG:boto:Path: /yxvvuh-19/dummy 2022-09-13T07:37:29.447 DEBUG:boto:Data: 2022-09-13T07:37:29.447 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.448 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.448 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.449 DEBUG:boto:Params: {} 2022-09-13T07:37:29.449 DEBUG:boto:Token: None 2022-09-13T07:37:29.450 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/dummy 2022-09-13T07:37:29.451 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EDRBuhl2DQpFaxXaZapft7dcqTE= 2022-09-13T07:37:29.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EDRBuhl2DQpFaxXaZapft7dcqTE=', 'Content-Length': '0'} 2022-09-13T07:37:29.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006f07d7929b9601fa-0063203339-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:37:29.457 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:29.458 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:29.459 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.460 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.461 DEBUG:boto:Method: GET 2022-09-13T07:37:29.461 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.462 DEBUG:boto:Data: 2022-09-13T07:37:29.463 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.463 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.464 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.464 DEBUG:boto:Params: {} 2022-09-13T07:37:29.465 DEBUG:boto:Token: None 2022-09-13T07:37:29.465 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.466 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc= 2022-09-13T07:37:29.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc=', 'Content-Length': '0'} 2022-09-13T07:37:29.469 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000863150b1068e3b5a-0063203339-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.470 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:37:29.471 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:37:29.471 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.472 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.472 DEBUG:boto:Method: GET 2022-09-13T07:37:29.473 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.474 DEBUG:boto:Data: 2022-09-13T07:37:29.474 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.475 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.475 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.476 DEBUG:boto:Params: {} 2022-09-13T07:37:29.476 DEBUG:boto:Token: None 2022-09-13T07:37:29.477 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:37:29.477 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc= 2022-09-13T07:37:29.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:V/EdwDGAWaiE5F/RZXNEH597rEc=', 'Content-Length': '0'} 2022-09-13T07:37:29.480 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005aeae5a77b6c7cc5-0063203339-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.481 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.482 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.482 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.483 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.484 DEBUG:boto:Method: GET 2022-09-13T07:37:29.485 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.486 DEBUG:boto:Data: 2022-09-13T07:37:29.487 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.488 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.489 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.490 DEBUG:boto:Params: {} 2022-09-13T07:37:29.491 DEBUG:boto:Token: None 2022-09-13T07:37:29.492 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.493 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b1c1/cBCeSpkeVSCh7uOKHmpj1M= 2022-09-13T07:37:29.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b1c1/cBCeSpkeVSCh7uOKHmpj1M=', 'Content-Length': '0'} 2022-09-13T07:37:29.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005afd75d300906beb-0063203339-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.498 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:29.499 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.500 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.501 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.501 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.502 DEBUG:boto:Method: GET 2022-09-13T07:37:29.503 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:37:29.503 DEBUG:boto:Data: 2022-09-13T07:37:29.504 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.504 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.505 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.505 DEBUG:boto:Params: {} 2022-09-13T07:37:29.506 DEBUG:boto:Token: None 2022-09-13T07:37:29.507 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.507 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b1c1/cBCeSpkeVSCh7uOKHmpj1M= 2022-09-13T07:37:29.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b1c1/cBCeSpkeVSCh7uOKHmpj1M=', 'Content-Length': '0'} 2022-09-13T07:37:29.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000787479fc6c193c49-0063203339-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.554 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:37:29.555 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-19 zones={test-zone3, test-zone2} 2022-09-13T07:37:29.556 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.556 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.557 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.558 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:29.558 DEBUG:boto:Data: 2022-09-13T07:37:29.559 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.559 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:29.560 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.560 DEBUG:boto:Params: {} 2022-09-13T07:37:29.561 DEBUG:boto:Token: None 2022-09-13T07:37:29.561 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.562 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U= 2022-09-13T07:37:29.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U=', 'Content-Length': '0'} 2022-09-13T07:37:29.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000feb4c3b7206b50f1-0063203339-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.567 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:29.567 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:29.568 DEBUG:boto:Method: PUT 2022-09-13T07:37:29.569 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:29.569 DEBUG:boto:Data: 2022-09-13T07:37:29.570 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:29.570 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:29.571 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.571 DEBUG:boto:Params: {} 2022-09-13T07:37:29.572 DEBUG:boto:Token: None 2022-09-13T07:37:29.572 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/obj 2022-09-13T07:37:29.573 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA= 2022-09-13T07:37:29.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA='} 2022-09-13T07:37:29.587 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-amz-request-id', 'tx0000038e028bba5a4b7fb-0063203339-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.588 DEBUG:rgw_multi.tests:zone=test-zone1 version=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:37:29.589 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.590 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.591 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.591 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:29.592 DEBUG:boto:Data: 2022-09-13T07:37:29.592 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.593 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.593 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.594 DEBUG:boto:Params: {} 2022-09-13T07:37:29.595 DEBUG:boto:Token: None 2022-09-13T07:37:29.595 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.596 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U= 2022-09-13T07:37:29.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U=', 'Content-Length': '0'} 2022-09-13T07:37:29.599 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009821ef1d572d1b5-0063203339-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.600 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:29.600 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:29.601 DEBUG:boto:Method: PUT 2022-09-13T07:37:29.601 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:29.602 DEBUG:boto:Data: 2022-09-13T07:37:29.602 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:29.603 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.603 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.604 DEBUG:boto:Params: {} 2022-09-13T07:37:29.605 DEBUG:boto:Token: None 2022-09-13T07:37:29.605 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/obj 2022-09-13T07:37:29.606 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA= 2022-09-13T07:37:29.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA='} 2022-09-13T07:37:29.625 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-amz-request-id', 'tx000008b576a8f02ce54ce-0063203339-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.627 DEBUG:rgw_multi.tests:zone=test-zone2 version=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:37:29.628 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.628 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.629 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.630 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:29.631 DEBUG:boto:Data: 2022-09-13T07:37:29.632 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.632 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.632 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.633 DEBUG:boto:Params: {} 2022-09-13T07:37:29.633 DEBUG:boto:Token: None 2022-09-13T07:37:29.634 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.635 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U= 2022-09-13T07:37:29.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U=', 'Content-Length': '0'} 2022-09-13T07:37:29.638 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041ac841fc37b9990-0063203339-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.639 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:29.639 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:29.640 DEBUG:boto:Method: PUT 2022-09-13T07:37:29.640 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:29.641 DEBUG:boto:Data: 2022-09-13T07:37:29.641 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:29.642 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.643 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.643 DEBUG:boto:Params: {} 2022-09-13T07:37:29.644 DEBUG:boto:Token: None 2022-09-13T07:37:29.645 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/obj 2022-09-13T07:37:29.645 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA= 2022-09-13T07:37:29.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA='} 2022-09-13T07:37:29.659 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-amz-request-id', 'tx000004dc4d7933537dfdb-0063203339-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.660 DEBUG:rgw_multi.tests:zone=test-zone3 version=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:37:29.661 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.662 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.663 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.664 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:29.665 DEBUG:boto:Data: 2022-09-13T07:37:29.666 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.667 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:29.668 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.669 DEBUG:boto:Params: {} 2022-09-13T07:37:29.670 DEBUG:boto:Token: None 2022-09-13T07:37:29.671 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.672 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U= 2022-09-13T07:37:29.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U=', 'Content-Length': '0'} 2022-09-13T07:37:29.677 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2959b7b2dd2ad84-0063203339-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.678 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:29.679 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:29.680 DEBUG:boto:Method: PUT 2022-09-13T07:37:29.681 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:29.682 DEBUG:boto:Data: 2022-09-13T07:37:29.683 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:29.685 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:29.686 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.687 DEBUG:boto:Params: {} 2022-09-13T07:37:29.687 DEBUG:boto:Token: None 2022-09-13T07:37:29.688 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/obj 2022-09-13T07:37:29.688 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA= 2022-09-13T07:37:29.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA='} 2022-09-13T07:37:29.704 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-amz-request-id', 'tx0000064d6eca8a13c51e2-0063203339-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.705 DEBUG:rgw_multi.tests:zone=test-zone1 version=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:37:29.706 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.707 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.707 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.709 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:29.710 DEBUG:boto:Data: 2022-09-13T07:37:29.710 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.711 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.712 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.713 DEBUG:boto:Params: {} 2022-09-13T07:37:29.714 DEBUG:boto:Token: None 2022-09-13T07:37:29.715 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.715 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U= 2022-09-13T07:37:29.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U=', 'Content-Length': '0'} 2022-09-13T07:37:29.718 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2580d627c4ff9af-0063203339-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.719 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:29.720 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:29.720 DEBUG:boto:Method: PUT 2022-09-13T07:37:29.721 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:29.722 DEBUG:boto:Data: 2022-09-13T07:37:29.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:29.723 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.724 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.725 DEBUG:boto:Params: {} 2022-09-13T07:37:29.725 DEBUG:boto:Token: None 2022-09-13T07:37:29.726 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/obj 2022-09-13T07:37:29.727 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA= 2022-09-13T07:37:29.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA='} 2022-09-13T07:37:29.743 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-amz-request-id', 'tx00000b4ed9e9664d9a1c7-0063203339-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.744 DEBUG:rgw_multi.tests:zone=test-zone2 version=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:37:29.745 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.746 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.747 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.748 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:29.749 DEBUG:boto:Data: 2022-09-13T07:37:29.750 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.750 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.751 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.752 DEBUG:boto:Params: {} 2022-09-13T07:37:29.753 DEBUG:boto:Token: None 2022-09-13T07:37:29.754 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.755 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U= 2022-09-13T07:37:29.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U=', 'Content-Length': '0'} 2022-09-13T07:37:29.759 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0bdf646abf1f5f2-0063203339-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.760 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:29.761 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:29.762 DEBUG:boto:Method: PUT 2022-09-13T07:37:29.762 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:29.763 DEBUG:boto:Data: 2022-09-13T07:37:29.763 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:29.764 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.765 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.766 DEBUG:boto:Params: {} 2022-09-13T07:37:29.767 DEBUG:boto:Token: None 2022-09-13T07:37:29.768 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/obj 2022-09-13T07:37:29.769 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA= 2022-09-13T07:37:29.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA='} 2022-09-13T07:37:29.791 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-amz-request-id', 'tx00000b0fef2361fa66859-0063203339-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.792 DEBUG:rgw_multi.tests:zone=test-zone3 version=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:37:29.792 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.793 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.793 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.794 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:29.794 DEBUG:boto:Data: 2022-09-13T07:37:29.795 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.795 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:29.796 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.797 DEBUG:boto:Params: {} 2022-09-13T07:37:29.798 DEBUG:boto:Token: None 2022-09-13T07:37:29.799 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.800 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U= 2022-09-13T07:37:29.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U=', 'Content-Length': '0'} 2022-09-13T07:37:29.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b23d5b6ceb8d9cc-0063203339-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.805 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:29.805 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:29.806 DEBUG:boto:Method: PUT 2022-09-13T07:37:29.807 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:29.808 DEBUG:boto:Data: 2022-09-13T07:37:29.809 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:29.810 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:29.811 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.811 DEBUG:boto:Params: {} 2022-09-13T07:37:29.812 DEBUG:boto:Token: None 2022-09-13T07:37:29.813 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/obj 2022-09-13T07:37:29.814 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA= 2022-09-13T07:37:29.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA='} 2022-09-13T07:37:29.825 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-amz-request-id', 'tx0000048de4352ccb3feaf-0063203339-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.826 DEBUG:rgw_multi.tests:zone=test-zone1 version=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:37:29.826 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.827 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.827 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.828 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:29.828 DEBUG:boto:Data: 2022-09-13T07:37:29.829 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.829 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.830 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.830 DEBUG:boto:Params: {} 2022-09-13T07:37:29.831 DEBUG:boto:Token: None 2022-09-13T07:37:29.831 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.832 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U= 2022-09-13T07:37:29.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U=', 'Content-Length': '0'} 2022-09-13T07:37:29.835 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003823e16716b799a7-0063203339-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.836 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:29.836 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:29.837 DEBUG:boto:Method: PUT 2022-09-13T07:37:29.837 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:29.838 DEBUG:boto:Data: 2022-09-13T07:37:29.838 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:29.839 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.839 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.840 DEBUG:boto:Params: {} 2022-09-13T07:37:29.840 DEBUG:boto:Token: None 2022-09-13T07:37:29.841 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/obj 2022-09-13T07:37:29.841 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA= 2022-09-13T07:37:29.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA='} 2022-09-13T07:37:29.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-amz-request-id', 'tx00000cf379555577f5041-0063203339-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.887 DEBUG:rgw_multi.tests:zone=test-zone2 version=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:37:29.888 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.888 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.889 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.889 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:29.890 DEBUG:boto:Data: 2022-09-13T07:37:29.890 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.891 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.891 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.893 DEBUG:boto:Params: {} 2022-09-13T07:37:29.893 DEBUG:boto:Token: None 2022-09-13T07:37:29.895 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.896 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U= 2022-09-13T07:37:29.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U=', 'Content-Length': '0'} 2022-09-13T07:37:29.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007add4496be10deeb-0063203339-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.901 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:29.901 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:29.902 DEBUG:boto:Method: PUT 2022-09-13T07:37:29.903 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:29.904 DEBUG:boto:Data: 2022-09-13T07:37:29.905 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:29.906 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:29.906 DEBUG:boto:Port: 8001 2022-09-13T07:37:29.907 DEBUG:boto:Params: {} 2022-09-13T07:37:29.907 DEBUG:boto:Token: None 2022-09-13T07:37:29.908 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/obj 2022-09-13T07:37:29.908 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA= 2022-09-13T07:37:29.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA='} 2022-09-13T07:37:29.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-amz-request-id', 'tx000007b0adca6a00f03ba-0063203339-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.955 DEBUG:rgw_multi.tests:zone=test-zone3 version=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:37:29.955 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.956 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.956 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.957 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:29.957 DEBUG:boto:Data: 2022-09-13T07:37:29.958 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.958 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:29.959 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.959 DEBUG:boto:Params: {} 2022-09-13T07:37:29.960 DEBUG:boto:Token: None 2022-09-13T07:37:29.960 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.961 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U= 2022-09-13T07:37:29.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U=', 'Content-Length': '0'} 2022-09-13T07:37:29.965 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009af5514a53850986-0063203339-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.966 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:29.966 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:29.967 DEBUG:boto:Method: PUT 2022-09-13T07:37:29.967 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:29.968 DEBUG:boto:Data: 2022-09-13T07:37:29.968 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:29.969 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:29.969 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.969 DEBUG:boto:Params: {} 2022-09-13T07:37:29.970 DEBUG:boto:Token: None 2022-09-13T07:37:29.971 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/obj 2022-09-13T07:37:29.971 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA= 2022-09-13T07:37:29.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Gw+fOwxNlh8uMnT7Dr5HaEJe4xA='} 2022-09-13T07:37:29.983 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-amz-request-id', 'tx00000c1a34ec70a87f1a2-0063203339-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.984 DEBUG:rgw_multi.tests:zone=test-zone1 version=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:37:29.985 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:29.985 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:29.986 DEBUG:boto:Method: HEAD 2022-09-13T07:37:29.986 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:29.987 DEBUG:boto:Data: 2022-09-13T07:37:29.987 DEBUG:boto:Headers: {} 2022-09-13T07:37:29.988 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.988 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.989 DEBUG:boto:Params: {} 2022-09-13T07:37:29.989 DEBUG:boto:Token: None 2022-09-13T07:37:29.990 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:29 GMT /yxvvuh-19/ 2022-09-13T07:37:29.990 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U= 2022-09-13T07:37:29.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XeWNPPjO26eRbLxcW1yu25RhU5U=', 'Content-Length': '0'} 2022-09-13T07:37:29.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005383c44c918c04d-0063203339-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:29.994 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:29.995 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:29.996 DEBUG:boto:Method: PUT 2022-09-13T07:37:29.996 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:29.997 DEBUG:boto:Data: 2022-09-13T07:37:29.997 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:29.998 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:29.998 DEBUG:boto:Port: 8000 2022-09-13T07:37:29.999 DEBUG:boto:Params: {} 2022-09-13T07:37:29.999 DEBUG:boto:Token: None 2022-09-13T07:37:30.000 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.000 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-amz-request-id', 'tx000002e8fe35696edad8f-0063203339-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.013 DEBUG:rgw_multi.tests:zone=test-zone2 version=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:37:30.013 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.014 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.014 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.015 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.015 DEBUG:boto:Data: 2022-09-13T07:37:30.016 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.016 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:30.017 DEBUG:boto:Port: 8001 2022-09-13T07:37:30.017 DEBUG:boto:Params: {} 2022-09-13T07:37:30.018 DEBUG:boto:Token: None 2022-09-13T07:37:30.018 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.019 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b682ed94c16eb478-006320333a-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.022 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.023 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.023 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.024 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.024 DEBUG:boto:Data: 2022-09-13T07:37:30.025 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.025 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:30.026 DEBUG:boto:Port: 8001 2022-09-13T07:37:30.026 DEBUG:boto:Params: {} 2022-09-13T07:37:30.027 DEBUG:boto:Token: None 2022-09-13T07:37:30.027 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.028 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.040 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-amz-request-id', 'tx00000c8376027b6395fc6-006320333a-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.041 DEBUG:rgw_multi.tests:zone=test-zone3 version=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:37:30.041 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.042 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.042 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.043 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.043 DEBUG:boto:Data: 2022-09-13T07:37:30.043 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.044 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:30.045 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.046 DEBUG:boto:Params: {} 2022-09-13T07:37:30.047 DEBUG:boto:Token: None 2022-09-13T07:37:30.048 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.049 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.053 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d80c91e7e24f5329-006320333a-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.053 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.054 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.054 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.055 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.055 DEBUG:boto:Data: 2022-09-13T07:37:30.056 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.056 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:30.057 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.057 DEBUG:boto:Params: {} 2022-09-13T07:37:30.058 DEBUG:boto:Token: None 2022-09-13T07:37:30.058 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.059 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-amz-request-id', 'tx00000d0adf73157cb2f59-006320333a-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.071 DEBUG:rgw_multi.tests:zone=test-zone1 version=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:37:30.071 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.072 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.072 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.073 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.073 DEBUG:boto:Data: 2022-09-13T07:37:30.074 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.074 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:30.075 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.075 DEBUG:boto:Params: {} 2022-09-13T07:37:30.076 DEBUG:boto:Token: None 2022-09-13T07:37:30.076 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.077 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.080 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000447492ea70e1c419-006320333a-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.081 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.082 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.083 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.083 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.084 DEBUG:boto:Data: 2022-09-13T07:37:30.085 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.086 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:30.086 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.087 DEBUG:boto:Params: {} 2022-09-13T07:37:30.088 DEBUG:boto:Token: None 2022-09-13T07:37:30.089 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.090 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-amz-request-id', 'tx00000fd3b0a846976adaf-006320333a-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.103 DEBUG:rgw_multi.tests:zone=test-zone2 version=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:37:30.104 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.104 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.105 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.105 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.106 DEBUG:boto:Data: 2022-09-13T07:37:30.107 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.108 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:30.108 DEBUG:boto:Port: 8001 2022-09-13T07:37:30.109 DEBUG:boto:Params: {} 2022-09-13T07:37:30.110 DEBUG:boto:Token: None 2022-09-13T07:37:30.110 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.111 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.114 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d137b833fd61421f-006320333a-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.114 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.115 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.115 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.116 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.116 DEBUG:boto:Data: 2022-09-13T07:37:30.117 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.117 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:30.118 DEBUG:boto:Port: 8001 2022-09-13T07:37:30.118 DEBUG:boto:Params: {} 2022-09-13T07:37:30.119 DEBUG:boto:Token: None 2022-09-13T07:37:30.120 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.120 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.135 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-amz-request-id', 'tx0000010d8057ee941e6a3-006320333a-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.136 DEBUG:rgw_multi.tests:zone=test-zone3 version=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:37:30.136 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.137 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.137 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.138 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.138 DEBUG:boto:Data: 2022-09-13T07:37:30.139 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.139 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:30.140 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.140 DEBUG:boto:Params: {} 2022-09-13T07:37:30.141 DEBUG:boto:Token: None 2022-09-13T07:37:30.142 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.143 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.146 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000902ded202da04aae-006320333a-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.147 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.147 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.147 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.148 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.148 DEBUG:boto:Data: 2022-09-13T07:37:30.149 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.149 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:30.150 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.150 DEBUG:boto:Params: {} 2022-09-13T07:37:30.151 DEBUG:boto:Token: None 2022-09-13T07:37:30.151 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.152 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-amz-request-id', 'tx00000e5b0d0875fd519c2-006320333a-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.171 DEBUG:rgw_multi.tests:zone=test-zone1 version=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:37:30.171 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.172 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.172 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.173 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.173 DEBUG:boto:Data: 2022-09-13T07:37:30.174 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.174 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:30.175 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.175 DEBUG:boto:Params: {} 2022-09-13T07:37:30.176 DEBUG:boto:Token: None 2022-09-13T07:37:30.176 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.177 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.180 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfca2317806690eb-006320333a-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.181 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.181 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.182 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.182 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.183 DEBUG:boto:Data: 2022-09-13T07:37:30.183 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.184 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:30.185 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.185 DEBUG:boto:Params: {} 2022-09-13T07:37:30.186 DEBUG:boto:Token: None 2022-09-13T07:37:30.186 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.187 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-amz-request-id', 'tx00000ad4a037cf730b812-006320333a-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.199 DEBUG:rgw_multi.tests:zone=test-zone2 version=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:37:30.199 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.200 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.200 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.201 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.201 DEBUG:boto:Data: 2022-09-13T07:37:30.202 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.202 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:30.203 DEBUG:boto:Port: 8001 2022-09-13T07:37:30.203 DEBUG:boto:Params: {} 2022-09-13T07:37:30.204 DEBUG:boto:Token: None 2022-09-13T07:37:30.204 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.205 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.208 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008225b7ec6c073f41-006320333a-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.209 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.209 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.210 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.211 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.212 DEBUG:boto:Data: 2022-09-13T07:37:30.212 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.213 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:30.214 DEBUG:boto:Port: 8001 2022-09-13T07:37:30.215 DEBUG:boto:Params: {} 2022-09-13T07:37:30.216 DEBUG:boto:Token: None 2022-09-13T07:37:30.217 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.218 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.233 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-amz-request-id', 'tx000002c4fbe491d6d0cd9-006320333a-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.234 DEBUG:rgw_multi.tests:zone=test-zone3 version=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:37:30.235 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.236 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.237 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.237 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.238 DEBUG:boto:Data: 2022-09-13T07:37:30.239 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.240 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:30.241 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.242 DEBUG:boto:Params: {} 2022-09-13T07:37:30.243 DEBUG:boto:Token: None 2022-09-13T07:37:30.244 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.244 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.249 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008acd88214c149674-006320333a-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.250 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.251 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.252 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.252 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.253 DEBUG:boto:Data: 2022-09-13T07:37:30.254 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.255 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:30.256 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.257 DEBUG:boto:Params: {} 2022-09-13T07:37:30.258 DEBUG:boto:Token: None 2022-09-13T07:37:30.259 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.260 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.279 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-amz-request-id', 'tx00000e906d70d5c09d390-006320333a-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.280 DEBUG:rgw_multi.tests:zone=test-zone1 version=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:37:30.281 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.282 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.283 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.284 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.284 DEBUG:boto:Data: 2022-09-13T07:37:30.285 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.286 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:30.287 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.288 DEBUG:boto:Params: {} 2022-09-13T07:37:30.289 DEBUG:boto:Token: None 2022-09-13T07:37:30.290 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.291 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efc2f4bd919b7f58-006320333a-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.296 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.297 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.298 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.299 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.300 DEBUG:boto:Data: 2022-09-13T07:37:30.301 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.302 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:30.303 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.303 DEBUG:boto:Params: {} 2022-09-13T07:37:30.305 DEBUG:boto:Token: None 2022-09-13T07:37:30.306 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.307 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-amz-request-id', 'tx0000014cdfddb6c730af0-006320333a-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.324 DEBUG:rgw_multi.tests:zone=test-zone2 version=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:37:30.324 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.325 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.325 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.326 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.326 DEBUG:boto:Data: 2022-09-13T07:37:30.327 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.327 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:30.328 DEBUG:boto:Port: 8001 2022-09-13T07:37:30.328 DEBUG:boto:Params: {} 2022-09-13T07:37:30.329 DEBUG:boto:Token: None 2022-09-13T07:37:30.329 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.330 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.333 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000225e48032e1b006a-006320333a-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.334 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.334 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.335 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.336 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.336 DEBUG:boto:Data: 2022-09-13T07:37:30.337 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.337 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:30.338 DEBUG:boto:Port: 8001 2022-09-13T07:37:30.338 DEBUG:boto:Params: {} 2022-09-13T07:37:30.339 DEBUG:boto:Token: None 2022-09-13T07:37:30.340 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.340 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-amz-request-id', 'tx0000037b335484e123ce3-006320333a-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.357 DEBUG:rgw_multi.tests:zone=test-zone3 version=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:37:30.358 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.359 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.360 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.361 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.362 DEBUG:boto:Data: 2022-09-13T07:37:30.363 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.364 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:30.364 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.365 DEBUG:boto:Params: {} 2022-09-13T07:37:30.366 DEBUG:boto:Token: None 2022-09-13T07:37:30.367 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.368 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090eac9878900f7db-006320333a-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.374 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.375 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.376 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.377 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.378 DEBUG:boto:Data: 2022-09-13T07:37:30.378 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.379 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:30.380 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.381 DEBUG:boto:Params: {} 2022-09-13T07:37:30.382 DEBUG:boto:Token: None 2022-09-13T07:37:30.383 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.384 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.397 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-amz-request-id', 'tx000008601ea56377a13db-006320333a-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.399 DEBUG:rgw_multi.tests:zone=test-zone1 version=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:37:30.399 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.400 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.401 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.402 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.403 DEBUG:boto:Data: 2022-09-13T07:37:30.404 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.404 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:30.405 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.405 DEBUG:boto:Params: {} 2022-09-13T07:37:30.406 DEBUG:boto:Token: None 2022-09-13T07:37:30.407 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.408 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018076a17bd6f4d3c-006320333a-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.413 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.414 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.415 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.416 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.416 DEBUG:boto:Data: 2022-09-13T07:37:30.417 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.418 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:30.419 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.420 DEBUG:boto:Params: {} 2022-09-13T07:37:30.421 DEBUG:boto:Token: None 2022-09-13T07:37:30.422 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.423 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.437 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-amz-request-id', 'tx000007922f15365994122-006320333a-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.439 DEBUG:rgw_multi.tests:zone=test-zone2 version=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:37:30.439 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.440 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.441 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.442 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.443 DEBUG:boto:Data: 2022-09-13T07:37:30.444 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.444 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:30.445 DEBUG:boto:Port: 8001 2022-09-13T07:37:30.446 DEBUG:boto:Params: {} 2022-09-13T07:37:30.447 DEBUG:boto:Token: None 2022-09-13T07:37:30.448 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.449 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.453 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef621c0766b7b859-006320333a-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.455 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.455 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.456 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.457 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.458 DEBUG:boto:Data: 2022-09-13T07:37:30.459 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.460 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:30.460 DEBUG:boto:Port: 8001 2022-09-13T07:37:30.461 DEBUG:boto:Params: {} 2022-09-13T07:37:30.462 DEBUG:boto:Token: None 2022-09-13T07:37:30.463 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.464 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-amz-request-id', 'tx00000f7706c068dca5d52-006320333a-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.477 DEBUG:rgw_multi.tests:zone=test-zone3 version=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:37:30.477 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.478 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.478 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.481 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.482 DEBUG:boto:Data: 2022-09-13T07:37:30.483 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.483 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:30.484 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.484 DEBUG:boto:Params: {} 2022-09-13T07:37:30.485 DEBUG:boto:Token: None 2022-09-13T07:37:30.485 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.486 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.491 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e27be59cde532f2f-006320333a-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.492 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.493 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.494 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.507 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.508 DEBUG:boto:Data: 2022-09-13T07:37:30.509 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.509 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:30.510 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.511 DEBUG:boto:Params: {} 2022-09-13T07:37:30.512 DEBUG:boto:Token: None 2022-09-13T07:37:30.513 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.514 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-amz-request-id', 'tx00000f2d3b6db87c8f929-006320333a-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.527 DEBUG:rgw_multi.tests:zone=test-zone1 version=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:37:30.527 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.528 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.528 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.530 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.531 DEBUG:boto:Data: 2022-09-13T07:37:30.532 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.532 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:30.533 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.534 DEBUG:boto:Params: {} 2022-09-13T07:37:30.535 DEBUG:boto:Token: None 2022-09-13T07:37:30.536 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.537 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.540 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000520d045ce10034a4-006320333a-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.541 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.542 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.542 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.543 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.543 DEBUG:boto:Data: 2022-09-13T07:37:30.544 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.544 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:30.545 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.545 DEBUG:boto:Params: {} 2022-09-13T07:37:30.546 DEBUG:boto:Token: None 2022-09-13T07:37:30.547 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.547 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-amz-request-id', 'tx000008f00d6fe4444bf4b-006320333a-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.568 DEBUG:rgw_multi.tests:zone=test-zone2 version=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:37:30.568 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.569 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.569 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.570 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.570 DEBUG:boto:Data: 2022-09-13T07:37:30.571 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.572 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:30.572 DEBUG:boto:Port: 8001 2022-09-13T07:37:30.573 DEBUG:boto:Params: {} 2022-09-13T07:37:30.574 DEBUG:boto:Token: None 2022-09-13T07:37:30.574 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.575 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.578 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004532c2f8fa73148d-006320333a-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.580 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.580 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.581 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.583 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.584 DEBUG:boto:Data: 2022-09-13T07:37:30.585 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.586 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:30.587 DEBUG:boto:Port: 8001 2022-09-13T07:37:30.587 DEBUG:boto:Params: {} 2022-09-13T07:37:30.588 DEBUG:boto:Token: None 2022-09-13T07:37:30.589 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.590 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-amz-request-id', 'tx00000b948519190a7d3cc-006320333a-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.605 DEBUG:rgw_multi.tests:zone=test-zone3 version=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:37:30.605 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.606 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.606 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.607 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.607 DEBUG:boto:Data: 2022-09-13T07:37:30.608 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.608 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:30.608 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.609 DEBUG:boto:Params: {} 2022-09-13T07:37:30.609 DEBUG:boto:Token: None 2022-09-13T07:37:30.610 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.611 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.614 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014346b78aa2610b4-006320333a-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.615 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.615 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.616 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.616 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.617 DEBUG:boto:Data: 2022-09-13T07:37:30.617 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.618 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:37:30.618 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.619 DEBUG:boto:Params: {} 2022-09-13T07:37:30.619 DEBUG:boto:Token: None 2022-09-13T07:37:30.620 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.620 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.631 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-amz-request-id', 'tx00000c42a0fa0ba2aff98-006320333a-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.632 DEBUG:rgw_multi.tests:zone=test-zone1 version=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:37:30.633 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.633 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.634 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.634 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.635 DEBUG:boto:Data: 2022-09-13T07:37:30.635 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.636 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:30.637 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.637 DEBUG:boto:Params: {} 2022-09-13T07:37:30.638 DEBUG:boto:Token: None 2022-09-13T07:37:30.639 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.639 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.642 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000457bf107022b8ee4-006320333a-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.643 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.645 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.646 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.646 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.647 DEBUG:boto:Data: 2022-09-13T07:37:30.647 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.648 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:37:30.649 DEBUG:boto:Port: 8000 2022-09-13T07:37:30.649 DEBUG:boto:Params: {} 2022-09-13T07:37:30.650 DEBUG:boto:Token: None 2022-09-13T07:37:30.650 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.651 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-amz-request-id', 'tx000005dc8b044ce37a9bd-006320333a-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.669 DEBUG:rgw_multi.tests:zone=test-zone2 version=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:37:30.670 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:37:30.670 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:37:30.671 DEBUG:boto:Method: HEAD 2022-09-13T07:37:30.671 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:37:30.672 DEBUG:boto:Data: 2022-09-13T07:37:30.673 DEBUG:boto:Headers: {} 2022-09-13T07:37:30.673 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:30.674 DEBUG:boto:Port: 8001 2022-09-13T07:37:30.674 DEBUG:boto:Params: {} 2022-09-13T07:37:30.675 DEBUG:boto:Token: None 2022-09-13T07:37:30.675 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/ 2022-09-13T07:37:30.676 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg= 2022-09-13T07:37:30.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wr5BS1WSYNE7gfHT9REs16cNEFg=', 'Content-Length': '0'} 2022-09-13T07:37:30.679 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004711c4eaa2ea065-006320333a-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.680 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:37:30.681 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:37:30.681 DEBUG:boto:Method: PUT 2022-09-13T07:37:30.682 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:37:30.682 DEBUG:boto:Data: 2022-09-13T07:37:30.683 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-13T07:37:30.683 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:37:30.684 DEBUG:boto:Port: 8001 2022-09-13T07:37:30.684 DEBUG:boto:Params: {} 2022-09-13T07:37:30.685 DEBUG:boto:Token: None 2022-09-13T07:37:30.686 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 13 Sep 2022 07:37:30 GMT /yxvvuh-19/obj 2022-09-13T07:37:30.686 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk= 2022-09-13T07:37:30.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:37:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:c31zUpgOMMbKrmd8sxkUgaaK0fk='} 2022-09-13T07:37:30.697 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-amz-request-id', 'tx00000216a77d324184190-006320333a-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:37:30.698 DEBUG:rgw_multi.tests:zone=test-zone3 version=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:37:30.699 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-19 2022-09-13T07:37:30.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:37:30.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:37:30.700 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:37:30.762 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:37:30.763 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:37:30.818 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:37:30.814+0000 7f24d746e600 1 waiting for incremental sync to catch up: 2022-09-13T07:37:30.818 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , 00000000005.12.9, , , , , ] 2022-09-13T07:37:30.819 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , 00000000005.12.9, , , 00000000020.52.11, , ] 2022-09-13T07:37:35.820 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:37:35.815+0000 7f24d746e600 1 waiting for incremental sync to catch up: 2022-09-13T07:37:35.821 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , 00000000005.12.9, , , , , ] 2022-09-13T07:37:35.821 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , 00000000005.12.9, , , 00000000020.52.11, , ] 2022-09-13T07:37:40.822 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:37:40.818+0000 7f24d746e600 1 waiting for incremental sync to catch up: 2022-09-13T07:37:40.824 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , 00000000005.12.9, , , , , ] 2022-09-13T07:37:40.824 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , 00000000005.12.9, , , 00000000020.52.11, , ] 2022-09-13T07:37:45.823 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:37:45.820+0000 7f24d746e600 1 waiting for incremental sync to catch up: 2022-09-13T07:37:45.824 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , 00000000005.12.9, , , , , ] 2022-09-13T07:37:45.825 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , 00000000005.12.9, , , 00000000020.52.11, , ] 2022-09-13T07:37:50.825 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:37:50.822+0000 7f24d746e600 1 bucket sync caught up with source: 2022-09-13T07:37:50.826 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , 00000000005.12.9, , , 00000000024.64.5, , ] 2022-09-13T07:37:50.827 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , 00000000005.12.9, , , 00000000020.52.11, , ] 2022-09-13T07:37:50.827 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:37:50.822+0000 7f24d746e600 0 bucket checkpoint complete 2022-09-13T07:37:50.843 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-19 2022-09-13T07:37:50.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:37:50.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:37:50.845 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:37:50.911 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:37:50.912 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:37:50.949 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:37:50.946+0000 7f8fcbef5600 1 waiting for incremental sync to catch up: 2022-09-13T07:37:50.949 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000005.12.9, , , 00000000020.52.11, , ] 2022-09-13T07:37:50.949 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000005.12.9, , , 00000000024.64.5, , ] 2022-09-13T07:37:55.951 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:37:55.946+0000 7f8fcbef5600 1 bucket sync caught up with source: 2022-09-13T07:37:55.951 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000005.12.9, , , 00000000024.64.5, , ] 2022-09-13T07:37:55.952 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000005.12.9, , , 00000000024.64.5, , ] 2022-09-13T07:37:55.952 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:37:55.946+0000 7f8fcbef5600 0 bucket checkpoint complete 2022-09-13T07:37:55.969 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-19 2022-09-13T07:37:55.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:37:55.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:37:55.971 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:37:56.036 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:37:56.037 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:37:56.081 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:37:56.077+0000 7fa320acd600 1 waiting for incremental sync to catch up: 2022-09-13T07:37:56.082 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000003.23.5, , ] 2022-09-13T07:37:56.082 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:38:01.083 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:38:01.079+0000 7fa320acd600 1 waiting for incremental sync to catch up: 2022-09-13T07:38:01.083 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000003.23.5, , ] 2022-09-13T07:38:01.084 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:38:06.084 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:38:06.080+0000 7fa320acd600 1 waiting for incremental sync to catch up: 2022-09-13T07:38:06.086 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000003.23.5, , ] 2022-09-13T07:38:06.086 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:38:11.086 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:38:11.082+0000 7fa320acd600 1 waiting for incremental sync to catch up: 2022-09-13T07:38:11.087 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000021.59.5, , ] 2022-09-13T07:38:11.087 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:38:16.088 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:38:16.084+0000 7fa320acd600 1 waiting for incremental sync to catch up: 2022-09-13T07:38:16.088 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000021.59.5, , ] 2022-09-13T07:38:16.088 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:38:21.091 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:38:21.085+0000 7fa320acd600 1 waiting for incremental sync to catch up: 2022-09-13T07:38:21.091 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000021.59.5, , ] 2022-09-13T07:38:21.092 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:38:26.091 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:38:26.087+0000 7fa320acd600 1 waiting for incremental sync to catch up: 2022-09-13T07:38:26.092 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000021.59.5, , ] 2022-09-13T07:38:26.092 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:38:31.092 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:38:31.088+0000 7fa320acd600 1 waiting for incremental sync to catch up: 2022-09-13T07:38:31.093 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000021.59.5, , ] 2022-09-13T07:38:31.094 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:38:36.094 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:38:36.090+0000 7fa320acd600 1 waiting for incremental sync to catch up: 2022-09-13T07:38:36.095 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000021.59.5, , ] 2022-09-13T07:38:36.095 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:38:41.095 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:38:41.091+0000 7fa320acd600 1 waiting for incremental sync to catch up: 2022-09-13T07:38:41.096 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000021.59.5, , ] 2022-09-13T07:38:41.096 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:38:46.097 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:38:46.093+0000 7fa320acd600 1 waiting for incremental sync to catch up: 2022-09-13T07:38:46.097 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000021.59.5, , ] 2022-09-13T07:38:46.098 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:38:51.098 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:38:51.094+0000 7fa320acd600 1 waiting for incremental sync to catch up: 2022-09-13T07:38:51.099 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000021.59.5, , ] 2022-09-13T07:38:51.099 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:38:56.100 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:38:56.096+0000 7fa320acd600 1 waiting for incremental sync to catch up: 2022-09-13T07:38:56.100 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000021.59.5, , ] 2022-09-13T07:38:56.101 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:39:01.101 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:39:01.097+0000 7fa320acd600 1 waiting for incremental sync to catch up: 2022-09-13T07:39:01.102 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000021.59.5, , ] 2022-09-13T07:39:01.102 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:39:06.104 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:39:06.099+0000 7fa320acd600 1 waiting for incremental sync to catch up: 2022-09-13T07:39:06.104 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000021.59.5, , ] 2022-09-13T07:39:06.105 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:39:11.105 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:39:11.102+0000 7fa320acd600 1 bucket sync caught up with source: 2022-09-13T07:39:11.105 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000044.122.9, , ] 2022-09-13T07:39:11.106 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000043.120.9, , ] 2022-09-13T07:39:11.106 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:39:11.102+0000 7fa320acd600 0 bucket checkpoint complete 2022-09-13T07:39:11.123 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-19 2022-09-13T07:39:11.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:39:11.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:39:11.127 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:39:11.193 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:11.194 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:11.230 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:39:11.227+0000 7f78e1d01600 1 bucket sync caught up with source: 2022-09-13T07:39:11.231 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000044.122.9, , ] 2022-09-13T07:39:11.231 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000044.122.9, , ] 2022-09-13T07:39:11.232 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:39:11.227+0000 7f78e1d01600 0 bucket checkpoint complete 2022-09-13T07:39:11.243 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-19 2022-09-13T07:39:11.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:39:11.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:39:11.245 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:39:11.305 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:11.306 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:11.347 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:39:11.344+0000 7f287ddd3600 1 waiting for incremental sync to catch up: 2022-09-13T07:39:11.348 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.23.9, , , 00000000028.87.5, , ] 2022-09-13T07:39:11.348 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.23.9, , , 00000000044.129.9, , ] 2022-09-13T07:39:16.348 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:39:16.345+0000 7f287ddd3600 1 waiting for incremental sync to catch up: 2022-09-13T07:39:16.349 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.23.9, , , 00000000028.87.5, , ] 2022-09-13T07:39:16.349 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.23.9, , , 00000000044.129.9, , ] 2022-09-13T07:39:21.350 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:39:21.347+0000 7f287ddd3600 1 waiting for incremental sync to catch up: 2022-09-13T07:39:21.351 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.23.9, , , 00000000028.87.5, , ] 2022-09-13T07:39:21.351 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.23.9, , , 00000000044.129.9, , ] 2022-09-13T07:39:26.351 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:39:26.348+0000 7f287ddd3600 1 bucket sync caught up with source: 2022-09-13T07:39:26.352 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , 00000000004.23.9, , , 00000000045.131.9, , ] 2022-09-13T07:39:26.352 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , 00000000004.23.9, , , 00000000044.129.9, , ] 2022-09-13T07:39:26.353 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:39:26.348+0000 7f287ddd3600 0 bucket checkpoint complete 2022-09-13T07:39:26.367 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-19 2022-09-13T07:39:26.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:39:26.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:39:26.369 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:39:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:26.474 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:39:26.469+0000 7f647cf59600 1 bucket sync caught up with source: 2022-09-13T07:39:26.474 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , 00000000004.23.9, , , 00000000045.131.9, , ] 2022-09-13T07:39:26.475 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , 00000000004.23.9, , , 00000000045.131.9, , ] 2022-09-13T07:39:26.475 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:39:26.469+0000 7f647cf59600 0 bucket checkpoint complete 2022-09-13T07:39:26.491 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-19 zones={test-zone2, test-zone1} 2022-09-13T07:39:26.492 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:26.493 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:26.495 DEBUG:boto:Method: HEAD 2022-09-13T07:39:26.495 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:39:26.496 DEBUG:boto:Data: 2022-09-13T07:39:26.497 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.498 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:26.499 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.500 DEBUG:boto:Params: {} 2022-09-13T07:39:26.502 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T07:39:26.503 DEBUG:boto:Token: None 2022-09-13T07:39:26.504 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/ 2022-09-13T07:39:26.505 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:91f+Icyu5Kqqu2z35crv3iRaYtU= 2022-09-13T07:39:26.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:91f+Icyu5Kqqu2z35crv3iRaYtU=', 'Content-Length': '0'} 2022-09-13T07:39:26.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009193857091f1921-00632033ae-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.514 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:26.515 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:26.516 DEBUG:boto:Method: HEAD 2022-09-13T07:39:26.516 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:39:26.517 DEBUG:boto:Data: 2022-09-13T07:39:26.518 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.519 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:26.520 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.521 DEBUG:boto:Params: {} 2022-09-13T07:39:26.522 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T07:39:26.522 DEBUG:boto:Token: None 2022-09-13T07:39:26.523 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/ 2022-09-13T07:39:26.524 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:91f+Icyu5Kqqu2z35crv3iRaYtU= 2022-09-13T07:39:26.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:91f+Icyu5Kqqu2z35crv3iRaYtU=', 'Content-Length': '0'} 2022-09-13T07:39:26.529 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8c31fc5c1ad5289-00632033ae-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.530 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:39:26.530 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:26.531 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:26.532 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:26.532 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:26.533 DEBUG:boto:Method: GET 2022-09-13T07:39:26.533 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:26.534 DEBUG:boto:Data: 2022-09-13T07:39:26.535 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.535 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:26.536 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.537 DEBUG:boto:Params: {} 2022-09-13T07:39:26.537 DEBUG:boto:Token: None 2022-09-13T07:39:26.538 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/?versions 2022-09-13T07:39:26.539 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wAjkn9HpOyvoprnZ5Tt36MJ9qvI= 2022-09-13T07:39:26.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wAjkn9HpOyvoprnZ5Tt36MJ9qvI=', 'Content-Length': '0'} 2022-09-13T07:39:26.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058a405f28e76311d-00632033ae-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.543 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEn7MftyiCP0pYMPcqn5sySJi4UkpaiZtrue2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmwBtbzZwRR4A0HEc28dvfcEWdzuIZDfalse2022-09-13T07:37:30.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7false2022-09-13T07:37:30.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTL9xVVVkIODKu69Lcj5HlhVjbyjGE3ffalse2022-09-13T07:37:30.515Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU.elqMxtYPda6ZsKEKDP833Q93.DHGxfalse2022-09-13T07:37:30.547Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji8qJKwt5yOnRkOjqtadujgkvccCo7V6false2022-09-13T07:37:30.592Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WBfalse2022-09-13T07:37:30.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtLDy1U4ZTxWriIUMx30zfLE7zPcVUt5false2022-09-13T07:37:30.424Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjulPqaKM9GsnZBkMYt.uabOf.VJ5P.5mfalse2022-09-13T07:37:30.465Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eAfalse2022-09-13T07:37:30.308Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnEgHn6dtpr-DjlqQutDDilFO8cCU9tVfalse2022-09-13T07:37:30.261Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzbAEfbqMIp9ReKOn2t8gGBeNLMCc26hfalse2022-09-13T07:37:30.341Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkKoYsAdAKf7lBJBYkG7TA7YHa0Qh7adfalse2022-09-13T07:37:30.152Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1false2022-09-13T07:37:30.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxwhfwIrEIbdFug3DCFDiVhftQi2CUm6false2022-09-13T07:37:30.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN1ijg0L2Sa1qSFcwgB7aYlf3fysHL.ofalse2022-09-13T07:37:30.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6pfalse2022-09-13T07:37:30.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjr3j0LS1bjhh8xQBCJCWMQPOkDqpqjuMfalse2022-09-13T07:37:30.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUFIYwFqhiCTkSFbmDIDHjWDiTsQToDGfalse2022-09-13T07:37:30.001Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjei7viyB1s5vEJjb-DgvRL3K6yeA94ngfalse2022-09-13T07:37:29.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHnhTbZbus4hq.TBh5.hecwSTeKVCaRfalse2022-09-13T07:37:30.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8bP-bkYz6JpHUV25JyCvc0fDB8aZwslfalse2022-09-13T07:37:29.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcPIo6H3tnk3IHLM4mBy4H8IXyB8CUPCfalse2022-09-13T07:37:29.842Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-false2022-09-13T07:37:29.815Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3VqUFgxLYLsisMjs5Rrmm15ZQudhjyfalse2022-09-13T07:37:29.688Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4false2022-09-13T07:37:29.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxshrAtab2xKYxtLaiLBmYWJxRHGcH2rfalse2022-09-13T07:37:29.770Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.Ifalse2022-09-13T07:37:29.573Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC-XByO2K9qUOxsnzB35XnbS7CZIHKxHfalse2022-09-13T07:37:29.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaErb35kohBMBuUIVZDyjKh5q.cT.bAJfalse2022-09-13T07:37:29.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:26.548 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:39:26.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.564 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:39:26.565 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:26.565 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:26.566 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:26.566 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:26.567 DEBUG:boto:Method: GET 2022-09-13T07:39:26.567 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:26.568 DEBUG:boto:Data: 2022-09-13T07:39:26.569 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.569 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:26.570 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.570 DEBUG:boto:Params: {} 2022-09-13T07:39:26.571 DEBUG:boto:Token: None 2022-09-13T07:39:26.571 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/?versions 2022-09-13T07:39:26.572 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wAjkn9HpOyvoprnZ5Tt36MJ9qvI= 2022-09-13T07:39:26.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wAjkn9HpOyvoprnZ5Tt36MJ9qvI=', 'Content-Length': '0'} 2022-09-13T07:39:26.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000290e92df535f3ede-00632033ae-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.577 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEn7MftyiCP0pYMPcqn5sySJi4UkpaiZtrue2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmwBtbzZwRR4A0HEc28dvfcEWdzuIZDfalse2022-09-13T07:37:30.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7false2022-09-13T07:37:30.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTL9xVVVkIODKu69Lcj5HlhVjbyjGE3ffalse2022-09-13T07:37:30.515Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU.elqMxtYPda6ZsKEKDP833Q93.DHGxfalse2022-09-13T07:37:30.547Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji8qJKwt5yOnRkOjqtadujgkvccCo7V6false2022-09-13T07:37:30.592Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WBfalse2022-09-13T07:37:30.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtLDy1U4ZTxWriIUMx30zfLE7zPcVUt5false2022-09-13T07:37:30.424Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjulPqaKM9GsnZBkMYt.uabOf.VJ5P.5mfalse2022-09-13T07:37:30.465Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eAfalse2022-09-13T07:37:30.308Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnEgHn6dtpr-DjlqQutDDilFO8cCU9tVfalse2022-09-13T07:37:30.261Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzbAEfbqMIp9ReKOn2t8gGBeNLMCc26hfalse2022-09-13T07:37:30.341Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkKoYsAdAKf7lBJBYkG7TA7YHa0Qh7adfalse2022-09-13T07:37:30.152Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1false2022-09-13T07:37:30.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxwhfwIrEIbdFug3DCFDiVhftQi2CUm6false2022-09-13T07:37:30.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN1ijg0L2Sa1qSFcwgB7aYlf3fysHL.ofalse2022-09-13T07:37:30.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6pfalse2022-09-13T07:37:30.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjr3j0LS1bjhh8xQBCJCWMQPOkDqpqjuMfalse2022-09-13T07:37:30.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUFIYwFqhiCTkSFbmDIDHjWDiTsQToDGfalse2022-09-13T07:37:30.001Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjei7viyB1s5vEJjb-DgvRL3K6yeA94ngfalse2022-09-13T07:37:29.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHnhTbZbus4hq.TBh5.hecwSTeKVCaRfalse2022-09-13T07:37:30.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8bP-bkYz6JpHUV25JyCvc0fDB8aZwslfalse2022-09-13T07:37:29.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcPIo6H3tnk3IHLM4mBy4H8IXyB8CUPCfalse2022-09-13T07:37:29.842Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-false2022-09-13T07:37:29.815Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3VqUFgxLYLsisMjs5Rrmm15ZQudhjyfalse2022-09-13T07:37:29.688Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4false2022-09-13T07:37:29.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxshrAtab2xKYxtLaiLBmYWJxRHGcH2rfalse2022-09-13T07:37:29.770Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.Ifalse2022-09-13T07:37:29.573Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC-XByO2K9qUOxsnzB35XnbS7CZIHKxHfalse2022-09-13T07:37:29.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaErb35kohBMBuUIVZDyjKh5q.cT.bAJfalse2022-09-13T07:37:29.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:26.580 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:39:26.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:26.597 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:26.597 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:26.598 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:26.599 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:26.600 DEBUG:boto:Method: GET 2022-09-13T07:39:26.600 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:26.601 DEBUG:boto:Data: 2022-09-13T07:39:26.601 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.602 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:26.603 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.603 DEBUG:boto:Params: {} 2022-09-13T07:39:26.604 DEBUG:boto:Token: None 2022-09-13T07:39:26.604 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/?versions 2022-09-13T07:39:26.605 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wAjkn9HpOyvoprnZ5Tt36MJ9qvI= 2022-09-13T07:39:26.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wAjkn9HpOyvoprnZ5Tt36MJ9qvI=', 'Content-Length': '0'} 2022-09-13T07:39:26.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013efd7f5fed7a535-00632033ae-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.610 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEn7MftyiCP0pYMPcqn5sySJi4UkpaiZtrue2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmwBtbzZwRR4A0HEc28dvfcEWdzuIZDfalse2022-09-13T07:37:30.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7false2022-09-13T07:37:30.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTL9xVVVkIODKu69Lcj5HlhVjbyjGE3ffalse2022-09-13T07:37:30.515Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU.elqMxtYPda6ZsKEKDP833Q93.DHGxfalse2022-09-13T07:37:30.547Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji8qJKwt5yOnRkOjqtadujgkvccCo7V6false2022-09-13T07:37:30.592Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WBfalse2022-09-13T07:37:30.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtLDy1U4ZTxWriIUMx30zfLE7zPcVUt5false2022-09-13T07:37:30.424Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjulPqaKM9GsnZBkMYt.uabOf.VJ5P.5mfalse2022-09-13T07:37:30.465Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eAfalse2022-09-13T07:37:30.308Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnEgHn6dtpr-DjlqQutDDilFO8cCU9tVfalse2022-09-13T07:37:30.261Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzbAEfbqMIp9ReKOn2t8gGBeNLMCc26hfalse2022-09-13T07:37:30.341Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkKoYsAdAKf7lBJBYkG7TA7YHa0Qh7adfalse2022-09-13T07:37:30.152Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1false2022-09-13T07:37:30.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxwhfwIrEIbdFug3DCFDiVhftQi2CUm6false2022-09-13T07:37:30.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN1ijg0L2Sa1qSFcwgB7aYlf3fysHL.ofalse2022-09-13T07:37:30.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6pfalse2022-09-13T07:37:30.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjr3j0LS1bjhh8xQBCJCWMQPOkDqpqjuMfalse2022-09-13T07:37:30.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUFIYwFqhiCTkSFbmDIDHjWDiTsQToDGfalse2022-09-13T07:37:30.001Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjei7viyB1s5vEJjb-DgvRL3K6yeA94ngfalse2022-09-13T07:37:29.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHnhTbZbus4hq.TBh5.hecwSTeKVCaRfalse2022-09-13T07:37:30.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8bP-bkYz6JpHUV25JyCvc0fDB8aZwslfalse2022-09-13T07:37:29.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcPIo6H3tnk3IHLM4mBy4H8IXyB8CUPCfalse2022-09-13T07:37:29.842Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-false2022-09-13T07:37:29.815Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3VqUFgxLYLsisMjs5Rrmm15ZQudhjyfalse2022-09-13T07:37:29.688Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4false2022-09-13T07:37:29.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxshrAtab2xKYxtLaiLBmYWJxRHGcH2rfalse2022-09-13T07:37:29.770Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.Ifalse2022-09-13T07:37:29.573Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC-XByO2K9qUOxsnzB35XnbS7CZIHKxHfalse2022-09-13T07:37:29.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaErb35kohBMBuUIVZDyjKh5q.cT.bAJfalse2022-09-13T07:37:29.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:26.612 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:26.612 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:26.613 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:26.613 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:26.614 DEBUG:boto:Method: GET 2022-09-13T07:39:26.615 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:26.615 DEBUG:boto:Data: 2022-09-13T07:39:26.616 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.616 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:26.617 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.617 DEBUG:boto:Params: {} 2022-09-13T07:39:26.618 DEBUG:boto:Token: None 2022-09-13T07:39:26.618 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/?versions 2022-09-13T07:39:26.619 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wAjkn9HpOyvoprnZ5Tt36MJ9qvI= 2022-09-13T07:39:26.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wAjkn9HpOyvoprnZ5Tt36MJ9qvI=', 'Content-Length': '0'} 2022-09-13T07:39:26.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b0a20b89fad94de-00632033ae-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.624 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEn7MftyiCP0pYMPcqn5sySJi4UkpaiZtrue2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmwBtbzZwRR4A0HEc28dvfcEWdzuIZDfalse2022-09-13T07:37:30.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7false2022-09-13T07:37:30.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTL9xVVVkIODKu69Lcj5HlhVjbyjGE3ffalse2022-09-13T07:37:30.515Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU.elqMxtYPda6ZsKEKDP833Q93.DHGxfalse2022-09-13T07:37:30.547Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji8qJKwt5yOnRkOjqtadujgkvccCo7V6false2022-09-13T07:37:30.592Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WBfalse2022-09-13T07:37:30.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtLDy1U4ZTxWriIUMx30zfLE7zPcVUt5false2022-09-13T07:37:30.424Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjulPqaKM9GsnZBkMYt.uabOf.VJ5P.5mfalse2022-09-13T07:37:30.465Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eAfalse2022-09-13T07:37:30.308Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnEgHn6dtpr-DjlqQutDDilFO8cCU9tVfalse2022-09-13T07:37:30.261Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzbAEfbqMIp9ReKOn2t8gGBeNLMCc26hfalse2022-09-13T07:37:30.341Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkKoYsAdAKf7lBJBYkG7TA7YHa0Qh7adfalse2022-09-13T07:37:30.152Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1false2022-09-13T07:37:30.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxwhfwIrEIbdFug3DCFDiVhftQi2CUm6false2022-09-13T07:37:30.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN1ijg0L2Sa1qSFcwgB7aYlf3fysHL.ofalse2022-09-13T07:37:30.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6pfalse2022-09-13T07:37:30.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjr3j0LS1bjhh8xQBCJCWMQPOkDqpqjuMfalse2022-09-13T07:37:30.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUFIYwFqhiCTkSFbmDIDHjWDiTsQToDGfalse2022-09-13T07:37:30.001Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjei7viyB1s5vEJjb-DgvRL3K6yeA94ngfalse2022-09-13T07:37:29.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHnhTbZbus4hq.TBh5.hecwSTeKVCaRfalse2022-09-13T07:37:30.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8bP-bkYz6JpHUV25JyCvc0fDB8aZwslfalse2022-09-13T07:37:29.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcPIo6H3tnk3IHLM4mBy4H8IXyB8CUPCfalse2022-09-13T07:37:29.842Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-false2022-09-13T07:37:29.815Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3VqUFgxLYLsisMjs5Rrmm15ZQudhjyfalse2022-09-13T07:37:29.688Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4false2022-09-13T07:37:29.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxshrAtab2xKYxtLaiLBmYWJxRHGcH2rfalse2022-09-13T07:37:29.770Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.Ifalse2022-09-13T07:37:29.573Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC-XByO2K9qUOxsnzB35XnbS7CZIHKxHfalse2022-09-13T07:37:29.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaErb35kohBMBuUIVZDyjKh5q.cT.bAJfalse2022-09-13T07:37:29.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:26.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:39:26.626 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:26.627 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:26.628 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.628 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.629 DEBUG:boto:Method: GET 2022-09-13T07:39:26.629 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.630 DEBUG:boto:Data: 2022-09-13T07:39:26.630 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.631 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:26.631 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.632 DEBUG:boto:Params: {} 2022-09-13T07:39:26.632 DEBUG:boto:Token: None 2022-09-13T07:39:26.633 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.633 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iYM9nOoyCEjUA05BABz7IBXJc0c= 2022-09-13T07:39:26.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iYM9nOoyCEjUA05BABz7IBXJc0c=', 'Content-Length': '0'} 2022-09-13T07:39:26.636 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000faaa76492fd1a100-00632033ae-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.637 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:26.637 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:26.638 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.638 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.639 DEBUG:boto:Method: GET 2022-09-13T07:39:26.639 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.640 DEBUG:boto:Data: 2022-09-13T07:39:26.640 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.641 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:26.642 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.642 DEBUG:boto:Params: {} 2022-09-13T07:39:26.643 DEBUG:boto:Token: None 2022-09-13T07:39:26.643 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.644 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iYM9nOoyCEjUA05BABz7IBXJc0c= 2022-09-13T07:39:26.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iYM9nOoyCEjUA05BABz7IBXJc0c=', 'Content-Length': '0'} 2022-09-13T07:39:26.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006de5689ca8da0bb4-00632033ae-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.648 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:26.648 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:26.649 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.649 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.650 DEBUG:boto:Method: HEAD 2022-09-13T07:39:26.650 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.651 DEBUG:boto:Data: 2022-09-13T07:39:26.652 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.652 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:26.653 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.653 DEBUG:boto:Params: {} 2022-09-13T07:39:26.654 DEBUG:boto:Token: None 2022-09-13T07:39:26.654 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.655 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fwEDvqXhfugtntzPfGEhYyS/PJM= 2022-09-13T07:39:26.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fwEDvqXhfugtntzPfGEhYyS/PJM=', 'Content-Length': '0'} 2022-09-13T07:39:26.657 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000627f17e581f9061d-00632033ae-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.659 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:26.659 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:26.660 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.661 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.662 DEBUG:boto:Method: HEAD 2022-09-13T07:39:26.663 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.664 DEBUG:boto:Data: 2022-09-13T07:39:26.665 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.666 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:26.667 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.668 DEBUG:boto:Params: {} 2022-09-13T07:39:26.668 DEBUG:boto:Token: None 2022-09-13T07:39:26.669 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.670 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fwEDvqXhfugtntzPfGEhYyS/PJM= 2022-09-13T07:39:26.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fwEDvqXhfugtntzPfGEhYyS/PJM=', 'Content-Length': '0'} 2022-09-13T07:39:26.674 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000082c89a0774d06cb6-00632033ae-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:39:26.676 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:26.677 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:26.678 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.679 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.680 DEBUG:boto:Method: GET 2022-09-13T07:39:26.680 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.681 DEBUG:boto:Data: 2022-09-13T07:39:26.682 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.683 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:26.684 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.685 DEBUG:boto:Params: {} 2022-09-13T07:39:26.686 DEBUG:boto:Token: None 2022-09-13T07:39:26.687 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.687 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iYM9nOoyCEjUA05BABz7IBXJc0c= 2022-09-13T07:39:26.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iYM9nOoyCEjUA05BABz7IBXJc0c=', 'Content-Length': '0'} 2022-09-13T07:39:26.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cbb41eb00dc8504f-00632033ae-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.691 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:26.692 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:26.692 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.693 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.693 DEBUG:boto:Method: GET 2022-09-13T07:39:26.694 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.695 DEBUG:boto:Data: 2022-09-13T07:39:26.695 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.696 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:26.696 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.697 DEBUG:boto:Params: {} 2022-09-13T07:39:26.698 DEBUG:boto:Token: None 2022-09-13T07:39:26.698 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.699 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iYM9nOoyCEjUA05BABz7IBXJc0c= 2022-09-13T07:39:26.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iYM9nOoyCEjUA05BABz7IBXJc0c=', 'Content-Length': '0'} 2022-09-13T07:39:26.701 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023d02d4ee15cec38-00632033ae-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.702 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:26.703 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:26.703 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:26.704 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:26.704 DEBUG:boto:Method: GET 2022-09-13T07:39:26.705 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:39:26.706 DEBUG:boto:Data: 2022-09-13T07:39:26.706 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.707 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:26.707 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.708 DEBUG:boto:Params: {} 2022-09-13T07:39:26.708 DEBUG:boto:Token: None 2022-09-13T07:39:26.709 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/ 2022-09-13T07:39:26.709 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7dWckaVRckWpWRmW1oxX9dgG0bE= 2022-09-13T07:39:26.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7dWckaVRckWpWRmW1oxX9dgG0bE=', 'Content-Length': '0'} 2022-09-13T07:39:26.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007749536d00e4e8c-00632033ae-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.713 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:26.714 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:26.715 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:26.715 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:26.716 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:26.716 DEBUG:boto:Method: GET 2022-09-13T07:39:26.717 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:39:26.717 DEBUG:boto:Data: 2022-09-13T07:39:26.718 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.719 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:26.719 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.720 DEBUG:boto:Params: {} 2022-09-13T07:39:26.720 DEBUG:boto:Token: None 2022-09-13T07:39:26.721 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/ 2022-09-13T07:39:26.721 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7dWckaVRckWpWRmW1oxX9dgG0bE= 2022-09-13T07:39:26.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7dWckaVRckWpWRmW1oxX9dgG0bE=', 'Content-Length': '0'} 2022-09-13T07:39:26.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb29a3a4e8c989dd-00632033ae-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.766 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:26.767 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:26.768 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:26.768 DEBUG:boto:Method: HEAD 2022-09-13T07:39:26.769 DEBUG:boto:Path: /yxvvuh-19/dummy 2022-09-13T07:39:26.769 DEBUG:boto:Data: 2022-09-13T07:39:26.770 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.770 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:26.771 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.771 DEBUG:boto:Params: {} 2022-09-13T07:39:26.772 DEBUG:boto:Token: None 2022-09-13T07:39:26.773 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/dummy 2022-09-13T07:39:26.774 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KvVeJZU3MBeZ+KEr76YDQ6RxO3Q= 2022-09-13T07:39:26.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KvVeJZU3MBeZ+KEr76YDQ6RxO3Q=', 'Content-Length': '0'} 2022-09-13T07:39:26.777 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a6b60961576e113a-00632033ae-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.779 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:26.779 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:26.780 DEBUG:boto:Method: HEAD 2022-09-13T07:39:26.781 DEBUG:boto:Path: /yxvvuh-19/dummy 2022-09-13T07:39:26.781 DEBUG:boto:Data: 2022-09-13T07:39:26.782 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.783 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:26.784 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.785 DEBUG:boto:Params: {} 2022-09-13T07:39:26.786 DEBUG:boto:Token: None 2022-09-13T07:39:26.787 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/dummy 2022-09-13T07:39:26.788 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KvVeJZU3MBeZ+KEr76YDQ6RxO3Q= 2022-09-13T07:39:26.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KvVeJZU3MBeZ+KEr76YDQ6RxO3Q=', 'Content-Length': '0'} 2022-09-13T07:39:26.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000968b91b3efe99ffe-00632033ae-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:39:26.795 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:26.796 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:26.796 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.797 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.797 DEBUG:boto:Method: GET 2022-09-13T07:39:26.798 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.798 DEBUG:boto:Data: 2022-09-13T07:39:26.799 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.799 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:26.800 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.801 DEBUG:boto:Params: {} 2022-09-13T07:39:26.802 DEBUG:boto:Token: None 2022-09-13T07:39:26.803 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.804 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iYM9nOoyCEjUA05BABz7IBXJc0c= 2022-09-13T07:39:26.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iYM9nOoyCEjUA05BABz7IBXJc0c=', 'Content-Length': '0'} 2022-09-13T07:39:26.807 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000056c0b548b07a21af-00632033ae-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.808 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:26.809 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:26.810 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.810 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.811 DEBUG:boto:Method: GET 2022-09-13T07:39:26.811 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.812 DEBUG:boto:Data: 2022-09-13T07:39:26.812 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.813 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:26.814 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.814 DEBUG:boto:Params: {} 2022-09-13T07:39:26.815 DEBUG:boto:Token: None 2022-09-13T07:39:26.815 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:26.816 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iYM9nOoyCEjUA05BABz7IBXJc0c= 2022-09-13T07:39:26.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iYM9nOoyCEjUA05BABz7IBXJc0c=', 'Content-Length': '0'} 2022-09-13T07:39:26.819 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e44e1ccaf753efff-00632033ae-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.820 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:26.820 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:26.821 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:26.821 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:26.822 DEBUG:boto:Method: GET 2022-09-13T07:39:26.822 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:39:26.823 DEBUG:boto:Data: 2022-09-13T07:39:26.824 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.824 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:26.825 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.825 DEBUG:boto:Params: {} 2022-09-13T07:39:26.826 DEBUG:boto:Token: None 2022-09-13T07:39:26.826 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/ 2022-09-13T07:39:26.827 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7dWckaVRckWpWRmW1oxX9dgG0bE= 2022-09-13T07:39:26.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7dWckaVRckWpWRmW1oxX9dgG0bE=', 'Content-Length': '0'} 2022-09-13T07:39:26.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005047c443ee2fe73-00632033ae-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.832 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:26.833 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:26.833 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:26.834 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:26.834 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:26.835 DEBUG:boto:Method: GET 2022-09-13T07:39:26.835 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:39:26.836 DEBUG:boto:Data: 2022-09-13T07:39:26.836 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.837 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:26.837 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.838 DEBUG:boto:Params: {} 2022-09-13T07:39:26.838 DEBUG:boto:Token: None 2022-09-13T07:39:26.839 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/ 2022-09-13T07:39:26.840 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7dWckaVRckWpWRmW1oxX9dgG0bE= 2022-09-13T07:39:26.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7dWckaVRckWpWRmW1oxX9dgG0bE=', 'Content-Length': '0'} 2022-09-13T07:39:26.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1b30820faed0698-00632033ae-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.844 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:26.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:26.845 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:26.846 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:26.846 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.847 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.847 DEBUG:boto:Method: GET 2022-09-13T07:39:26.848 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.848 DEBUG:boto:Data: 2022-09-13T07:39:26.849 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.849 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:26.850 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.850 DEBUG:boto:Params: {} 2022-09-13T07:39:26.851 DEBUG:boto:Token: None 2022-09-13T07:39:26.852 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.852 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dlQc9SwDpq18tiWHb6loygtObCY= 2022-09-13T07:39:26.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dlQc9SwDpq18tiWHb6loygtObCY=', 'Content-Length': '0'} 2022-09-13T07:39:26.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca58ccb3ee6ea813-00632033ae-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.856 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:26.856 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:26.857 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.858 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.858 DEBUG:boto:Method: GET 2022-09-13T07:39:26.859 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.859 DEBUG:boto:Data: 2022-09-13T07:39:26.860 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.860 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:26.861 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.861 DEBUG:boto:Params: {} 2022-09-13T07:39:26.862 DEBUG:boto:Token: None 2022-09-13T07:39:26.863 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.863 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dlQc9SwDpq18tiWHb6loygtObCY= 2022-09-13T07:39:26.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dlQc9SwDpq18tiWHb6loygtObCY=', 'Content-Length': '0'} 2022-09-13T07:39:26.866 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b530916e3a9d083-00632033ae-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.910 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:26.911 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:26.911 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.912 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.912 DEBUG:boto:Method: HEAD 2022-09-13T07:39:26.913 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.913 DEBUG:boto:Data: 2022-09-13T07:39:26.914 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.914 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:26.915 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.915 DEBUG:boto:Params: {} 2022-09-13T07:39:26.916 DEBUG:boto:Token: None 2022-09-13T07:39:26.916 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.917 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vJUgguawGJmysUzRV8Znrp/EWh4= 2022-09-13T07:39:26.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vJUgguawGJmysUzRV8Znrp/EWh4=', 'Content-Length': '0'} 2022-09-13T07:39:26.920 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f17c87e6878e05f7-00632033ae-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.920 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:26.921 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:26.921 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.922 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.922 DEBUG:boto:Method: HEAD 2022-09-13T07:39:26.923 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.924 DEBUG:boto:Data: 2022-09-13T07:39:26.924 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.925 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:26.925 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.926 DEBUG:boto:Params: {} 2022-09-13T07:39:26.926 DEBUG:boto:Token: None 2022-09-13T07:39:26.927 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.927 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vJUgguawGJmysUzRV8Znrp/EWh4= 2022-09-13T07:39:26.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vJUgguawGJmysUzRV8Znrp/EWh4=', 'Content-Length': '0'} 2022-09-13T07:39:26.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ad33011d2a5785e-00632033ae-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:26.931 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:26.932 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:26.932 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.933 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.934 DEBUG:boto:Method: GET 2022-09-13T07:39:26.934 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.935 DEBUG:boto:Data: 2022-09-13T07:39:26.935 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.936 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:26.937 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.937 DEBUG:boto:Params: {} 2022-09-13T07:39:26.938 DEBUG:boto:Token: None 2022-09-13T07:39:26.938 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.939 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dlQc9SwDpq18tiWHb6loygtObCY= 2022-09-13T07:39:26.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dlQc9SwDpq18tiWHb6loygtObCY=', 'Content-Length': '0'} 2022-09-13T07:39:26.942 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000733ba496d4789717-00632033ae-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.943 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:26.943 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:26.944 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.944 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.945 DEBUG:boto:Method: GET 2022-09-13T07:39:26.946 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.946 DEBUG:boto:Data: 2022-09-13T07:39:26.947 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.948 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:26.948 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.949 DEBUG:boto:Params: {} 2022-09-13T07:39:26.949 DEBUG:boto:Token: None 2022-09-13T07:39:26.950 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:26.951 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dlQc9SwDpq18tiWHb6loygtObCY= 2022-09-13T07:39:26.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dlQc9SwDpq18tiWHb6loygtObCY=', 'Content-Length': '0'} 2022-09-13T07:39:26.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000264f843cc3a7d994-00632033ae-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.954 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:26.955 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:26.956 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:26.956 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:26.957 DEBUG:boto:Method: GET 2022-09-13T07:39:26.957 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:26.958 DEBUG:boto:Data: 2022-09-13T07:39:26.959 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.960 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:26.961 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.961 DEBUG:boto:Params: {} 2022-09-13T07:39:26.962 DEBUG:boto:Token: None 2022-09-13T07:39:26.962 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/ 2022-09-13T07:39:26.963 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7dWckaVRckWpWRmW1oxX9dgG0bE= 2022-09-13T07:39:26.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7dWckaVRckWpWRmW1oxX9dgG0bE=', 'Content-Length': '0'} 2022-09-13T07:39:26.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000457d911dd78f58e6-00632033ae-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.967 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:26.968 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:26.969 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:26.970 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:26.970 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:26.971 DEBUG:boto:Method: GET 2022-09-13T07:39:26.972 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:26.973 DEBUG:boto:Data: 2022-09-13T07:39:26.974 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.975 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:26.975 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.976 DEBUG:boto:Params: {} 2022-09-13T07:39:26.977 DEBUG:boto:Token: None 2022-09-13T07:39:26.978 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/ 2022-09-13T07:39:26.979 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7dWckaVRckWpWRmW1oxX9dgG0bE= 2022-09-13T07:39:26.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7dWckaVRckWpWRmW1oxX9dgG0bE=', 'Content-Length': '0'} 2022-09-13T07:39:26.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000044905839aee288e-00632033ae-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.984 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:26.985 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:26.985 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:26.986 DEBUG:boto:Method: HEAD 2022-09-13T07:39:26.987 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:26.988 DEBUG:boto:Data: 2022-09-13T07:39:26.989 DEBUG:boto:Headers: {} 2022-09-13T07:39:26.989 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:26.990 DEBUG:boto:Port: 8000 2022-09-13T07:39:26.991 DEBUG:boto:Params: {} 2022-09-13T07:39:26.992 DEBUG:boto:Token: None 2022-09-13T07:39:26.993 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:26 GMT /yxvvuh-19/obj 2022-09-13T07:39:26.994 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LmBugC2Dcm/Jiu5zC5VjAUgF7Ps= 2022-09-13T07:39:26.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:26 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LmBugC2Dcm/Jiu5zC5VjAUgF7Ps=', 'Content-Length': '0'} 2022-09-13T07:39:26.998 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d4185415c2365de-00632033ae-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:26.999 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.000 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.000 DEBUG:boto:Method: HEAD 2022-09-13T07:39:27.001 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:27.002 DEBUG:boto:Data: 2022-09-13T07:39:27.002 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.003 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.003 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.004 DEBUG:boto:Params: {} 2022-09-13T07:39:27.004 DEBUG:boto:Token: None 2022-09-13T07:39:27.005 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj 2022-09-13T07:39:27.005 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uhck/W3nzw0nEF0+ZmEs/viFZFI= 2022-09-13T07:39:27.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uhck/W3nzw0nEF0+ZmEs/viFZFI=', 'Content-Length': '0'} 2022-09-13T07:39:27.010 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005859d49e01c5b0e4-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:27.012 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.013 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.014 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.014 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.015 DEBUG:boto:Method: GET 2022-09-13T07:39:27.015 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.016 DEBUG:boto:Data: 2022-09-13T07:39:27.016 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.017 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.017 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.018 DEBUG:boto:Params: {} 2022-09-13T07:39:27.018 DEBUG:boto:Token: None 2022-09-13T07:39:27.019 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.020 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I= 2022-09-13T07:39:27.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I=', 'Content-Length': '0'} 2022-09-13T07:39:27.022 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000def6bbb8b56563ab-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.067 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.068 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.069 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.070 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.071 DEBUG:boto:Method: GET 2022-09-13T07:39:27.072 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.073 DEBUG:boto:Data: 2022-09-13T07:39:27.074 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.075 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.076 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.077 DEBUG:boto:Params: {} 2022-09-13T07:39:27.078 DEBUG:boto:Token: None 2022-09-13T07:39:27.079 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.080 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I= 2022-09-13T07:39:27.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I=', 'Content-Length': '0'} 2022-09-13T07:39:27.083 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7b3f456b010f65d-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.084 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:27.084 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:27.085 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.085 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.086 DEBUG:boto:Method: GET 2022-09-13T07:39:27.086 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.087 DEBUG:boto:Data: 2022-09-13T07:39:27.088 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.088 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.089 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.089 DEBUG:boto:Params: {} 2022-09-13T07:39:27.090 DEBUG:boto:Token: None 2022-09-13T07:39:27.091 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/ 2022-09-13T07:39:27.091 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU= 2022-09-13T07:39:27.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU=', 'Content-Length': '0'} 2022-09-13T07:39:27.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f101dce19e958a5-00632033af-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.095 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:27.096 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:27.097 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:27.098 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.098 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.099 DEBUG:boto:Method: GET 2022-09-13T07:39:27.099 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.100 DEBUG:boto:Data: 2022-09-13T07:39:27.100 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.101 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.101 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.102 DEBUG:boto:Params: {} 2022-09-13T07:39:27.102 DEBUG:boto:Token: None 2022-09-13T07:39:27.103 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/ 2022-09-13T07:39:27.104 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU= 2022-09-13T07:39:27.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU=', 'Content-Length': '0'} 2022-09-13T07:39:27.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000406b638b92fc717f-00632033af-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.108 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:27.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:27.110 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.110 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.111 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.112 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.112 DEBUG:boto:Method: GET 2022-09-13T07:39:27.113 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.113 DEBUG:boto:Data: 2022-09-13T07:39:27.114 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.114 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.115 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.115 DEBUG:boto:Params: {} 2022-09-13T07:39:27.117 DEBUG:boto:Token: None 2022-09-13T07:39:27.117 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.118 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dqxkjGiAlAE+CbInjYNpRg04ONY= 2022-09-13T07:39:27.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dqxkjGiAlAE+CbInjYNpRg04ONY=', 'Content-Length': '0'} 2022-09-13T07:39:27.122 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fc7f8fa643886ef-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.123 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.124 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.124 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.125 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.125 DEBUG:boto:Method: GET 2022-09-13T07:39:27.126 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.126 DEBUG:boto:Data: 2022-09-13T07:39:27.127 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.127 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.128 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.128 DEBUG:boto:Params: {} 2022-09-13T07:39:27.129 DEBUG:boto:Token: None 2022-09-13T07:39:27.130 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.130 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dqxkjGiAlAE+CbInjYNpRg04ONY= 2022-09-13T07:39:27.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dqxkjGiAlAE+CbInjYNpRg04ONY=', 'Content-Length': '0'} 2022-09-13T07:39:27.134 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a1c336d7a7a4fc5-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.135 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.135 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.136 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.137 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.138 DEBUG:boto:Method: HEAD 2022-09-13T07:39:27.139 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.139 DEBUG:boto:Data: 2022-09-13T07:39:27.140 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.141 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.142 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.143 DEBUG:boto:Params: {} 2022-09-13T07:39:27.144 DEBUG:boto:Token: None 2022-09-13T07:39:27.145 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.146 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9+R6IPlUXr6mIM+HpMRu7WvPXW8= 2022-09-13T07:39:27.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9+R6IPlUXr6mIM+HpMRu7WvPXW8=', 'Content-Length': '0'} 2022-09-13T07:39:27.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003860cbce54465251-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.151 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.152 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.153 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.153 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.154 DEBUG:boto:Method: HEAD 2022-09-13T07:39:27.155 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.156 DEBUG:boto:Data: 2022-09-13T07:39:27.156 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.157 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.158 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.158 DEBUG:boto:Params: {} 2022-09-13T07:39:27.159 DEBUG:boto:Token: None 2022-09-13T07:39:27.160 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.160 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9+R6IPlUXr6mIM+HpMRu7WvPXW8= 2022-09-13T07:39:27.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9+R6IPlUXr6mIM+HpMRu7WvPXW8=', 'Content-Length': '0'} 2022-09-13T07:39:27.163 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000980f5060ca91468e-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:27.164 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.165 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.165 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.166 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.167 DEBUG:boto:Method: GET 2022-09-13T07:39:27.167 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.168 DEBUG:boto:Data: 2022-09-13T07:39:27.169 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.169 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.170 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.170 DEBUG:boto:Params: {} 2022-09-13T07:39:27.172 DEBUG:boto:Token: None 2022-09-13T07:39:27.173 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.174 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dqxkjGiAlAE+CbInjYNpRg04ONY= 2022-09-13T07:39:27.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dqxkjGiAlAE+CbInjYNpRg04ONY=', 'Content-Length': '0'} 2022-09-13T07:39:27.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ebbe00c1df534206-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.218 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.219 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.220 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.220 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.221 DEBUG:boto:Method: GET 2022-09-13T07:39:27.222 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.222 DEBUG:boto:Data: 2022-09-13T07:39:27.223 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.223 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.224 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.224 DEBUG:boto:Params: {} 2022-09-13T07:39:27.225 DEBUG:boto:Token: None 2022-09-13T07:39:27.226 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:27.227 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dqxkjGiAlAE+CbInjYNpRg04ONY= 2022-09-13T07:39:27.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dqxkjGiAlAE+CbInjYNpRg04ONY=', 'Content-Length': '0'} 2022-09-13T07:39:27.230 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d715510394a1a29e-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.231 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:27.231 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:27.232 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.232 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.233 DEBUG:boto:Method: GET 2022-09-13T07:39:27.234 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.234 DEBUG:boto:Data: 2022-09-13T07:39:27.235 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.235 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.236 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.236 DEBUG:boto:Params: {} 2022-09-13T07:39:27.237 DEBUG:boto:Token: None 2022-09-13T07:39:27.237 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/ 2022-09-13T07:39:27.238 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU= 2022-09-13T07:39:27.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU=', 'Content-Length': '0'} 2022-09-13T07:39:27.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e4ee6f2a2ff892b-00632033af-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.242 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:27.243 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:27.244 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:27.244 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.245 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.245 DEBUG:boto:Method: GET 2022-09-13T07:39:27.246 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.247 DEBUG:boto:Data: 2022-09-13T07:39:27.247 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.248 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.248 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.249 DEBUG:boto:Params: {} 2022-09-13T07:39:27.249 DEBUG:boto:Token: None 2022-09-13T07:39:27.250 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/ 2022-09-13T07:39:27.251 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU= 2022-09-13T07:39:27.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU=', 'Content-Length': '0'} 2022-09-13T07:39:27.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bbcba4d17e5ee31-00632033af-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.298 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:27.299 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.300 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.300 DEBUG:boto:Method: HEAD 2022-09-13T07:39:27.301 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:27.302 DEBUG:boto:Data: 2022-09-13T07:39:27.302 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.303 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.303 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.304 DEBUG:boto:Params: {} 2022-09-13T07:39:27.304 DEBUG:boto:Token: None 2022-09-13T07:39:27.305 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj 2022-09-13T07:39:27.305 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uhck/W3nzw0nEF0+ZmEs/viFZFI= 2022-09-13T07:39:27.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uhck/W3nzw0nEF0+ZmEs/viFZFI=', 'Content-Length': '0'} 2022-09-13T07:39:27.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bfdf37e8b038305-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.312 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.312 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.313 DEBUG:boto:Method: HEAD 2022-09-13T07:39:27.313 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:27.314 DEBUG:boto:Data: 2022-09-13T07:39:27.314 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.315 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.315 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.316 DEBUG:boto:Params: {} 2022-09-13T07:39:27.316 DEBUG:boto:Token: None 2022-09-13T07:39:27.317 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj 2022-09-13T07:39:27.318 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uhck/W3nzw0nEF0+ZmEs/viFZFI= 2022-09-13T07:39:27.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uhck/W3nzw0nEF0+ZmEs/viFZFI=', 'Content-Length': '0'} 2022-09-13T07:39:27.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e88eac763ba6253-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:27.324 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.325 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.325 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.326 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.327 DEBUG:boto:Method: GET 2022-09-13T07:39:27.328 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.328 DEBUG:boto:Data: 2022-09-13T07:39:27.329 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.329 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.330 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.330 DEBUG:boto:Params: {} 2022-09-13T07:39:27.331 DEBUG:boto:Token: None 2022-09-13T07:39:27.332 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.332 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I= 2022-09-13T07:39:27.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I=', 'Content-Length': '0'} 2022-09-13T07:39:27.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6e5508f968676f4-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.336 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.337 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.338 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.338 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.339 DEBUG:boto:Method: GET 2022-09-13T07:39:27.339 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.340 DEBUG:boto:Data: 2022-09-13T07:39:27.340 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.341 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.341 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.342 DEBUG:boto:Params: {} 2022-09-13T07:39:27.342 DEBUG:boto:Token: None 2022-09-13T07:39:27.343 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.343 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I= 2022-09-13T07:39:27.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I=', 'Content-Length': '0'} 2022-09-13T07:39:27.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052755ab66358f76e-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.347 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:27.348 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:27.348 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.349 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.349 DEBUG:boto:Method: GET 2022-09-13T07:39:27.350 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.350 DEBUG:boto:Data: 2022-09-13T07:39:27.351 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.351 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.352 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.352 DEBUG:boto:Params: {} 2022-09-13T07:39:27.353 DEBUG:boto:Token: None 2022-09-13T07:39:27.354 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/ 2022-09-13T07:39:27.354 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU= 2022-09-13T07:39:27.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU=', 'Content-Length': '0'} 2022-09-13T07:39:27.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057205a5abd980f45-00632033af-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.402 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:27.403 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:27.404 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:27.404 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.405 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.405 DEBUG:boto:Method: GET 2022-09-13T07:39:27.406 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.406 DEBUG:boto:Data: 2022-09-13T07:39:27.407 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.407 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.408 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.408 DEBUG:boto:Params: {} 2022-09-13T07:39:27.409 DEBUG:boto:Token: None 2022-09-13T07:39:27.410 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/ 2022-09-13T07:39:27.410 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU= 2022-09-13T07:39:27.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU=', 'Content-Length': '0'} 2022-09-13T07:39:27.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c833c61c5d5a72a-00632033af-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.415 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:27.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:27.416 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.417 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.417 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.418 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.418 DEBUG:boto:Method: GET 2022-09-13T07:39:27.419 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.419 DEBUG:boto:Data: 2022-09-13T07:39:27.420 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.420 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.421 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.421 DEBUG:boto:Params: {} 2022-09-13T07:39:27.422 DEBUG:boto:Token: None 2022-09-13T07:39:27.423 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.423 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2xsw3qyWK57Chk699xaOKAaYnkU= 2022-09-13T07:39:27.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2xsw3qyWK57Chk699xaOKAaYnkU=', 'Content-Length': '0'} 2022-09-13T07:39:27.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe9f1c5985841754-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.427 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.427 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.428 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.428 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.429 DEBUG:boto:Method: GET 2022-09-13T07:39:27.429 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.430 DEBUG:boto:Data: 2022-09-13T07:39:27.430 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.431 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.431 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.432 DEBUG:boto:Params: {} 2022-09-13T07:39:27.432 DEBUG:boto:Token: None 2022-09-13T07:39:27.433 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.433 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2xsw3qyWK57Chk699xaOKAaYnkU= 2022-09-13T07:39:27.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2xsw3qyWK57Chk699xaOKAaYnkU=', 'Content-Length': '0'} 2022-09-13T07:39:27.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5fb1127145cb220-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.436 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.437 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.437 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.438 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.438 DEBUG:boto:Method: HEAD 2022-09-13T07:39:27.439 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.439 DEBUG:boto:Data: 2022-09-13T07:39:27.440 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.440 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.441 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.441 DEBUG:boto:Params: {} 2022-09-13T07:39:27.442 DEBUG:boto:Token: None 2022-09-13T07:39:27.442 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.443 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Mj2UrjgxmrSdw8BMhbewJnvQUuE= 2022-09-13T07:39:27.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Mj2UrjgxmrSdw8BMhbewJnvQUuE=', 'Content-Length': '0'} 2022-09-13T07:39:27.445 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ef70411ba7c728e-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.446 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.447 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.447 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.448 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.448 DEBUG:boto:Method: HEAD 2022-09-13T07:39:27.448 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.449 DEBUG:boto:Data: 2022-09-13T07:39:27.449 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.450 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.450 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.451 DEBUG:boto:Params: {} 2022-09-13T07:39:27.451 DEBUG:boto:Token: None 2022-09-13T07:39:27.452 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.452 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Mj2UrjgxmrSdw8BMhbewJnvQUuE= 2022-09-13T07:39:27.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Mj2UrjgxmrSdw8BMhbewJnvQUuE=', 'Content-Length': '0'} 2022-09-13T07:39:27.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000122c6f93fdf15112-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:27.457 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.457 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.458 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.459 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.459 DEBUG:boto:Method: GET 2022-09-13T07:39:27.460 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.461 DEBUG:boto:Data: 2022-09-13T07:39:27.462 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.463 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.464 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.464 DEBUG:boto:Params: {} 2022-09-13T07:39:27.465 DEBUG:boto:Token: None 2022-09-13T07:39:27.466 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.467 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2xsw3qyWK57Chk699xaOKAaYnkU= 2022-09-13T07:39:27.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2xsw3qyWK57Chk699xaOKAaYnkU=', 'Content-Length': '0'} 2022-09-13T07:39:27.470 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e41a8f9e072f19c8-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.471 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.472 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.472 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.473 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.473 DEBUG:boto:Method: GET 2022-09-13T07:39:27.474 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.475 DEBUG:boto:Data: 2022-09-13T07:39:27.476 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.476 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.477 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.478 DEBUG:boto:Params: {} 2022-09-13T07:39:27.478 DEBUG:boto:Token: None 2022-09-13T07:39:27.479 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:27.480 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2xsw3qyWK57Chk699xaOKAaYnkU= 2022-09-13T07:39:27.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2xsw3qyWK57Chk699xaOKAaYnkU=', 'Content-Length': '0'} 2022-09-13T07:39:27.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3463471df249618-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.484 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:27.484 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:27.485 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.485 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.486 DEBUG:boto:Method: GET 2022-09-13T07:39:27.486 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.487 DEBUG:boto:Data: 2022-09-13T07:39:27.487 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.488 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.488 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.489 DEBUG:boto:Params: {} 2022-09-13T07:39:27.489 DEBUG:boto:Token: None 2022-09-13T07:39:27.490 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/ 2022-09-13T07:39:27.490 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU= 2022-09-13T07:39:27.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU=', 'Content-Length': '0'} 2022-09-13T07:39:27.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd3d3956a80d3983-00632033af-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.534 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:27.535 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:27.536 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:27.536 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.537 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.538 DEBUG:boto:Method: GET 2022-09-13T07:39:27.539 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.540 DEBUG:boto:Data: 2022-09-13T07:39:27.541 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.541 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.542 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.543 DEBUG:boto:Params: {} 2022-09-13T07:39:27.544 DEBUG:boto:Token: None 2022-09-13T07:39:27.545 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/ 2022-09-13T07:39:27.545 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU= 2022-09-13T07:39:27.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU=', 'Content-Length': '0'} 2022-09-13T07:39:27.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db9752b4e913f90d-00632033af-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.550 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:27.550 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.551 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.552 DEBUG:boto:Method: HEAD 2022-09-13T07:39:27.552 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:27.553 DEBUG:boto:Data: 2022-09-13T07:39:27.553 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.554 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.554 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.555 DEBUG:boto:Params: {} 2022-09-13T07:39:27.556 DEBUG:boto:Token: None 2022-09-13T07:39:27.556 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj 2022-09-13T07:39:27.557 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uhck/W3nzw0nEF0+ZmEs/viFZFI= 2022-09-13T07:39:27.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uhck/W3nzw0nEF0+ZmEs/viFZFI=', 'Content-Length': '0'} 2022-09-13T07:39:27.561 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011d4f52521011fa2-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.562 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.562 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.563 DEBUG:boto:Method: HEAD 2022-09-13T07:39:27.563 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:27.564 DEBUG:boto:Data: 2022-09-13T07:39:27.565 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.565 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.566 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.566 DEBUG:boto:Params: {} 2022-09-13T07:39:27.567 DEBUG:boto:Token: None 2022-09-13T07:39:27.567 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj 2022-09-13T07:39:27.568 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uhck/W3nzw0nEF0+ZmEs/viFZFI= 2022-09-13T07:39:27.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uhck/W3nzw0nEF0+ZmEs/viFZFI=', 'Content-Length': '0'} 2022-09-13T07:39:27.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000208f672acb3f367a-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:27.573 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.574 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.574 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.575 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.575 DEBUG:boto:Method: GET 2022-09-13T07:39:27.576 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.577 DEBUG:boto:Data: 2022-09-13T07:39:27.577 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.578 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.579 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.579 DEBUG:boto:Params: {} 2022-09-13T07:39:27.580 DEBUG:boto:Token: None 2022-09-13T07:39:27.581 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.581 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I= 2022-09-13T07:39:27.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I=', 'Content-Length': '0'} 2022-09-13T07:39:27.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fd6b994950b5391-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.585 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.586 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.586 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.587 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.587 DEBUG:boto:Method: GET 2022-09-13T07:39:27.588 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.589 DEBUG:boto:Data: 2022-09-13T07:39:27.589 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.590 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.590 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.591 DEBUG:boto:Params: {} 2022-09-13T07:39:27.591 DEBUG:boto:Token: None 2022-09-13T07:39:27.592 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.593 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I= 2022-09-13T07:39:27.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I=', 'Content-Length': '0'} 2022-09-13T07:39:27.595 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ec09ec214ebdd54-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.638 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:27.639 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:27.640 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.640 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.641 DEBUG:boto:Method: GET 2022-09-13T07:39:27.642 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.643 DEBUG:boto:Data: 2022-09-13T07:39:27.644 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.645 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.646 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.647 DEBUG:boto:Params: {} 2022-09-13T07:39:27.648 DEBUG:boto:Token: None 2022-09-13T07:39:27.649 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/ 2022-09-13T07:39:27.650 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU= 2022-09-13T07:39:27.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU=', 'Content-Length': '0'} 2022-09-13T07:39:27.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd9f694a62f96552-00632033af-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.656 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:27.657 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:27.658 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:27.659 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.660 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.661 DEBUG:boto:Method: GET 2022-09-13T07:39:27.662 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.663 DEBUG:boto:Data: 2022-09-13T07:39:27.663 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.664 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.665 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.666 DEBUG:boto:Params: {} 2022-09-13T07:39:27.667 DEBUG:boto:Token: None 2022-09-13T07:39:27.667 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/ 2022-09-13T07:39:27.668 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU= 2022-09-13T07:39:27.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU=', 'Content-Length': '0'} 2022-09-13T07:39:27.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f36d4b514b2645e3-00632033af-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.673 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:27.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:27.675 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.676 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.676 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.677 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.677 DEBUG:boto:Method: GET 2022-09-13T07:39:27.678 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.678 DEBUG:boto:Data: 2022-09-13T07:39:27.679 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.680 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.680 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.681 DEBUG:boto:Params: {} 2022-09-13T07:39:27.681 DEBUG:boto:Token: None 2022-09-13T07:39:27.682 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.682 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cJKxU9ZVxwWLC0Et+MKNs6eDxgI= 2022-09-13T07:39:27.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cJKxU9ZVxwWLC0Et+MKNs6eDxgI=', 'Content-Length': '0'} 2022-09-13T07:39:27.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb7439fea94a14c6-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.686 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.687 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.687 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.688 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.688 DEBUG:boto:Method: GET 2022-09-13T07:39:27.689 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.689 DEBUG:boto:Data: 2022-09-13T07:39:27.690 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.690 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.691 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.691 DEBUG:boto:Params: {} 2022-09-13T07:39:27.692 DEBUG:boto:Token: None 2022-09-13T07:39:27.692 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.693 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cJKxU9ZVxwWLC0Et+MKNs6eDxgI= 2022-09-13T07:39:27.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cJKxU9ZVxwWLC0Et+MKNs6eDxgI=', 'Content-Length': '0'} 2022-09-13T07:39:27.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024dea9981926770d-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.696 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.697 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.697 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.698 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.698 DEBUG:boto:Method: HEAD 2022-09-13T07:39:27.699 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.699 DEBUG:boto:Data: 2022-09-13T07:39:27.700 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.700 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.701 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.701 DEBUG:boto:Params: {} 2022-09-13T07:39:27.702 DEBUG:boto:Token: None 2022-09-13T07:39:27.702 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.703 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cbqWvt+scD8KKBLDe3nWxNU/2Hs= 2022-09-13T07:39:27.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cbqWvt+scD8KKBLDe3nWxNU/2Hs=', 'Content-Length': '0'} 2022-09-13T07:39:27.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000390f60bddcf3eebf-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.706 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.707 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.707 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.708 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.708 DEBUG:boto:Method: HEAD 2022-09-13T07:39:27.709 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.709 DEBUG:boto:Data: 2022-09-13T07:39:27.709 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.710 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.710 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.711 DEBUG:boto:Params: {} 2022-09-13T07:39:27.711 DEBUG:boto:Token: None 2022-09-13T07:39:27.712 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.713 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cbqWvt+scD8KKBLDe3nWxNU/2Hs= 2022-09-13T07:39:27.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cbqWvt+scD8KKBLDe3nWxNU/2Hs=', 'Content-Length': '0'} 2022-09-13T07:39:27.715 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033049ddf9c1fb8bb-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:27.717 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.717 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.718 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.718 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.719 DEBUG:boto:Method: GET 2022-09-13T07:39:27.719 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.720 DEBUG:boto:Data: 2022-09-13T07:39:27.720 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.721 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.721 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.722 DEBUG:boto:Params: {} 2022-09-13T07:39:27.722 DEBUG:boto:Token: None 2022-09-13T07:39:27.723 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.724 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cJKxU9ZVxwWLC0Et+MKNs6eDxgI= 2022-09-13T07:39:27.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cJKxU9ZVxwWLC0Et+MKNs6eDxgI=', 'Content-Length': '0'} 2022-09-13T07:39:27.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c644b0aef647fb72-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.728 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.728 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.729 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.729 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.730 DEBUG:boto:Method: GET 2022-09-13T07:39:27.730 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.731 DEBUG:boto:Data: 2022-09-13T07:39:27.731 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.732 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.732 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.733 DEBUG:boto:Params: {} 2022-09-13T07:39:27.733 DEBUG:boto:Token: None 2022-09-13T07:39:27.734 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:27.735 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cJKxU9ZVxwWLC0Et+MKNs6eDxgI= 2022-09-13T07:39:27.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cJKxU9ZVxwWLC0Et+MKNs6eDxgI=', 'Content-Length': '0'} 2022-09-13T07:39:27.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000440971917fd994f1-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.738 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:27.738 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:27.739 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.739 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.740 DEBUG:boto:Method: GET 2022-09-13T07:39:27.740 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.741 DEBUG:boto:Data: 2022-09-13T07:39:27.741 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.742 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.743 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.743 DEBUG:boto:Params: {} 2022-09-13T07:39:27.744 DEBUG:boto:Token: None 2022-09-13T07:39:27.745 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/ 2022-09-13T07:39:27.746 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU= 2022-09-13T07:39:27.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU=', 'Content-Length': '0'} 2022-09-13T07:39:27.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e6dcc6e9b5b5cf4-00632033af-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.790 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:27.792 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:27.793 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:27.794 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.795 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.796 DEBUG:boto:Method: GET 2022-09-13T07:39:27.797 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.798 DEBUG:boto:Data: 2022-09-13T07:39:27.798 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.799 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.799 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.800 DEBUG:boto:Params: {} 2022-09-13T07:39:27.800 DEBUG:boto:Token: None 2022-09-13T07:39:27.801 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/ 2022-09-13T07:39:27.802 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU= 2022-09-13T07:39:27.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU=', 'Content-Length': '0'} 2022-09-13T07:39:27.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0777516c375b1e9-00632033af-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.806 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:27.807 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.807 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.808 DEBUG:boto:Method: HEAD 2022-09-13T07:39:27.808 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:27.809 DEBUG:boto:Data: 2022-09-13T07:39:27.809 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.809 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.810 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.810 DEBUG:boto:Params: {} 2022-09-13T07:39:27.811 DEBUG:boto:Token: None 2022-09-13T07:39:27.811 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj 2022-09-13T07:39:27.812 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uhck/W3nzw0nEF0+ZmEs/viFZFI= 2022-09-13T07:39:27.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uhck/W3nzw0nEF0+ZmEs/viFZFI=', 'Content-Length': '0'} 2022-09-13T07:39:27.816 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1c4f16eaa7c18f9-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.817 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.817 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.818 DEBUG:boto:Method: HEAD 2022-09-13T07:39:27.818 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:27.819 DEBUG:boto:Data: 2022-09-13T07:39:27.819 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.820 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.820 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.821 DEBUG:boto:Params: {} 2022-09-13T07:39:27.821 DEBUG:boto:Token: None 2022-09-13T07:39:27.822 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj 2022-09-13T07:39:27.823 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uhck/W3nzw0nEF0+ZmEs/viFZFI= 2022-09-13T07:39:27.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uhck/W3nzw0nEF0+ZmEs/viFZFI=', 'Content-Length': '0'} 2022-09-13T07:39:27.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be7eecae03d044e2-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:27.828 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.828 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.829 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.829 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.830 DEBUG:boto:Method: GET 2022-09-13T07:39:27.830 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.831 DEBUG:boto:Data: 2022-09-13T07:39:27.832 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.832 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.833 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.833 DEBUG:boto:Params: {} 2022-09-13T07:39:27.834 DEBUG:boto:Token: None 2022-09-13T07:39:27.834 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.835 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I= 2022-09-13T07:39:27.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I=', 'Content-Length': '0'} 2022-09-13T07:39:27.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfd8efb6b76365ff-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.838 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.839 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.839 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.840 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.840 DEBUG:boto:Method: GET 2022-09-13T07:39:27.841 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.841 DEBUG:boto:Data: 2022-09-13T07:39:27.842 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.842 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.843 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.843 DEBUG:boto:Params: {} 2022-09-13T07:39:27.844 DEBUG:boto:Token: None 2022-09-13T07:39:27.844 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:27.845 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I= 2022-09-13T07:39:27.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:laUr1fxdsMA+kwpIH2+alTbKy8I=', 'Content-Length': '0'} 2022-09-13T07:39:27.848 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000879b6922b9bb621c-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.894 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:27.896 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:27.897 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.898 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.899 DEBUG:boto:Method: GET 2022-09-13T07:39:27.900 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.901 DEBUG:boto:Data: 2022-09-13T07:39:27.902 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.903 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.904 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.905 DEBUG:boto:Params: {} 2022-09-13T07:39:27.906 DEBUG:boto:Token: None 2022-09-13T07:39:27.907 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/ 2022-09-13T07:39:27.908 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU= 2022-09-13T07:39:27.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU=', 'Content-Length': '0'} 2022-09-13T07:39:27.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fce82e916af5c01c-00632033af-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.914 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:27.915 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:27.916 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:27.917 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.918 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.919 DEBUG:boto:Method: GET 2022-09-13T07:39:27.920 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:27.921 DEBUG:boto:Data: 2022-09-13T07:39:27.922 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.924 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.924 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.925 DEBUG:boto:Params: {} 2022-09-13T07:39:27.927 DEBUG:boto:Token: None 2022-09-13T07:39:27.928 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/ 2022-09-13T07:39:27.929 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU= 2022-09-13T07:39:27.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+iVxm2fghwpCJyh1p13N+VfT7TU=', 'Content-Length': '0'} 2022-09-13T07:39:27.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9825d5b1577fce6-00632033af-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.935 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:27.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:27.937 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.938 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.939 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.940 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.941 DEBUG:boto:Method: GET 2022-09-13T07:39:27.942 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.943 DEBUG:boto:Data: 2022-09-13T07:39:27.944 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.945 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.946 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.947 DEBUG:boto:Params: {} 2022-09-13T07:39:27.948 DEBUG:boto:Token: None 2022-09-13T07:39:27.949 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.950 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:C8YImckkLXSt8TtV0qQpATonfJY= 2022-09-13T07:39:27.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:C8YImckkLXSt8TtV0qQpATonfJY=', 'Content-Length': '0'} 2022-09-13T07:39:27.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071949f466278daa0-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.956 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.956 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.957 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.958 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.958 DEBUG:boto:Method: GET 2022-09-13T07:39:27.959 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.959 DEBUG:boto:Data: 2022-09-13T07:39:27.960 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.961 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.961 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.962 DEBUG:boto:Params: {} 2022-09-13T07:39:27.963 DEBUG:boto:Token: None 2022-09-13T07:39:27.963 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.964 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:C8YImckkLXSt8TtV0qQpATonfJY= 2022-09-13T07:39:27.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:C8YImckkLXSt8TtV0qQpATonfJY=', 'Content-Length': '0'} 2022-09-13T07:39:27.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d069877c3efe74c-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.968 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.969 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.969 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.970 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.971 DEBUG:boto:Method: HEAD 2022-09-13T07:39:27.972 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.973 DEBUG:boto:Data: 2022-09-13T07:39:27.974 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.975 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:27.975 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.976 DEBUG:boto:Params: {} 2022-09-13T07:39:27.985 DEBUG:boto:Token: None 2022-09-13T07:39:27.986 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.986 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ANqR/GDIoQf+OPXFe75HrXFAXzI= 2022-09-13T07:39:27.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ANqR/GDIoQf+OPXFe75HrXFAXzI=', 'Content-Length': '0'} 2022-09-13T07:39:27.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a20fc386e99f0bcc-00632033af-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:27.991 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:27.992 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:27.992 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.993 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.993 DEBUG:boto:Method: HEAD 2022-09-13T07:39:27.994 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.994 DEBUG:boto:Data: 2022-09-13T07:39:27.995 DEBUG:boto:Headers: {} 2022-09-13T07:39:27.995 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:27.996 DEBUG:boto:Port: 8000 2022-09-13T07:39:27.996 DEBUG:boto:Params: {} 2022-09-13T07:39:27.997 DEBUG:boto:Token: None 2022-09-13T07:39:27.998 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:27 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:27.998 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ANqR/GDIoQf+OPXFe75HrXFAXzI= 2022-09-13T07:39:27.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:27 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ANqR/GDIoQf+OPXFe75HrXFAXzI=', 'Content-Length': '0'} 2022-09-13T07:39:28.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0d13e290f4eab97-00632033af-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:28.003 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.003 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.004 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:28.004 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:28.005 DEBUG:boto:Method: GET 2022-09-13T07:39:28.005 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:28.006 DEBUG:boto:Data: 2022-09-13T07:39:28.006 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.007 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.007 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.008 DEBUG:boto:Params: {} 2022-09-13T07:39:28.008 DEBUG:boto:Token: None 2022-09-13T07:39:28.009 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:28.010 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:o1cqnYskdkwF5MWCIVRn0AvnGa8= 2022-09-13T07:39:28.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:o1cqnYskdkwF5MWCIVRn0AvnGa8=', 'Content-Length': '0'} 2022-09-13T07:39:28.013 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e202507103d8cda0-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.058 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.059 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.060 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:28.061 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:28.063 DEBUG:boto:Method: GET 2022-09-13T07:39:28.063 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:28.064 DEBUG:boto:Data: 2022-09-13T07:39:28.065 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.066 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.067 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.068 DEBUG:boto:Params: {} 2022-09-13T07:39:28.068 DEBUG:boto:Token: None 2022-09-13T07:39:28.069 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:28.070 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:o1cqnYskdkwF5MWCIVRn0AvnGa8= 2022-09-13T07:39:28.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:o1cqnYskdkwF5MWCIVRn0AvnGa8=', 'Content-Length': '0'} 2022-09-13T07:39:28.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f44fc5037103d9f-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.074 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.074 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.075 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.075 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.076 DEBUG:boto:Method: GET 2022-09-13T07:39:28.076 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.077 DEBUG:boto:Data: 2022-09-13T07:39:28.077 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.078 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.078 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.079 DEBUG:boto:Params: {} 2022-09-13T07:39:28.080 DEBUG:boto:Token: None 2022-09-13T07:39:28.080 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.081 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050947f86483009a1-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.085 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.086 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.087 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.087 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.088 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.088 DEBUG:boto:Method: GET 2022-09-13T07:39:28.089 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.089 DEBUG:boto:Data: 2022-09-13T07:39:28.090 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.090 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.091 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.091 DEBUG:boto:Params: {} 2022-09-13T07:39:28.092 DEBUG:boto:Token: None 2022-09-13T07:39:28.093 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.093 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c977d4ae8e7c2515-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.098 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.099 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.099 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.100 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.100 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:28.101 DEBUG:boto:Data: 2022-09-13T07:39:28.101 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.102 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.102 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.103 DEBUG:boto:Params: {} 2022-09-13T07:39:28.103 DEBUG:boto:Token: None 2022-09-13T07:39:28.104 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj 2022-09-13T07:39:28.104 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU= 2022-09-13T07:39:28.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU=', 'Content-Length': '0'} 2022-09-13T07:39:28.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000525e2e0980c98654-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.109 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.110 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.110 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.111 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:28.111 DEBUG:boto:Data: 2022-09-13T07:39:28.112 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.112 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.113 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.113 DEBUG:boto:Params: {} 2022-09-13T07:39:28.114 DEBUG:boto:Token: None 2022-09-13T07:39:28.114 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj 2022-09-13T07:39:28.115 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU= 2022-09-13T07:39:28.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU=', 'Content-Length': '0'} 2022-09-13T07:39:28.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b3a3e98ece029de-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:28.120 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.121 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.121 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.122 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.122 DEBUG:boto:Method: GET 2022-09-13T07:39:28.123 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.123 DEBUG:boto:Data: 2022-09-13T07:39:28.124 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.124 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.125 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.125 DEBUG:boto:Params: {} 2022-09-13T07:39:28.126 DEBUG:boto:Token: None 2022-09-13T07:39:28.127 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.127 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8= 2022-09-13T07:39:28.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8=', 'Content-Length': '0'} 2022-09-13T07:39:28.130 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d98c415427fc8f90-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.131 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.131 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.132 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.132 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.133 DEBUG:boto:Method: GET 2022-09-13T07:39:28.134 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.135 DEBUG:boto:Data: 2022-09-13T07:39:28.135 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.136 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.136 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.137 DEBUG:boto:Params: {} 2022-09-13T07:39:28.137 DEBUG:boto:Token: None 2022-09-13T07:39:28.138 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.138 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8= 2022-09-13T07:39:28.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8=', 'Content-Length': '0'} 2022-09-13T07:39:28.141 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006df8939cfd65bb6d-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.182 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.183 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.183 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.184 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.184 DEBUG:boto:Method: GET 2022-09-13T07:39:28.185 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.185 DEBUG:boto:Data: 2022-09-13T07:39:28.186 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.187 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.188 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.188 DEBUG:boto:Params: {} 2022-09-13T07:39:28.189 DEBUG:boto:Token: None 2022-09-13T07:39:28.190 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.190 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a636fff230e189a2-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.195 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.195 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.196 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.196 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.197 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.197 DEBUG:boto:Method: GET 2022-09-13T07:39:28.198 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.199 DEBUG:boto:Data: 2022-09-13T07:39:28.199 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.200 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.200 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.201 DEBUG:boto:Params: {} 2022-09-13T07:39:28.201 DEBUG:boto:Token: None 2022-09-13T07:39:28.202 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.203 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e2a5d65a60bfb24-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.207 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:28.208 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.209 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.209 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.210 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.210 DEBUG:boto:Method: GET 2022-09-13T07:39:28.211 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.211 DEBUG:boto:Data: 2022-09-13T07:39:28.212 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.212 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.213 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.213 DEBUG:boto:Params: {} 2022-09-13T07:39:28.214 DEBUG:boto:Token: None 2022-09-13T07:39:28.214 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.215 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kmzLER7ra2rjrxAfvUOYyIVcDAw= 2022-09-13T07:39:28.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kmzLER7ra2rjrxAfvUOYyIVcDAw=', 'Content-Length': '0'} 2022-09-13T07:39:28.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073cc557d08d16c27-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.266 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.267 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.268 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.268 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.269 DEBUG:boto:Method: GET 2022-09-13T07:39:28.270 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.270 DEBUG:boto:Data: 2022-09-13T07:39:28.271 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.271 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.272 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.272 DEBUG:boto:Params: {} 2022-09-13T07:39:28.273 DEBUG:boto:Token: None 2022-09-13T07:39:28.274 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.274 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kmzLER7ra2rjrxAfvUOYyIVcDAw= 2022-09-13T07:39:28.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kmzLER7ra2rjrxAfvUOYyIVcDAw=', 'Content-Length': '0'} 2022-09-13T07:39:28.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053e2f48d62c0760b-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.278 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.279 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.279 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.280 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.280 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.281 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.281 DEBUG:boto:Data: 2022-09-13T07:39:28.282 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.282 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.283 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.283 DEBUG:boto:Params: {} 2022-09-13T07:39:28.284 DEBUG:boto:Token: None 2022-09-13T07:39:28.285 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.285 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NAydAnWCPW7uwnsMgGj+LezJEZc= 2022-09-13T07:39:28.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NAydAnWCPW7uwnsMgGj+LezJEZc=', 'Content-Length': '0'} 2022-09-13T07:39:28.288 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000280eea8b3657a6c-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.289 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.290 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.290 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.291 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.291 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.292 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.292 DEBUG:boto:Data: 2022-09-13T07:39:28.293 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.294 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.294 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.295 DEBUG:boto:Params: {} 2022-09-13T07:39:28.296 DEBUG:boto:Token: None 2022-09-13T07:39:28.297 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.297 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NAydAnWCPW7uwnsMgGj+LezJEZc= 2022-09-13T07:39:28.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NAydAnWCPW7uwnsMgGj+LezJEZc=', 'Content-Length': '0'} 2022-09-13T07:39:28.300 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1ac2a84f5891ce1-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:28.301 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.302 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.302 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.303 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.303 DEBUG:boto:Method: GET 2022-09-13T07:39:28.304 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.304 DEBUG:boto:Data: 2022-09-13T07:39:28.305 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.305 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.306 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.306 DEBUG:boto:Params: {} 2022-09-13T07:39:28.307 DEBUG:boto:Token: None 2022-09-13T07:39:28.308 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.308 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kmzLER7ra2rjrxAfvUOYyIVcDAw= 2022-09-13T07:39:28.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kmzLER7ra2rjrxAfvUOYyIVcDAw=', 'Content-Length': '0'} 2022-09-13T07:39:28.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080bb074b6acdcc43-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.312 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.312 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.313 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.313 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.314 DEBUG:boto:Method: GET 2022-09-13T07:39:28.314 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.315 DEBUG:boto:Data: 2022-09-13T07:39:28.315 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.316 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.316 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.317 DEBUG:boto:Params: {} 2022-09-13T07:39:28.318 DEBUG:boto:Token: None 2022-09-13T07:39:28.318 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:28.319 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kmzLER7ra2rjrxAfvUOYyIVcDAw= 2022-09-13T07:39:28.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kmzLER7ra2rjrxAfvUOYyIVcDAw=', 'Content-Length': '0'} 2022-09-13T07:39:28.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003143ce7bf043d9ac-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.323 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.323 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.324 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.324 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.325 DEBUG:boto:Method: GET 2022-09-13T07:39:28.325 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.326 DEBUG:boto:Data: 2022-09-13T07:39:28.326 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.327 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.327 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.328 DEBUG:boto:Params: {} 2022-09-13T07:39:28.328 DEBUG:boto:Token: None 2022-09-13T07:39:28.329 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.329 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000251c184b9a8e51af-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.335 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.335 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.336 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.336 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.337 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.338 DEBUG:boto:Method: GET 2022-09-13T07:39:28.338 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.339 DEBUG:boto:Data: 2022-09-13T07:39:28.339 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.340 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.341 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.341 DEBUG:boto:Params: {} 2022-09-13T07:39:28.342 DEBUG:boto:Token: None 2022-09-13T07:39:28.342 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.343 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000772807ac6d813fd4-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.348 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.349 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.350 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.350 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.351 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:28.351 DEBUG:boto:Data: 2022-09-13T07:39:28.352 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.352 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.353 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.353 DEBUG:boto:Params: {} 2022-09-13T07:39:28.354 DEBUG:boto:Token: None 2022-09-13T07:39:28.355 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj 2022-09-13T07:39:28.356 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU= 2022-09-13T07:39:28.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU=', 'Content-Length': '0'} 2022-09-13T07:39:28.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8668489c377b83a-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.363 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.365 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.366 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.366 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:28.367 DEBUG:boto:Data: 2022-09-13T07:39:28.368 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.369 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.370 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.371 DEBUG:boto:Params: {} 2022-09-13T07:39:28.373 DEBUG:boto:Token: None 2022-09-13T07:39:28.374 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj 2022-09-13T07:39:28.375 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU= 2022-09-13T07:39:28.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU=', 'Content-Length': '0'} 2022-09-13T07:39:28.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c9b774bf2fd1105-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:28.383 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.384 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.386 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.387 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.388 DEBUG:boto:Method: GET 2022-09-13T07:39:28.389 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.390 DEBUG:boto:Data: 2022-09-13T07:39:28.391 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.392 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.393 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.394 DEBUG:boto:Params: {} 2022-09-13T07:39:28.395 DEBUG:boto:Token: None 2022-09-13T07:39:28.396 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.397 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8= 2022-09-13T07:39:28.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8=', 'Content-Length': '0'} 2022-09-13T07:39:28.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f99c301513e922f-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.403 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.405 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.406 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.406 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.407 DEBUG:boto:Method: GET 2022-09-13T07:39:28.408 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.409 DEBUG:boto:Data: 2022-09-13T07:39:28.410 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.411 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.412 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.413 DEBUG:boto:Params: {} 2022-09-13T07:39:28.415 DEBUG:boto:Token: None 2022-09-13T07:39:28.416 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.417 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8= 2022-09-13T07:39:28.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8=', 'Content-Length': '0'} 2022-09-13T07:39:28.421 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3713a3a0f89ec23-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.466 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.467 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.468 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.468 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.469 DEBUG:boto:Method: GET 2022-09-13T07:39:28.469 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.470 DEBUG:boto:Data: 2022-09-13T07:39:28.470 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.471 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.472 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.473 DEBUG:boto:Params: {} 2022-09-13T07:39:28.474 DEBUG:boto:Token: None 2022-09-13T07:39:28.475 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.476 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000016ebdcc15228416-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.482 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.483 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.483 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.484 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.484 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.485 DEBUG:boto:Method: GET 2022-09-13T07:39:28.486 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.486 DEBUG:boto:Data: 2022-09-13T07:39:28.487 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.488 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.488 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.489 DEBUG:boto:Params: {} 2022-09-13T07:39:28.489 DEBUG:boto:Token: None 2022-09-13T07:39:28.490 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.490 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5746d0de778254d-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.495 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:28.496 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.497 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.497 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.498 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.499 DEBUG:boto:Method: GET 2022-09-13T07:39:28.499 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.500 DEBUG:boto:Data: 2022-09-13T07:39:28.500 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.501 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.501 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.502 DEBUG:boto:Params: {} 2022-09-13T07:39:28.503 DEBUG:boto:Token: None 2022-09-13T07:39:28.503 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.504 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GNLTm4PTUaqgpGVImHpcqu/E5sg= 2022-09-13T07:39:28.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GNLTm4PTUaqgpGVImHpcqu/E5sg=', 'Content-Length': '0'} 2022-09-13T07:39:28.507 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002432efaf07607e03-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.509 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.509 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.510 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.511 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.511 DEBUG:boto:Method: GET 2022-09-13T07:39:28.512 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.512 DEBUG:boto:Data: 2022-09-13T07:39:28.513 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.514 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.515 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.515 DEBUG:boto:Params: {} 2022-09-13T07:39:28.516 DEBUG:boto:Token: None 2022-09-13T07:39:28.516 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.517 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GNLTm4PTUaqgpGVImHpcqu/E5sg= 2022-09-13T07:39:28.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GNLTm4PTUaqgpGVImHpcqu/E5sg=', 'Content-Length': '0'} 2022-09-13T07:39:28.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000617bd5868c14cfb-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.521 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.521 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.522 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.522 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.523 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.523 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.524 DEBUG:boto:Data: 2022-09-13T07:39:28.524 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.525 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.525 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.526 DEBUG:boto:Params: {} 2022-09-13T07:39:28.527 DEBUG:boto:Token: None 2022-09-13T07:39:28.527 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.528 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6pi1ucMiQPm61n8ln32bPqeQhFY= 2022-09-13T07:39:28.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6pi1ucMiQPm61n8ln32bPqeQhFY=', 'Content-Length': '0'} 2022-09-13T07:39:28.531 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021dccb2cc5cb2664-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.532 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.533 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.533 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.533 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.534 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.535 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.535 DEBUG:boto:Data: 2022-09-13T07:39:28.535 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.536 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.536 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.537 DEBUG:boto:Params: {} 2022-09-13T07:39:28.537 DEBUG:boto:Token: None 2022-09-13T07:39:28.538 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.539 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6pi1ucMiQPm61n8ln32bPqeQhFY= 2022-09-13T07:39:28.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6pi1ucMiQPm61n8ln32bPqeQhFY=', 'Content-Length': '0'} 2022-09-13T07:39:28.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000544c524b6c2f11a-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:28.543 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.543 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.544 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.544 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.545 DEBUG:boto:Method: GET 2022-09-13T07:39:28.546 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.546 DEBUG:boto:Data: 2022-09-13T07:39:28.547 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.547 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.548 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.548 DEBUG:boto:Params: {} 2022-09-13T07:39:28.549 DEBUG:boto:Token: None 2022-09-13T07:39:28.550 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.550 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GNLTm4PTUaqgpGVImHpcqu/E5sg= 2022-09-13T07:39:28.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GNLTm4PTUaqgpGVImHpcqu/E5sg=', 'Content-Length': '0'} 2022-09-13T07:39:28.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b60692417ac09355-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.594 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.595 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.596 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.596 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.597 DEBUG:boto:Method: GET 2022-09-13T07:39:28.597 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.598 DEBUG:boto:Data: 2022-09-13T07:39:28.598 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.599 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.599 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.600 DEBUG:boto:Params: {} 2022-09-13T07:39:28.600 DEBUG:boto:Token: None 2022-09-13T07:39:28.601 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:28.601 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GNLTm4PTUaqgpGVImHpcqu/E5sg= 2022-09-13T07:39:28.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GNLTm4PTUaqgpGVImHpcqu/E5sg=', 'Content-Length': '0'} 2022-09-13T07:39:28.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfbb2081cdbd5125-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.606 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.606 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.607 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.607 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.608 DEBUG:boto:Method: GET 2022-09-13T07:39:28.608 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.609 DEBUG:boto:Data: 2022-09-13T07:39:28.609 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.610 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.610 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.611 DEBUG:boto:Params: {} 2022-09-13T07:39:28.611 DEBUG:boto:Token: None 2022-09-13T07:39:28.612 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.612 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e22fc8fbd5c2931-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.617 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.618 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.619 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.619 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.620 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.620 DEBUG:boto:Method: GET 2022-09-13T07:39:28.620 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.621 DEBUG:boto:Data: 2022-09-13T07:39:28.621 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.622 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.622 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.623 DEBUG:boto:Params: {} 2022-09-13T07:39:28.624 DEBUG:boto:Token: None 2022-09-13T07:39:28.624 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.625 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d53ac7eeb758b80-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.630 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.630 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.631 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.631 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.632 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:28.633 DEBUG:boto:Data: 2022-09-13T07:39:28.633 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.634 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.634 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.635 DEBUG:boto:Params: {} 2022-09-13T07:39:28.635 DEBUG:boto:Token: None 2022-09-13T07:39:28.636 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj 2022-09-13T07:39:28.636 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU= 2022-09-13T07:39:28.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU=', 'Content-Length': '0'} 2022-09-13T07:39:28.641 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000addd2e1470868ef4-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.642 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.643 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.643 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.644 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:28.644 DEBUG:boto:Data: 2022-09-13T07:39:28.645 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.645 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.646 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.647 DEBUG:boto:Params: {} 2022-09-13T07:39:28.647 DEBUG:boto:Token: None 2022-09-13T07:39:28.648 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj 2022-09-13T07:39:28.648 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU= 2022-09-13T07:39:28.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU=', 'Content-Length': '0'} 2022-09-13T07:39:28.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054d5654bb8868995-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:28.654 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.654 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.655 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.656 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.656 DEBUG:boto:Method: GET 2022-09-13T07:39:28.657 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.658 DEBUG:boto:Data: 2022-09-13T07:39:28.659 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.659 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.660 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.660 DEBUG:boto:Params: {} 2022-09-13T07:39:28.661 DEBUG:boto:Token: None 2022-09-13T07:39:28.661 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.662 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8= 2022-09-13T07:39:28.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8=', 'Content-Length': '0'} 2022-09-13T07:39:28.664 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3ab36c9ec8aae68-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.666 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.666 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.667 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.668 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.668 DEBUG:boto:Method: GET 2022-09-13T07:39:28.669 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.669 DEBUG:boto:Data: 2022-09-13T07:39:28.670 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.670 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.671 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.671 DEBUG:boto:Params: {} 2022-09-13T07:39:28.672 DEBUG:boto:Token: None 2022-09-13T07:39:28.672 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.673 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8= 2022-09-13T07:39:28.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8=', 'Content-Length': '0'} 2022-09-13T07:39:28.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c24bb6ece890e23-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.677 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.677 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.678 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.678 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.679 DEBUG:boto:Method: GET 2022-09-13T07:39:28.679 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.680 DEBUG:boto:Data: 2022-09-13T07:39:28.680 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.681 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.681 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.682 DEBUG:boto:Params: {} 2022-09-13T07:39:28.682 DEBUG:boto:Token: None 2022-09-13T07:39:28.683 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.684 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fff5ca8843b43da4-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.689 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.690 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.690 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.691 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.691 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.692 DEBUG:boto:Method: GET 2022-09-13T07:39:28.692 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.693 DEBUG:boto:Data: 2022-09-13T07:39:28.693 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.694 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.694 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.695 DEBUG:boto:Params: {} 2022-09-13T07:39:28.696 DEBUG:boto:Token: None 2022-09-13T07:39:28.696 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.697 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cae97d3da2980ce6-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.701 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:28.702 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.703 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.703 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.704 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.704 DEBUG:boto:Method: GET 2022-09-13T07:39:28.705 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.705 DEBUG:boto:Data: 2022-09-13T07:39:28.706 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.706 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.707 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.707 DEBUG:boto:Params: {} 2022-09-13T07:39:28.708 DEBUG:boto:Token: None 2022-09-13T07:39:28.708 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.709 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZYJI03+yX2Ql/zJPMLxbhkzqLL0= 2022-09-13T07:39:28.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZYJI03+yX2Ql/zJPMLxbhkzqLL0=', 'Content-Length': '0'} 2022-09-13T07:39:28.712 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0cef2d36216dc20-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.754 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.755 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.755 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.756 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.756 DEBUG:boto:Method: GET 2022-09-13T07:39:28.757 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.758 DEBUG:boto:Data: 2022-09-13T07:39:28.758 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.759 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.759 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.760 DEBUG:boto:Params: {} 2022-09-13T07:39:28.761 DEBUG:boto:Token: None 2022-09-13T07:39:28.761 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.762 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZYJI03+yX2Ql/zJPMLxbhkzqLL0= 2022-09-13T07:39:28.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZYJI03+yX2Ql/zJPMLxbhkzqLL0=', 'Content-Length': '0'} 2022-09-13T07:39:28.765 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e33e614a9bf5cd4-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.766 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.767 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.767 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.768 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.768 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.769 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.769 DEBUG:boto:Data: 2022-09-13T07:39:28.770 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.770 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.771 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.772 DEBUG:boto:Params: {} 2022-09-13T07:39:28.773 DEBUG:boto:Token: None 2022-09-13T07:39:28.773 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.774 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:meFapAs2bQHDpBPYeLqyeFUN35I= 2022-09-13T07:39:28.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:meFapAs2bQHDpBPYeLqyeFUN35I=', 'Content-Length': '0'} 2022-09-13T07:39:28.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fb98ca11e9bba90-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.778 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.778 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.779 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.779 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.780 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.780 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.781 DEBUG:boto:Data: 2022-09-13T07:39:28.781 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.782 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.782 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.783 DEBUG:boto:Params: {} 2022-09-13T07:39:28.784 DEBUG:boto:Token: None 2022-09-13T07:39:28.784 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.785 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:meFapAs2bQHDpBPYeLqyeFUN35I= 2022-09-13T07:39:28.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:meFapAs2bQHDpBPYeLqyeFUN35I=', 'Content-Length': '0'} 2022-09-13T07:39:28.789 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f106a37e11bcae3-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:28.790 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.790 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.791 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.791 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.792 DEBUG:boto:Method: GET 2022-09-13T07:39:28.793 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.793 DEBUG:boto:Data: 2022-09-13T07:39:28.794 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.794 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.795 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.795 DEBUG:boto:Params: {} 2022-09-13T07:39:28.796 DEBUG:boto:Token: None 2022-09-13T07:39:28.797 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.798 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZYJI03+yX2Ql/zJPMLxbhkzqLL0= 2022-09-13T07:39:28.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZYJI03+yX2Ql/zJPMLxbhkzqLL0=', 'Content-Length': '0'} 2022-09-13T07:39:28.800 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6f791e13339aa9d-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.801 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.802 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.802 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.803 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.803 DEBUG:boto:Method: GET 2022-09-13T07:39:28.804 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.805 DEBUG:boto:Data: 2022-09-13T07:39:28.805 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.806 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.806 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.807 DEBUG:boto:Params: {} 2022-09-13T07:39:28.807 DEBUG:boto:Token: None 2022-09-13T07:39:28.808 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:28.809 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZYJI03+yX2Ql/zJPMLxbhkzqLL0= 2022-09-13T07:39:28.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZYJI03+yX2Ql/zJPMLxbhkzqLL0=', 'Content-Length': '0'} 2022-09-13T07:39:28.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006cbdf01ea0c8bc19-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.812 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.813 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.814 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.814 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.815 DEBUG:boto:Method: GET 2022-09-13T07:39:28.816 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.816 DEBUG:boto:Data: 2022-09-13T07:39:28.817 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.817 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.818 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.818 DEBUG:boto:Params: {} 2022-09-13T07:39:28.819 DEBUG:boto:Token: None 2022-09-13T07:39:28.820 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.820 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085b2d220a739b393-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.825 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.826 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.826 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.827 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.827 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.828 DEBUG:boto:Method: GET 2022-09-13T07:39:28.828 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.829 DEBUG:boto:Data: 2022-09-13T07:39:28.829 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.830 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.831 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.831 DEBUG:boto:Params: {} 2022-09-13T07:39:28.832 DEBUG:boto:Token: None 2022-09-13T07:39:28.833 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.833 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000802105e4ef91f084-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.838 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.838 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.839 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.840 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.840 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:28.841 DEBUG:boto:Data: 2022-09-13T07:39:28.841 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.842 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.842 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.843 DEBUG:boto:Params: {} 2022-09-13T07:39:28.843 DEBUG:boto:Token: None 2022-09-13T07:39:28.844 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj 2022-09-13T07:39:28.844 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU= 2022-09-13T07:39:28.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU=', 'Content-Length': '0'} 2022-09-13T07:39:28.848 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001789c7a24efec6c-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.849 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.850 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.850 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.851 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:28.851 DEBUG:boto:Data: 2022-09-13T07:39:28.852 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.852 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.853 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.853 DEBUG:boto:Params: {} 2022-09-13T07:39:28.854 DEBUG:boto:Token: None 2022-09-13T07:39:28.855 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj 2022-09-13T07:39:28.855 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU= 2022-09-13T07:39:28.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wHRkv4g/UYTt3G9CqqCeUxwwTpU=', 'Content-Length': '0'} 2022-09-13T07:39:28.859 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9d3b1e76b3d9ece-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:28.861 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.861 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.862 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.862 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.863 DEBUG:boto:Method: GET 2022-09-13T07:39:28.863 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.864 DEBUG:boto:Data: 2022-09-13T07:39:28.865 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.865 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.866 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.866 DEBUG:boto:Params: {} 2022-09-13T07:39:28.867 DEBUG:boto:Token: None 2022-09-13T07:39:28.868 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.868 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8= 2022-09-13T07:39:28.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8=', 'Content-Length': '0'} 2022-09-13T07:39:28.871 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a07f364f7da5cba-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.914 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.915 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.916 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.916 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.917 DEBUG:boto:Method: GET 2022-09-13T07:39:28.917 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.918 DEBUG:boto:Data: 2022-09-13T07:39:28.918 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.919 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.920 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.920 DEBUG:boto:Params: {} 2022-09-13T07:39:28.921 DEBUG:boto:Token: None 2022-09-13T07:39:28.921 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:28.922 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8= 2022-09-13T07:39:28.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pko32HdVqxeCAK0FNOfjPaZi8F8=', 'Content-Length': '0'} 2022-09-13T07:39:28.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9bbdab171396b39-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.926 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.927 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.927 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.928 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.928 DEBUG:boto:Method: GET 2022-09-13T07:39:28.929 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.929 DEBUG:boto:Data: 2022-09-13T07:39:28.930 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.930 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.931 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.931 DEBUG:boto:Params: {} 2022-09-13T07:39:28.932 DEBUG:boto:Token: None 2022-09-13T07:39:28.933 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.933 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008caad7e115dce5de-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.938 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.938 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:28.939 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:28.940 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.940 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.941 DEBUG:boto:Method: GET 2022-09-13T07:39:28.941 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:28.942 DEBUG:boto:Data: 2022-09-13T07:39:28.942 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.943 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.943 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.944 DEBUG:boto:Params: {} 2022-09-13T07:39:28.945 DEBUG:boto:Token: None 2022-09-13T07:39:28.945 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/ 2022-09-13T07:39:28.946 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s= 2022-09-13T07:39:28.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VC1LTc91Rmyn2UjaDkxc1jtVJ1s=', 'Content-Length': '0'} 2022-09-13T07:39:28.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e88cc2914060a88-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.950 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:28.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:28.952 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.952 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.953 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.953 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.954 DEBUG:boto:Method: GET 2022-09-13T07:39:28.954 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.955 DEBUG:boto:Data: 2022-09-13T07:39:28.956 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.956 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.957 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.957 DEBUG:boto:Params: {} 2022-09-13T07:39:28.958 DEBUG:boto:Token: None 2022-09-13T07:39:28.958 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.959 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GKYoHuitE/MASut9RC4bM3kqKp0= 2022-09-13T07:39:28.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GKYoHuitE/MASut9RC4bM3kqKp0=', 'Content-Length': '0'} 2022-09-13T07:39:28.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002286bd59edd82f14-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.963 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.963 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.964 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.964 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.965 DEBUG:boto:Method: GET 2022-09-13T07:39:28.965 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.966 DEBUG:boto:Data: 2022-09-13T07:39:28.966 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.967 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.967 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.968 DEBUG:boto:Params: {} 2022-09-13T07:39:28.969 DEBUG:boto:Token: None 2022-09-13T07:39:28.969 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.970 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GKYoHuitE/MASut9RC4bM3kqKp0= 2022-09-13T07:39:28.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GKYoHuitE/MASut9RC4bM3kqKp0=', 'Content-Length': '0'} 2022-09-13T07:39:28.973 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091a59437d4b7e120-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.973 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.974 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.974 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.975 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.975 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.976 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.976 DEBUG:boto:Data: 2022-09-13T07:39:28.977 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.977 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.978 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.978 DEBUG:boto:Params: {} 2022-09-13T07:39:28.979 DEBUG:boto:Token: None 2022-09-13T07:39:28.979 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.980 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cf1IUyHzVZk5N8/tEndQTBdDm5I= 2022-09-13T07:39:28.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cf1IUyHzVZk5N8/tEndQTBdDm5I=', 'Content-Length': '0'} 2022-09-13T07:39:28.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a894bab8d6a5798-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.983 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.984 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.984 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.985 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.985 DEBUG:boto:Method: HEAD 2022-09-13T07:39:28.986 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.986 DEBUG:boto:Data: 2022-09-13T07:39:28.987 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.987 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:28.988 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.988 DEBUG:boto:Params: {} 2022-09-13T07:39:28.989 DEBUG:boto:Token: None 2022-09-13T07:39:28.989 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:28 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.990 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cf1IUyHzVZk5N8/tEndQTBdDm5I= 2022-09-13T07:39:28.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cf1IUyHzVZk5N8/tEndQTBdDm5I=', 'Content-Length': '0'} 2022-09-13T07:39:28.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa863e347ca594c7-00632033b0-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:28.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:28.994 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:28.994 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:28.995 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.995 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.996 DEBUG:boto:Method: GET 2022-09-13T07:39:28.996 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:28.997 DEBUG:boto:Data: 2022-09-13T07:39:28.997 DEBUG:boto:Headers: {} 2022-09-13T07:39:28.998 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:28.998 DEBUG:boto:Port: 8000 2022-09-13T07:39:28.999 DEBUG:boto:Params: {} 2022-09-13T07:39:28.999 DEBUG:boto:Token: None 2022-09-13T07:39:29.000 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:29.000 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:sO8SjWQLSRpHJd75r7R8sR9z4oU= 2022-09-13T07:39:29.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:sO8SjWQLSRpHJd75r7R8sR9z4oU=', 'Content-Length': '0'} 2022-09-13T07:39:29.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007013f4ac1bfa78ec-00632033b0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.046 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.047 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.047 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:29.048 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:29.048 DEBUG:boto:Method: GET 2022-09-13T07:39:29.049 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:29.049 DEBUG:boto:Data: 2022-09-13T07:39:29.050 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.050 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.051 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.052 DEBUG:boto:Params: {} 2022-09-13T07:39:29.053 DEBUG:boto:Token: None 2022-09-13T07:39:29.053 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:29.054 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:sO8SjWQLSRpHJd75r7R8sR9z4oU= 2022-09-13T07:39:29.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:sO8SjWQLSRpHJd75r7R8sR9z4oU=', 'Content-Length': '0'} 2022-09-13T07:39:29.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eae375ef4b1d2727-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.058 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.058 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.059 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.059 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.060 DEBUG:boto:Method: GET 2022-09-13T07:39:29.060 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.061 DEBUG:boto:Data: 2022-09-13T07:39:29.062 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.062 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.063 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.063 DEBUG:boto:Params: {} 2022-09-13T07:39:29.064 DEBUG:boto:Token: None 2022-09-13T07:39:29.064 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.065 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010c0efaf926c6a10-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.069 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.071 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.071 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.072 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.072 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.073 DEBUG:boto:Method: GET 2022-09-13T07:39:29.073 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.074 DEBUG:boto:Data: 2022-09-13T07:39:29.074 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.075 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.075 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.076 DEBUG:boto:Params: {} 2022-09-13T07:39:29.076 DEBUG:boto:Token: None 2022-09-13T07:39:29.077 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.078 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c23db03f370886f0-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.083 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.083 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.084 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.084 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.085 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:29.085 DEBUG:boto:Data: 2022-09-13T07:39:29.086 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.086 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.087 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.087 DEBUG:boto:Params: {} 2022-09-13T07:39:29.088 DEBUG:boto:Token: None 2022-09-13T07:39:29.088 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj 2022-09-13T07:39:29.089 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo= 2022-09-13T07:39:29.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo=', 'Content-Length': '0'} 2022-09-13T07:39:29.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1e60e1d25bbaf0c-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.094 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.094 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.095 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.095 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:29.096 DEBUG:boto:Data: 2022-09-13T07:39:29.097 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.097 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.098 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.098 DEBUG:boto:Params: {} 2022-09-13T07:39:29.099 DEBUG:boto:Token: None 2022-09-13T07:39:29.099 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj 2022-09-13T07:39:29.100 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo= 2022-09-13T07:39:29.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo=', 'Content-Length': '0'} 2022-09-13T07:39:29.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073ec16e751f8d491-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:29.105 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.105 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.106 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.106 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.107 DEBUG:boto:Method: GET 2022-09-13T07:39:29.107 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.108 DEBUG:boto:Data: 2022-09-13T07:39:29.108 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.109 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.109 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.110 DEBUG:boto:Params: {} 2022-09-13T07:39:29.111 DEBUG:boto:Token: None 2022-09-13T07:39:29.111 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.112 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU= 2022-09-13T07:39:29.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU=', 'Content-Length': '0'} 2022-09-13T07:39:29.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cfff1d5babc7732-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.115 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.116 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.116 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.117 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.117 DEBUG:boto:Method: GET 2022-09-13T07:39:29.118 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.118 DEBUG:boto:Data: 2022-09-13T07:39:29.119 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.119 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.120 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.120 DEBUG:boto:Params: {} 2022-09-13T07:39:29.121 DEBUG:boto:Token: None 2022-09-13T07:39:29.121 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.122 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU= 2022-09-13T07:39:29.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU=', 'Content-Length': '0'} 2022-09-13T07:39:29.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000598f205320eb7519-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.167 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.168 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.168 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.169 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.169 DEBUG:boto:Method: GET 2022-09-13T07:39:29.170 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.170 DEBUG:boto:Data: 2022-09-13T07:39:29.171 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.171 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.172 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.172 DEBUG:boto:Params: {} 2022-09-13T07:39:29.173 DEBUG:boto:Token: None 2022-09-13T07:39:29.174 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.174 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000412d8499a2042a52-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.179 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.179 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.180 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.181 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.181 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.182 DEBUG:boto:Method: GET 2022-09-13T07:39:29.182 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.183 DEBUG:boto:Data: 2022-09-13T07:39:29.183 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.184 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.184 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.185 DEBUG:boto:Params: {} 2022-09-13T07:39:29.185 DEBUG:boto:Token: None 2022-09-13T07:39:29.186 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.187 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000940dd44cdfa528aa-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.191 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:29.192 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.193 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.194 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.194 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.195 DEBUG:boto:Method: GET 2022-09-13T07:39:29.195 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.196 DEBUG:boto:Data: 2022-09-13T07:39:29.196 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.197 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.197 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.198 DEBUG:boto:Params: {} 2022-09-13T07:39:29.199 DEBUG:boto:Token: None 2022-09-13T07:39:29.199 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.200 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WiemANt28EXGXk6GH0xhWMaJ8go= 2022-09-13T07:39:29.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WiemANt28EXGXk6GH0xhWMaJ8go=', 'Content-Length': '0'} 2022-09-13T07:39:29.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004e2c0f9724bbb48-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.203 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.204 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.205 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.205 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.206 DEBUG:boto:Method: GET 2022-09-13T07:39:29.206 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.207 DEBUG:boto:Data: 2022-09-13T07:39:29.207 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.208 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.209 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.209 DEBUG:boto:Params: {} 2022-09-13T07:39:29.210 DEBUG:boto:Token: None 2022-09-13T07:39:29.210 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.211 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WiemANt28EXGXk6GH0xhWMaJ8go= 2022-09-13T07:39:29.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WiemANt28EXGXk6GH0xhWMaJ8go=', 'Content-Length': '0'} 2022-09-13T07:39:29.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3ba9ebc332fc1a8-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.215 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.215 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.216 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.216 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.217 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.218 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.218 DEBUG:boto:Data: 2022-09-13T07:39:29.219 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.219 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.220 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.220 DEBUG:boto:Params: {} 2022-09-13T07:39:29.221 DEBUG:boto:Token: None 2022-09-13T07:39:29.221 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.222 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IuDyCx0pzvWKkO7nb9rcJZ3wHUA= 2022-09-13T07:39:29.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IuDyCx0pzvWKkO7nb9rcJZ3wHUA=', 'Content-Length': '0'} 2022-09-13T07:39:29.226 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c4e0cddff25a071-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.227 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.228 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.228 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.229 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.229 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.230 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.230 DEBUG:boto:Data: 2022-09-13T07:39:29.231 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.231 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.232 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.232 DEBUG:boto:Params: {} 2022-09-13T07:39:29.233 DEBUG:boto:Token: None 2022-09-13T07:39:29.233 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.234 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IuDyCx0pzvWKkO7nb9rcJZ3wHUA= 2022-09-13T07:39:29.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IuDyCx0pzvWKkO7nb9rcJZ3wHUA=', 'Content-Length': '0'} 2022-09-13T07:39:29.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069c12571e7ffddac-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:29.238 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.239 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.239 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.240 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.240 DEBUG:boto:Method: GET 2022-09-13T07:39:29.241 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.241 DEBUG:boto:Data: 2022-09-13T07:39:29.242 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.242 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.243 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.243 DEBUG:boto:Params: {} 2022-09-13T07:39:29.244 DEBUG:boto:Token: None 2022-09-13T07:39:29.244 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.245 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WiemANt28EXGXk6GH0xhWMaJ8go= 2022-09-13T07:39:29.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WiemANt28EXGXk6GH0xhWMaJ8go=', 'Content-Length': '0'} 2022-09-13T07:39:29.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a5507d63f987b2b-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.290 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.291 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.291 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.292 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.292 DEBUG:boto:Method: GET 2022-09-13T07:39:29.293 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.293 DEBUG:boto:Data: 2022-09-13T07:39:29.294 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.294 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.295 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.296 DEBUG:boto:Params: {} 2022-09-13T07:39:29.296 DEBUG:boto:Token: None 2022-09-13T07:39:29.297 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:29.297 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WiemANt28EXGXk6GH0xhWMaJ8go= 2022-09-13T07:39:29.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WiemANt28EXGXk6GH0xhWMaJ8go=', 'Content-Length': '0'} 2022-09-13T07:39:29.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a3e2d39471d89a1-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.301 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.302 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.302 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.303 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.304 DEBUG:boto:Method: GET 2022-09-13T07:39:29.305 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.305 DEBUG:boto:Data: 2022-09-13T07:39:29.306 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.306 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.307 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.307 DEBUG:boto:Params: {} 2022-09-13T07:39:29.308 DEBUG:boto:Token: None 2022-09-13T07:39:29.309 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.309 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de0b8cc798647e10-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.316 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.317 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.317 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.319 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.319 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.320 DEBUG:boto:Method: GET 2022-09-13T07:39:29.321 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.322 DEBUG:boto:Data: 2022-09-13T07:39:29.322 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.323 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.323 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.324 DEBUG:boto:Params: {} 2022-09-13T07:39:29.325 DEBUG:boto:Token: None 2022-09-13T07:39:29.326 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.326 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006acd8066081d0547-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.333 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.335 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.336 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.336 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.337 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:29.338 DEBUG:boto:Data: 2022-09-13T07:39:29.338 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.339 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.339 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.340 DEBUG:boto:Params: {} 2022-09-13T07:39:29.340 DEBUG:boto:Token: None 2022-09-13T07:39:29.341 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj 2022-09-13T07:39:29.342 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo= 2022-09-13T07:39:29.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo=', 'Content-Length': '0'} 2022-09-13T07:39:29.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001eb58431431debf5-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.347 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.347 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.348 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.348 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:29.349 DEBUG:boto:Data: 2022-09-13T07:39:29.349 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.350 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.350 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.351 DEBUG:boto:Params: {} 2022-09-13T07:39:29.352 DEBUG:boto:Token: None 2022-09-13T07:39:29.352 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj 2022-09-13T07:39:29.353 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo= 2022-09-13T07:39:29.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo=', 'Content-Length': '0'} 2022-09-13T07:39:29.358 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000321ae3d43e5c4f9e-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:29.359 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.360 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.360 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.361 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.362 DEBUG:boto:Method: GET 2022-09-13T07:39:29.362 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.363 DEBUG:boto:Data: 2022-09-13T07:39:29.364 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.364 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.365 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.365 DEBUG:boto:Params: {} 2022-09-13T07:39:29.366 DEBUG:boto:Token: None 2022-09-13T07:39:29.366 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.367 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU= 2022-09-13T07:39:29.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU=', 'Content-Length': '0'} 2022-09-13T07:39:29.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d146a5bb253e0034-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.372 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.373 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.373 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.374 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.374 DEBUG:boto:Method: GET 2022-09-13T07:39:29.375 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.375 DEBUG:boto:Data: 2022-09-13T07:39:29.376 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.377 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.377 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.378 DEBUG:boto:Params: {} 2022-09-13T07:39:29.378 DEBUG:boto:Token: None 2022-09-13T07:39:29.379 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.380 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU= 2022-09-13T07:39:29.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU=', 'Content-Length': '0'} 2022-09-13T07:39:29.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030c4254377c50554-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.383 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.384 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.384 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.385 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.385 DEBUG:boto:Method: GET 2022-09-13T07:39:29.386 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.387 DEBUG:boto:Data: 2022-09-13T07:39:29.387 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.388 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.388 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.389 DEBUG:boto:Params: {} 2022-09-13T07:39:29.389 DEBUG:boto:Token: None 2022-09-13T07:39:29.390 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.391 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d95297939573ba8a-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.438 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.439 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.440 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.440 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.441 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.441 DEBUG:boto:Method: GET 2022-09-13T07:39:29.442 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.442 DEBUG:boto:Data: 2022-09-13T07:39:29.443 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.443 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.444 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.445 DEBUG:boto:Params: {} 2022-09-13T07:39:29.445 DEBUG:boto:Token: None 2022-09-13T07:39:29.446 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.447 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eabd14be1c48fbd6-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.453 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:29.455 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.455 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.456 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.456 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.457 DEBUG:boto:Method: GET 2022-09-13T07:39:29.457 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.458 DEBUG:boto:Data: 2022-09-13T07:39:29.458 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.459 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.459 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.460 DEBUG:boto:Params: {} 2022-09-13T07:39:29.460 DEBUG:boto:Token: None 2022-09-13T07:39:29.461 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.461 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Eqqzs2iIGbGfM177Jc+AMQP/pDc= 2022-09-13T07:39:29.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Eqqzs2iIGbGfM177Jc+AMQP/pDc=', 'Content-Length': '0'} 2022-09-13T07:39:29.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f80f200c71c979b1-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.465 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.466 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.466 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.467 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.467 DEBUG:boto:Method: GET 2022-09-13T07:39:29.468 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.468 DEBUG:boto:Data: 2022-09-13T07:39:29.469 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.469 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.470 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.470 DEBUG:boto:Params: {} 2022-09-13T07:39:29.471 DEBUG:boto:Token: None 2022-09-13T07:39:29.472 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.473 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Eqqzs2iIGbGfM177Jc+AMQP/pDc= 2022-09-13T07:39:29.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Eqqzs2iIGbGfM177Jc+AMQP/pDc=', 'Content-Length': '0'} 2022-09-13T07:39:29.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000402dd85502f54fff-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.478 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.478 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.479 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.479 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.480 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.480 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.481 DEBUG:boto:Data: 2022-09-13T07:39:29.481 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.482 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.483 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.483 DEBUG:boto:Params: {} 2022-09-13T07:39:29.484 DEBUG:boto:Token: None 2022-09-13T07:39:29.485 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.486 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DVETmY3ni+vHizM8+1a3N1K54Aw= 2022-09-13T07:39:29.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DVETmY3ni+vHizM8+1a3N1K54Aw=', 'Content-Length': '0'} 2022-09-13T07:39:29.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a7a41b196f624cc-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.490 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.491 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.492 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.493 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.494 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.495 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.496 DEBUG:boto:Data: 2022-09-13T07:39:29.496 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.497 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.497 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.498 DEBUG:boto:Params: {} 2022-09-13T07:39:29.498 DEBUG:boto:Token: None 2022-09-13T07:39:29.499 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.499 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DVETmY3ni+vHizM8+1a3N1K54Aw= 2022-09-13T07:39:29.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DVETmY3ni+vHizM8+1a3N1K54Aw=', 'Content-Length': '0'} 2022-09-13T07:39:29.502 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bd0ac8072984ee2-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:29.503 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.503 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.504 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.504 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.505 DEBUG:boto:Method: GET 2022-09-13T07:39:29.505 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.506 DEBUG:boto:Data: 2022-09-13T07:39:29.506 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.507 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.507 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.508 DEBUG:boto:Params: {} 2022-09-13T07:39:29.509 DEBUG:boto:Token: None 2022-09-13T07:39:29.509 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.510 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Eqqzs2iIGbGfM177Jc+AMQP/pDc= 2022-09-13T07:39:29.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Eqqzs2iIGbGfM177Jc+AMQP/pDc=', 'Content-Length': '0'} 2022-09-13T07:39:29.512 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009691993987811d1b-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.514 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.514 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.515 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.516 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.517 DEBUG:boto:Method: GET 2022-09-13T07:39:29.517 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.518 DEBUG:boto:Data: 2022-09-13T07:39:29.518 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.519 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.519 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.520 DEBUG:boto:Params: {} 2022-09-13T07:39:29.520 DEBUG:boto:Token: None 2022-09-13T07:39:29.521 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:29.522 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Eqqzs2iIGbGfM177Jc+AMQP/pDc= 2022-09-13T07:39:29.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Eqqzs2iIGbGfM177Jc+AMQP/pDc=', 'Content-Length': '0'} 2022-09-13T07:39:29.525 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069c0c955b8c2a7ee-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.525 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.526 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.526 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.527 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.527 DEBUG:boto:Method: GET 2022-09-13T07:39:29.528 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.528 DEBUG:boto:Data: 2022-09-13T07:39:29.529 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.530 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.531 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.531 DEBUG:boto:Params: {} 2022-09-13T07:39:29.532 DEBUG:boto:Token: None 2022-09-13T07:39:29.532 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.533 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000808ce8c343d960c4-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.537 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.538 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.539 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.539 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.540 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.541 DEBUG:boto:Method: GET 2022-09-13T07:39:29.542 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.543 DEBUG:boto:Data: 2022-09-13T07:39:29.544 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.544 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.545 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.546 DEBUG:boto:Params: {} 2022-09-13T07:39:29.547 DEBUG:boto:Token: None 2022-09-13T07:39:29.548 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.549 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b512ed7d8702a5b3-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.555 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.556 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.557 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.558 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.559 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:29.559 DEBUG:boto:Data: 2022-09-13T07:39:29.560 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.561 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.562 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.563 DEBUG:boto:Params: {} 2022-09-13T07:39:29.564 DEBUG:boto:Token: None 2022-09-13T07:39:29.565 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj 2022-09-13T07:39:29.566 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo= 2022-09-13T07:39:29.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo=', 'Content-Length': '0'} 2022-09-13T07:39:29.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ee91b5998620be8-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.572 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.572 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.573 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.573 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:29.574 DEBUG:boto:Data: 2022-09-13T07:39:29.574 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.574 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.575 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.575 DEBUG:boto:Params: {} 2022-09-13T07:39:29.576 DEBUG:boto:Token: None 2022-09-13T07:39:29.576 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj 2022-09-13T07:39:29.577 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo= 2022-09-13T07:39:29.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo=', 'Content-Length': '0'} 2022-09-13T07:39:29.581 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006023b1b20e411e4f-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:29.582 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.583 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.583 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.584 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.584 DEBUG:boto:Method: GET 2022-09-13T07:39:29.585 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.585 DEBUG:boto:Data: 2022-09-13T07:39:29.586 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.586 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.587 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.587 DEBUG:boto:Params: {} 2022-09-13T07:39:29.588 DEBUG:boto:Token: None 2022-09-13T07:39:29.589 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.589 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU= 2022-09-13T07:39:29.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU=', 'Content-Length': '0'} 2022-09-13T07:39:29.592 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d539df82401dce6d-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.593 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.593 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.594 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.594 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.595 DEBUG:boto:Method: GET 2022-09-13T07:39:29.595 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.596 DEBUG:boto:Data: 2022-09-13T07:39:29.596 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.597 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.597 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.598 DEBUG:boto:Params: {} 2022-09-13T07:39:29.598 DEBUG:boto:Token: None 2022-09-13T07:39:29.599 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.600 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU= 2022-09-13T07:39:29.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU=', 'Content-Length': '0'} 2022-09-13T07:39:29.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030ffb616041899ea-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.604 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.605 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.606 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.607 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.608 DEBUG:boto:Method: GET 2022-09-13T07:39:29.609 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.610 DEBUG:boto:Data: 2022-09-13T07:39:29.611 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.612 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.613 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.614 DEBUG:boto:Params: {} 2022-09-13T07:39:29.615 DEBUG:boto:Token: None 2022-09-13T07:39:29.616 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.617 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c29e4d034344b50-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.623 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.624 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.625 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.625 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.626 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.627 DEBUG:boto:Method: GET 2022-09-13T07:39:29.628 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.629 DEBUG:boto:Data: 2022-09-13T07:39:29.630 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.630 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.631 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.632 DEBUG:boto:Params: {} 2022-09-13T07:39:29.633 DEBUG:boto:Token: None 2022-09-13T07:39:29.634 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.635 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6e97ce19a68ea9d-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.642 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:29.644 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.645 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.645 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.646 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.647 DEBUG:boto:Method: GET 2022-09-13T07:39:29.647 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.648 DEBUG:boto:Data: 2022-09-13T07:39:29.649 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.649 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.650 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.650 DEBUG:boto:Params: {} 2022-09-13T07:39:29.651 DEBUG:boto:Token: None 2022-09-13T07:39:29.651 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.652 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O9agk0+7KZH4FhyubV2/kCW3Jss= 2022-09-13T07:39:29.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O9agk0+7KZH4FhyubV2/kCW3Jss=', 'Content-Length': '0'} 2022-09-13T07:39:29.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006933ae6d5b34284e-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.655 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.656 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.656 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.657 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.657 DEBUG:boto:Method: GET 2022-09-13T07:39:29.658 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.659 DEBUG:boto:Data: 2022-09-13T07:39:29.659 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.660 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.660 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.661 DEBUG:boto:Params: {} 2022-09-13T07:39:29.661 DEBUG:boto:Token: None 2022-09-13T07:39:29.662 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.663 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O9agk0+7KZH4FhyubV2/kCW3Jss= 2022-09-13T07:39:29.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O9agk0+7KZH4FhyubV2/kCW3Jss=', 'Content-Length': '0'} 2022-09-13T07:39:29.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024f2287118b22cdb-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.667 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.668 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.669 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.670 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.671 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.672 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.672 DEBUG:boto:Data: 2022-09-13T07:39:29.673 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.674 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.675 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.676 DEBUG:boto:Params: {} 2022-09-13T07:39:29.677 DEBUG:boto:Token: None 2022-09-13T07:39:29.678 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.679 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YIXfLrtfKoLBClnPsqwcbj+7hBg= 2022-09-13T07:39:29.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YIXfLrtfKoLBClnPsqwcbj+7hBg=', 'Content-Length': '0'} 2022-09-13T07:39:29.683 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f12b9e07087ed81-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.684 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.685 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.686 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.687 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.688 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.689 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.690 DEBUG:boto:Data: 2022-09-13T07:39:29.691 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.692 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.693 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.694 DEBUG:boto:Params: {} 2022-09-13T07:39:29.695 DEBUG:boto:Token: None 2022-09-13T07:39:29.696 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.697 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YIXfLrtfKoLBClnPsqwcbj+7hBg= 2022-09-13T07:39:29.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YIXfLrtfKoLBClnPsqwcbj+7hBg=', 'Content-Length': '0'} 2022-09-13T07:39:29.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044b48e6f41154627-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:29.703 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.704 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.705 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.706 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.707 DEBUG:boto:Method: GET 2022-09-13T07:39:29.708 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.708 DEBUG:boto:Data: 2022-09-13T07:39:29.709 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.709 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.710 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.710 DEBUG:boto:Params: {} 2022-09-13T07:39:29.711 DEBUG:boto:Token: None 2022-09-13T07:39:29.712 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.712 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O9agk0+7KZH4FhyubV2/kCW3Jss= 2022-09-13T07:39:29.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O9agk0+7KZH4FhyubV2/kCW3Jss=', 'Content-Length': '0'} 2022-09-13T07:39:29.715 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000630b8b6509aca808-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.758 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.759 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.760 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.760 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.761 DEBUG:boto:Method: GET 2022-09-13T07:39:29.761 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.762 DEBUG:boto:Data: 2022-09-13T07:39:29.763 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.763 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.764 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.764 DEBUG:boto:Params: {} 2022-09-13T07:39:29.765 DEBUG:boto:Token: None 2022-09-13T07:39:29.766 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:29.767 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O9agk0+7KZH4FhyubV2/kCW3Jss= 2022-09-13T07:39:29.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O9agk0+7KZH4FhyubV2/kCW3Jss=', 'Content-Length': '0'} 2022-09-13T07:39:29.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004002d79bf5282db3-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.771 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.771 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.772 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.772 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.773 DEBUG:boto:Method: GET 2022-09-13T07:39:29.774 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.775 DEBUG:boto:Data: 2022-09-13T07:39:29.775 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.776 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.776 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.777 DEBUG:boto:Params: {} 2022-09-13T07:39:29.778 DEBUG:boto:Token: None 2022-09-13T07:39:29.778 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.779 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f90fd9dd1b18bb3-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.783 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.784 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.785 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.785 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.786 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.786 DEBUG:boto:Method: GET 2022-09-13T07:39:29.787 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.787 DEBUG:boto:Data: 2022-09-13T07:39:29.788 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.788 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.789 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.789 DEBUG:boto:Params: {} 2022-09-13T07:39:29.790 DEBUG:boto:Token: None 2022-09-13T07:39:29.791 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.792 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d556fbcc2e01d9ef-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.796 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.798 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.798 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.799 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.800 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:29.801 DEBUG:boto:Data: 2022-09-13T07:39:29.802 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.803 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.804 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.804 DEBUG:boto:Params: {} 2022-09-13T07:39:29.805 DEBUG:boto:Token: None 2022-09-13T07:39:29.806 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj 2022-09-13T07:39:29.808 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo= 2022-09-13T07:39:29.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo=', 'Content-Length': '0'} 2022-09-13T07:39:29.812 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000962b40f08abe3c98-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.814 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.814 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.815 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.816 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:29.817 DEBUG:boto:Data: 2022-09-13T07:39:29.817 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.818 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.819 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.820 DEBUG:boto:Params: {} 2022-09-13T07:39:29.821 DEBUG:boto:Token: None 2022-09-13T07:39:29.821 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj 2022-09-13T07:39:29.822 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo= 2022-09-13T07:39:29.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QrH4vRl2aM50xZKa2og3Oi3zZNo=', 'Content-Length': '0'} 2022-09-13T07:39:29.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004bfe2c37881a8316-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:29.830 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.830 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.831 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.831 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.832 DEBUG:boto:Method: GET 2022-09-13T07:39:29.832 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.833 DEBUG:boto:Data: 2022-09-13T07:39:29.834 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.834 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.835 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.836 DEBUG:boto:Params: {} 2022-09-13T07:39:29.837 DEBUG:boto:Token: None 2022-09-13T07:39:29.837 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.838 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU= 2022-09-13T07:39:29.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU=', 'Content-Length': '0'} 2022-09-13T07:39:29.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000da3d5aec7b4e2e-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.882 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.883 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.883 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.884 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.885 DEBUG:boto:Method: GET 2022-09-13T07:39:29.891 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.892 DEBUG:boto:Data: 2022-09-13T07:39:29.892 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.893 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.893 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.894 DEBUG:boto:Params: {} 2022-09-13T07:39:29.895 DEBUG:boto:Token: None 2022-09-13T07:39:29.895 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:29.897 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU= 2022-09-13T07:39:29.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kKzc7l8hEfXuv1EDOuwj4CFH6tU=', 'Content-Length': '0'} 2022-09-13T07:39:29.900 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ecaf072bfb4bb7a-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.901 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.901 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.902 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.902 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.903 DEBUG:boto:Method: GET 2022-09-13T07:39:29.903 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.904 DEBUG:boto:Data: 2022-09-13T07:39:29.904 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.905 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.906 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.906 DEBUG:boto:Params: {} 2022-09-13T07:39:29.907 DEBUG:boto:Token: None 2022-09-13T07:39:29.908 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.908 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052817b1c9b6e86f6-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.913 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.913 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:29.914 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:29.915 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.915 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.916 DEBUG:boto:Method: GET 2022-09-13T07:39:29.916 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:29.917 DEBUG:boto:Data: 2022-09-13T07:39:29.917 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.918 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.918 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.919 DEBUG:boto:Params: {} 2022-09-13T07:39:29.919 DEBUG:boto:Token: None 2022-09-13T07:39:29.920 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/ 2022-09-13T07:39:29.920 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w= 2022-09-13T07:39:29.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dj5yWGOaDTK9PEFmbOnc5VND7+w=', 'Content-Length': '0'} 2022-09-13T07:39:29.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000093a275b6cf58b4-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.925 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:29.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:29.927 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.927 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.928 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.928 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.929 DEBUG:boto:Method: GET 2022-09-13T07:39:29.929 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.930 DEBUG:boto:Data: 2022-09-13T07:39:29.930 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.931 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.931 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.932 DEBUG:boto:Params: {} 2022-09-13T07:39:29.932 DEBUG:boto:Token: None 2022-09-13T07:39:29.933 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.934 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QsWHCvtcQsR4pyIk6U31svs0Sjo= 2022-09-13T07:39:29.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QsWHCvtcQsR4pyIk6U31svs0Sjo=', 'Content-Length': '0'} 2022-09-13T07:39:29.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000012fd731d2f2cbac-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.938 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.938 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.939 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.939 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.940 DEBUG:boto:Method: GET 2022-09-13T07:39:29.940 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.941 DEBUG:boto:Data: 2022-09-13T07:39:29.941 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.942 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.942 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.943 DEBUG:boto:Params: {} 2022-09-13T07:39:29.944 DEBUG:boto:Token: None 2022-09-13T07:39:29.944 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.945 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QsWHCvtcQsR4pyIk6U31svs0Sjo= 2022-09-13T07:39:29.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QsWHCvtcQsR4pyIk6U31svs0Sjo=', 'Content-Length': '0'} 2022-09-13T07:39:29.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd48711b88d537ea-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.948 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.949 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.949 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.950 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.951 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.951 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.952 DEBUG:boto:Data: 2022-09-13T07:39:29.952 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.953 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.953 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.954 DEBUG:boto:Params: {} 2022-09-13T07:39:29.954 DEBUG:boto:Token: None 2022-09-13T07:39:29.955 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.956 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gl1z2CmUENRNidzZZWF4ID9O9Oo= 2022-09-13T07:39:29.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gl1z2CmUENRNidzZZWF4ID9O9Oo=', 'Content-Length': '0'} 2022-09-13T07:39:29.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007814b59bceed1682-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.959 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.959 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.960 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.961 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.961 DEBUG:boto:Method: HEAD 2022-09-13T07:39:29.962 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.962 DEBUG:boto:Data: 2022-09-13T07:39:29.963 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.963 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:29.964 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.964 DEBUG:boto:Params: {} 2022-09-13T07:39:29.965 DEBUG:boto:Token: None 2022-09-13T07:39:29.966 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.966 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gl1z2CmUENRNidzZZWF4ID9O9Oo= 2022-09-13T07:39:29.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gl1z2CmUENRNidzZZWF4ID9O9Oo=', 'Content-Length': '0'} 2022-09-13T07:39:29.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097b33f55933b765c-00632033b1-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:29.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:29.971 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:29.972 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:29.973 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.973 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.974 DEBUG:boto:Method: GET 2022-09-13T07:39:29.975 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.976 DEBUG:boto:Data: 2022-09-13T07:39:29.977 DEBUG:boto:Headers: {} 2022-09-13T07:39:29.977 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:29.978 DEBUG:boto:Port: 8000 2022-09-13T07:39:29.978 DEBUG:boto:Params: {} 2022-09-13T07:39:29.979 DEBUG:boto:Token: None 2022-09-13T07:39:29.980 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:29 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:29.980 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QsWHCvtcQsR4pyIk6U31svs0Sjo= 2022-09-13T07:39:29.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QsWHCvtcQsR4pyIk6U31svs0Sjo=', 'Content-Length': '0'} 2022-09-13T07:39:29.983 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009de9e23f54645260-00632033b1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.030 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.032 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.032 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:30.033 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:30.033 DEBUG:boto:Method: GET 2022-09-13T07:39:30.034 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:30.034 DEBUG:boto:Data: 2022-09-13T07:39:30.035 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.035 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.036 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.036 DEBUG:boto:Params: {} 2022-09-13T07:39:30.037 DEBUG:boto:Token: None 2022-09-13T07:39:30.038 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:30.039 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wgwMhtqoaIhUqICgmrFdueLrDpo= 2022-09-13T07:39:30.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wgwMhtqoaIhUqICgmrFdueLrDpo=', 'Content-Length': '0'} 2022-09-13T07:39:30.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090c6ab408311e230-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.043 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.044 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.045 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.046 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.047 DEBUG:boto:Method: GET 2022-09-13T07:39:30.047 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.048 DEBUG:boto:Data: 2022-09-13T07:39:30.049 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.050 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.051 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.052 DEBUG:boto:Params: {} 2022-09-13T07:39:30.053 DEBUG:boto:Token: None 2022-09-13T07:39:30.054 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/ 2022-09-13T07:39:30.055 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw= 2022-09-13T07:39:30.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw=', 'Content-Length': '0'} 2022-09-13T07:39:30.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ccd43599cce4e46-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.060 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:30.061 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.062 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.062 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.063 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.063 DEBUG:boto:Method: GET 2022-09-13T07:39:30.064 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.064 DEBUG:boto:Data: 2022-09-13T07:39:30.065 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.065 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.066 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.066 DEBUG:boto:Params: {} 2022-09-13T07:39:30.067 DEBUG:boto:Token: None 2022-09-13T07:39:30.068 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/ 2022-09-13T07:39:30.068 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw= 2022-09-13T07:39:30.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw=', 'Content-Length': '0'} 2022-09-13T07:39:30.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011028b51b8074d27-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.073 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:30.073 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.074 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.074 DEBUG:boto:Method: HEAD 2022-09-13T07:39:30.075 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:30.075 DEBUG:boto:Data: 2022-09-13T07:39:30.076 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.076 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.077 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.078 DEBUG:boto:Params: {} 2022-09-13T07:39:30.078 DEBUG:boto:Token: None 2022-09-13T07:39:30.079 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj 2022-09-13T07:39:30.079 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0= 2022-09-13T07:39:30.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0=', 'Content-Length': '0'} 2022-09-13T07:39:30.084 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000198902dafe114025-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.085 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.086 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.087 DEBUG:boto:Method: HEAD 2022-09-13T07:39:30.088 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:30.089 DEBUG:boto:Data: 2022-09-13T07:39:30.089 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.090 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.091 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.092 DEBUG:boto:Params: {} 2022-09-13T07:39:30.093 DEBUG:boto:Token: None 2022-09-13T07:39:30.094 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj 2022-09-13T07:39:30.095 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0= 2022-09-13T07:39:30.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0=', 'Content-Length': '0'} 2022-09-13T07:39:30.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000dbe0114337ba71-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:30.102 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.103 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.103 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.104 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.105 DEBUG:boto:Method: GET 2022-09-13T07:39:30.106 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.107 DEBUG:boto:Data: 2022-09-13T07:39:30.108 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.109 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.110 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.110 DEBUG:boto:Params: {} 2022-09-13T07:39:30.111 DEBUG:boto:Token: None 2022-09-13T07:39:30.112 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.113 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM= 2022-09-13T07:39:30.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM=', 'Content-Length': '0'} 2022-09-13T07:39:30.116 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ab52715eb5ff897-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.158 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.159 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.159 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.160 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.161 DEBUG:boto:Method: GET 2022-09-13T07:39:30.161 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.162 DEBUG:boto:Data: 2022-09-13T07:39:30.162 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.163 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.164 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.164 DEBUG:boto:Params: {} 2022-09-13T07:39:30.165 DEBUG:boto:Token: None 2022-09-13T07:39:30.166 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.167 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM= 2022-09-13T07:39:30.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM=', 'Content-Length': '0'} 2022-09-13T07:39:30.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd46c7d5eebc1bca-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.171 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.172 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.173 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.174 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.175 DEBUG:boto:Method: GET 2022-09-13T07:39:30.176 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.177 DEBUG:boto:Data: 2022-09-13T07:39:30.178 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.178 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.179 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.180 DEBUG:boto:Params: {} 2022-09-13T07:39:30.181 DEBUG:boto:Token: None 2022-09-13T07:39:30.182 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/ 2022-09-13T07:39:30.182 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw= 2022-09-13T07:39:30.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw=', 'Content-Length': '0'} 2022-09-13T07:39:30.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fff37ef19667eac-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.187 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:30.187 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.188 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.189 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.190 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.190 DEBUG:boto:Method: GET 2022-09-13T07:39:30.191 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.192 DEBUG:boto:Data: 2022-09-13T07:39:30.192 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.193 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.193 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.194 DEBUG:boto:Params: {} 2022-09-13T07:39:30.195 DEBUG:boto:Token: None 2022-09-13T07:39:30.195 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/ 2022-09-13T07:39:30.196 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw= 2022-09-13T07:39:30.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw=', 'Content-Length': '0'} 2022-09-13T07:39:30.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b180344fe9f9b29-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.200 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:30.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:30.202 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.202 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.203 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.204 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.204 DEBUG:boto:Method: GET 2022-09-13T07:39:30.205 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.205 DEBUG:boto:Data: 2022-09-13T07:39:30.206 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.206 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.207 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.208 DEBUG:boto:Params: {} 2022-09-13T07:39:30.209 DEBUG:boto:Token: None 2022-09-13T07:39:30.209 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.210 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cjjdOjOGmF8KbL/LQQOrVTHVf2M= 2022-09-13T07:39:30.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cjjdOjOGmF8KbL/LQQOrVTHVf2M=', 'Content-Length': '0'} 2022-09-13T07:39:30.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000902ec0759a0221e6-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.214 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.215 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.215 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.216 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.216 DEBUG:boto:Method: GET 2022-09-13T07:39:30.217 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.218 DEBUG:boto:Data: 2022-09-13T07:39:30.218 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.219 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.219 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.220 DEBUG:boto:Params: {} 2022-09-13T07:39:30.220 DEBUG:boto:Token: None 2022-09-13T07:39:30.221 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.222 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cjjdOjOGmF8KbL/LQQOrVTHVf2M= 2022-09-13T07:39:30.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cjjdOjOGmF8KbL/LQQOrVTHVf2M=', 'Content-Length': '0'} 2022-09-13T07:39:30.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c1cf6b1d29ec0b7-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.270 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.271 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.271 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.272 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.272 DEBUG:boto:Method: HEAD 2022-09-13T07:39:30.273 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.274 DEBUG:boto:Data: 2022-09-13T07:39:30.274 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.275 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.275 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.275 DEBUG:boto:Params: {} 2022-09-13T07:39:30.276 DEBUG:boto:Token: None 2022-09-13T07:39:30.277 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.277 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:d6DjTl6u/wTyEUPf8kSy0vmn4Us= 2022-09-13T07:39:30.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:d6DjTl6u/wTyEUPf8kSy0vmn4Us=', 'Content-Length': '0'} 2022-09-13T07:39:30.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001924f6c7aef80941-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.280 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.281 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.282 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.282 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.283 DEBUG:boto:Method: HEAD 2022-09-13T07:39:30.283 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.283 DEBUG:boto:Data: 2022-09-13T07:39:30.284 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.284 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.285 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.285 DEBUG:boto:Params: {} 2022-09-13T07:39:30.286 DEBUG:boto:Token: None 2022-09-13T07:39:30.287 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.287 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:d6DjTl6u/wTyEUPf8kSy0vmn4Us= 2022-09-13T07:39:30.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:d6DjTl6u/wTyEUPf8kSy0vmn4Us=', 'Content-Length': '0'} 2022-09-13T07:39:30.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000898288298b6af116-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:30.291 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.291 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.292 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.293 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.293 DEBUG:boto:Method: GET 2022-09-13T07:39:30.294 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.294 DEBUG:boto:Data: 2022-09-13T07:39:30.295 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.295 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.296 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.296 DEBUG:boto:Params: {} 2022-09-13T07:39:30.297 DEBUG:boto:Token: None 2022-09-13T07:39:30.297 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.298 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cjjdOjOGmF8KbL/LQQOrVTHVf2M= 2022-09-13T07:39:30.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cjjdOjOGmF8KbL/LQQOrVTHVf2M=', 'Content-Length': '0'} 2022-09-13T07:39:30.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e66924ce683e0dab-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.301 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.302 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.303 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.303 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.304 DEBUG:boto:Method: GET 2022-09-13T07:39:30.305 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.306 DEBUG:boto:Data: 2022-09-13T07:39:30.306 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.307 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.307 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.308 DEBUG:boto:Params: {} 2022-09-13T07:39:30.308 DEBUG:boto:Token: None 2022-09-13T07:39:30.309 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:30.309 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cjjdOjOGmF8KbL/LQQOrVTHVf2M= 2022-09-13T07:39:30.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cjjdOjOGmF8KbL/LQQOrVTHVf2M=', 'Content-Length': '0'} 2022-09-13T07:39:30.312 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000431b5f0367894b4b-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.313 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.313 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.314 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.314 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.315 DEBUG:boto:Method: GET 2022-09-13T07:39:30.315 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.316 DEBUG:boto:Data: 2022-09-13T07:39:30.316 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.317 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.317 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.318 DEBUG:boto:Params: {} 2022-09-13T07:39:30.318 DEBUG:boto:Token: None 2022-09-13T07:39:30.319 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/ 2022-09-13T07:39:30.320 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw= 2022-09-13T07:39:30.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw=', 'Content-Length': '0'} 2022-09-13T07:39:30.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afa7707374f45d40-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.323 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:30.324 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.325 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.325 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.326 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.326 DEBUG:boto:Method: GET 2022-09-13T07:39:30.327 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.327 DEBUG:boto:Data: 2022-09-13T07:39:30.328 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.328 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.329 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.329 DEBUG:boto:Params: {} 2022-09-13T07:39:30.330 DEBUG:boto:Token: None 2022-09-13T07:39:30.331 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/ 2022-09-13T07:39:30.331 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw= 2022-09-13T07:39:30.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw=', 'Content-Length': '0'} 2022-09-13T07:39:30.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8c76beaf5e13da7-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.335 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:30.336 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.337 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.337 DEBUG:boto:Method: HEAD 2022-09-13T07:39:30.338 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:30.338 DEBUG:boto:Data: 2022-09-13T07:39:30.339 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.339 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.340 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.341 DEBUG:boto:Params: {} 2022-09-13T07:39:30.341 DEBUG:boto:Token: None 2022-09-13T07:39:30.342 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj 2022-09-13T07:39:30.342 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0= 2022-09-13T07:39:30.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0=', 'Content-Length': '0'} 2022-09-13T07:39:30.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085e3bdae3de7332c-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.347 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.348 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.348 DEBUG:boto:Method: HEAD 2022-09-13T07:39:30.349 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:30.349 DEBUG:boto:Data: 2022-09-13T07:39:30.349 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.350 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.350 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.351 DEBUG:boto:Params: {} 2022-09-13T07:39:30.352 DEBUG:boto:Token: None 2022-09-13T07:39:30.352 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj 2022-09-13T07:39:30.353 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0= 2022-09-13T07:39:30.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0=', 'Content-Length': '0'} 2022-09-13T07:39:30.357 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5ebb54853e99a50-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:30.358 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.359 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.360 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.360 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.361 DEBUG:boto:Method: GET 2022-09-13T07:39:30.361 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.362 DEBUG:boto:Data: 2022-09-13T07:39:30.362 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.363 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.363 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.364 DEBUG:boto:Params: {} 2022-09-13T07:39:30.364 DEBUG:boto:Token: None 2022-09-13T07:39:30.365 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.366 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM= 2022-09-13T07:39:30.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM=', 'Content-Length': '0'} 2022-09-13T07:39:30.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7efe8c1ef785e88-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.410 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.411 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.411 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.412 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.412 DEBUG:boto:Method: GET 2022-09-13T07:39:30.413 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.414 DEBUG:boto:Data: 2022-09-13T07:39:30.414 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.415 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.415 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.416 DEBUG:boto:Params: {} 2022-09-13T07:39:30.416 DEBUG:boto:Token: None 2022-09-13T07:39:30.417 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.417 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM= 2022-09-13T07:39:30.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM=', 'Content-Length': '0'} 2022-09-13T07:39:30.420 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038d8454019f4821a-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.421 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.422 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.422 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.423 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.423 DEBUG:boto:Method: GET 2022-09-13T07:39:30.424 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.424 DEBUG:boto:Data: 2022-09-13T07:39:30.425 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.426 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.426 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.427 DEBUG:boto:Params: {} 2022-09-13T07:39:30.427 DEBUG:boto:Token: None 2022-09-13T07:39:30.428 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/ 2022-09-13T07:39:30.429 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw= 2022-09-13T07:39:30.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw=', 'Content-Length': '0'} 2022-09-13T07:39:30.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003925a918ca6b6a24-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.433 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:30.434 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.435 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.435 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.436 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.436 DEBUG:boto:Method: GET 2022-09-13T07:39:30.437 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.437 DEBUG:boto:Data: 2022-09-13T07:39:30.438 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.438 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.439 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.439 DEBUG:boto:Params: {} 2022-09-13T07:39:30.440 DEBUG:boto:Token: None 2022-09-13T07:39:30.441 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/ 2022-09-13T07:39:30.441 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw= 2022-09-13T07:39:30.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw=', 'Content-Length': '0'} 2022-09-13T07:39:30.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeb0d1c5fa77b59c-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.446 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:30.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:30.447 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.448 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.448 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.449 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.449 DEBUG:boto:Method: GET 2022-09-13T07:39:30.450 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.451 DEBUG:boto:Data: 2022-09-13T07:39:30.451 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.452 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.452 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.453 DEBUG:boto:Params: {} 2022-09-13T07:39:30.454 DEBUG:boto:Token: None 2022-09-13T07:39:30.454 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.455 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cwjqZCS2SuGQYIMC4gsAjJe/OMo= 2022-09-13T07:39:30.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cwjqZCS2SuGQYIMC4gsAjJe/OMo=', 'Content-Length': '0'} 2022-09-13T07:39:30.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001039621b8dcac0bd-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.459 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.459 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.460 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.460 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.461 DEBUG:boto:Method: GET 2022-09-13T07:39:30.461 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.462 DEBUG:boto:Data: 2022-09-13T07:39:30.462 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.463 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.463 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.464 DEBUG:boto:Params: {} 2022-09-13T07:39:30.464 DEBUG:boto:Token: None 2022-09-13T07:39:30.465 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.466 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cwjqZCS2SuGQYIMC4gsAjJe/OMo= 2022-09-13T07:39:30.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cwjqZCS2SuGQYIMC4gsAjJe/OMo=', 'Content-Length': '0'} 2022-09-13T07:39:30.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056a2867db5ceb9e2-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.470 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.470 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.471 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.471 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.472 DEBUG:boto:Method: HEAD 2022-09-13T07:39:30.472 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.473 DEBUG:boto:Data: 2022-09-13T07:39:30.473 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.474 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.474 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.475 DEBUG:boto:Params: {} 2022-09-13T07:39:30.475 DEBUG:boto:Token: None 2022-09-13T07:39:30.476 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.476 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:G4siBBdi1gDswfoXHq/pXFGFtKw= 2022-09-13T07:39:30.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:G4siBBdi1gDswfoXHq/pXFGFtKw=', 'Content-Length': '0'} 2022-09-13T07:39:30.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f1f67e7f14118d1-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.480 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.481 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.481 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.482 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.483 DEBUG:boto:Method: HEAD 2022-09-13T07:39:30.483 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.484 DEBUG:boto:Data: 2022-09-13T07:39:30.484 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.485 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.485 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.486 DEBUG:boto:Params: {} 2022-09-13T07:39:30.487 DEBUG:boto:Token: None 2022-09-13T07:39:30.488 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.489 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:G4siBBdi1gDswfoXHq/pXFGFtKw= 2022-09-13T07:39:30.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:G4siBBdi1gDswfoXHq/pXFGFtKw=', 'Content-Length': '0'} 2022-09-13T07:39:30.492 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be32ccda7985d6f1-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:30.495 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.496 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.497 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.498 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.499 DEBUG:boto:Method: GET 2022-09-13T07:39:30.500 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.501 DEBUG:boto:Data: 2022-09-13T07:39:30.503 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.504 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.505 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.506 DEBUG:boto:Params: {} 2022-09-13T07:39:30.507 DEBUG:boto:Token: None 2022-09-13T07:39:30.508 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.509 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cwjqZCS2SuGQYIMC4gsAjJe/OMo= 2022-09-13T07:39:30.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cwjqZCS2SuGQYIMC4gsAjJe/OMo=', 'Content-Length': '0'} 2022-09-13T07:39:30.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000112a36300fee8d21-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.514 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.515 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.516 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.517 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.517 DEBUG:boto:Method: GET 2022-09-13T07:39:30.518 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.519 DEBUG:boto:Data: 2022-09-13T07:39:30.520 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.521 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.522 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.522 DEBUG:boto:Params: {} 2022-09-13T07:39:30.523 DEBUG:boto:Token: None 2022-09-13T07:39:30.523 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:30.524 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cwjqZCS2SuGQYIMC4gsAjJe/OMo= 2022-09-13T07:39:30.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cwjqZCS2SuGQYIMC4gsAjJe/OMo=', 'Content-Length': '0'} 2022-09-13T07:39:30.527 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee26f8e2c8a3e109-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.528 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.528 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.529 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.529 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.530 DEBUG:boto:Method: GET 2022-09-13T07:39:30.531 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.531 DEBUG:boto:Data: 2022-09-13T07:39:30.532 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.532 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.533 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.533 DEBUG:boto:Params: {} 2022-09-13T07:39:30.534 DEBUG:boto:Token: None 2022-09-13T07:39:30.534 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/ 2022-09-13T07:39:30.535 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw= 2022-09-13T07:39:30.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw=', 'Content-Length': '0'} 2022-09-13T07:39:30.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002603041a87f96972-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.543 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:30.545 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.546 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.547 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.548 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.548 DEBUG:boto:Method: GET 2022-09-13T07:39:30.549 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.549 DEBUG:boto:Data: 2022-09-13T07:39:30.550 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.550 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.551 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.552 DEBUG:boto:Params: {} 2022-09-13T07:39:30.552 DEBUG:boto:Token: None 2022-09-13T07:39:30.553 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/ 2022-09-13T07:39:30.554 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw= 2022-09-13T07:39:30.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw=', 'Content-Length': '0'} 2022-09-13T07:39:30.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be39347142620d9e-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.598 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:30.599 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.600 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.600 DEBUG:boto:Method: HEAD 2022-09-13T07:39:30.601 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:30.601 DEBUG:boto:Data: 2022-09-13T07:39:30.602 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.602 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.603 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.603 DEBUG:boto:Params: {} 2022-09-13T07:39:30.604 DEBUG:boto:Token: None 2022-09-13T07:39:30.604 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj 2022-09-13T07:39:30.605 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0= 2022-09-13T07:39:30.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0=', 'Content-Length': '0'} 2022-09-13T07:39:30.611 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f47f5719d6654871-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.612 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.613 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.613 DEBUG:boto:Method: HEAD 2022-09-13T07:39:30.614 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:30.614 DEBUG:boto:Data: 2022-09-13T07:39:30.615 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.615 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.616 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.616 DEBUG:boto:Params: {} 2022-09-13T07:39:30.617 DEBUG:boto:Token: None 2022-09-13T07:39:30.618 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj 2022-09-13T07:39:30.619 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0= 2022-09-13T07:39:30.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0=', 'Content-Length': '0'} 2022-09-13T07:39:30.623 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d393b3473d72d2ae-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:30.625 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.625 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.626 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.626 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.627 DEBUG:boto:Method: GET 2022-09-13T07:39:30.627 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.628 DEBUG:boto:Data: 2022-09-13T07:39:30.628 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.629 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.629 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.630 DEBUG:boto:Params: {} 2022-09-13T07:39:30.631 DEBUG:boto:Token: None 2022-09-13T07:39:30.631 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.632 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM= 2022-09-13T07:39:30.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM=', 'Content-Length': '0'} 2022-09-13T07:39:30.635 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4e96de95ebb5ad1-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.682 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.683 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.683 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.685 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.686 DEBUG:boto:Method: GET 2022-09-13T07:39:30.686 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.687 DEBUG:boto:Data: 2022-09-13T07:39:30.687 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.688 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.689 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.689 DEBUG:boto:Params: {} 2022-09-13T07:39:30.690 DEBUG:boto:Token: None 2022-09-13T07:39:30.691 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.692 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM= 2022-09-13T07:39:30.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM=', 'Content-Length': '0'} 2022-09-13T07:39:30.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073a7571f0ff5cad5-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.695 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.696 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.696 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.697 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.697 DEBUG:boto:Method: GET 2022-09-13T07:39:30.698 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.699 DEBUG:boto:Data: 2022-09-13T07:39:30.700 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.701 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.702 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.703 DEBUG:boto:Params: {} 2022-09-13T07:39:30.705 DEBUG:boto:Token: None 2022-09-13T07:39:30.706 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/ 2022-09-13T07:39:30.707 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw= 2022-09-13T07:39:30.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw=', 'Content-Length': '0'} 2022-09-13T07:39:30.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c97dfc2b36784498-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.712 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:30.714 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.715 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.715 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.717 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.718 DEBUG:boto:Method: GET 2022-09-13T07:39:30.719 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.720 DEBUG:boto:Data: 2022-09-13T07:39:30.721 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.722 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.723 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.724 DEBUG:boto:Params: {} 2022-09-13T07:39:30.725 DEBUG:boto:Token: None 2022-09-13T07:39:30.726 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/ 2022-09-13T07:39:30.727 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw= 2022-09-13T07:39:30.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw=', 'Content-Length': '0'} 2022-09-13T07:39:30.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a5e63718c2d88a8-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.733 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:30.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:30.735 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.736 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.736 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.737 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.737 DEBUG:boto:Method: GET 2022-09-13T07:39:30.738 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.738 DEBUG:boto:Data: 2022-09-13T07:39:30.739 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.740 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.740 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.741 DEBUG:boto:Params: {} 2022-09-13T07:39:30.742 DEBUG:boto:Token: None 2022-09-13T07:39:30.744 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.746 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:K2xtZDiKrFTnYZdUTVfuEEc2PH8= 2022-09-13T07:39:30.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:K2xtZDiKrFTnYZdUTVfuEEc2PH8=', 'Content-Length': '0'} 2022-09-13T07:39:30.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8089ba53a25ca5d-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.750 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.750 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.751 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.751 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.752 DEBUG:boto:Method: GET 2022-09-13T07:39:30.753 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.753 DEBUG:boto:Data: 2022-09-13T07:39:30.754 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.754 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.755 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.755 DEBUG:boto:Params: {} 2022-09-13T07:39:30.756 DEBUG:boto:Token: None 2022-09-13T07:39:30.757 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.757 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:K2xtZDiKrFTnYZdUTVfuEEc2PH8= 2022-09-13T07:39:30.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:K2xtZDiKrFTnYZdUTVfuEEc2PH8=', 'Content-Length': '0'} 2022-09-13T07:39:30.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3e70bccbbd615fb-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.761 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.762 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.762 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.763 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.763 DEBUG:boto:Method: HEAD 2022-09-13T07:39:30.764 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.765 DEBUG:boto:Data: 2022-09-13T07:39:30.765 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.766 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.766 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.767 DEBUG:boto:Params: {} 2022-09-13T07:39:30.767 DEBUG:boto:Token: None 2022-09-13T07:39:30.768 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.769 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:I55b6+HjWw0k0N2EsR4Nefr8J5Y= 2022-09-13T07:39:30.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:I55b6+HjWw0k0N2EsR4Nefr8J5Y=', 'Content-Length': '0'} 2022-09-13T07:39:30.772 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007cf46f8d33bf6fd-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.772 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.773 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.774 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.774 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.775 DEBUG:boto:Method: HEAD 2022-09-13T07:39:30.778 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.779 DEBUG:boto:Data: 2022-09-13T07:39:30.780 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.780 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.781 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.782 DEBUG:boto:Params: {} 2022-09-13T07:39:30.783 DEBUG:boto:Token: None 2022-09-13T07:39:30.784 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.784 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:I55b6+HjWw0k0N2EsR4Nefr8J5Y= 2022-09-13T07:39:30.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:I55b6+HjWw0k0N2EsR4Nefr8J5Y=', 'Content-Length': '0'} 2022-09-13T07:39:30.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b927438e9d12e98-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:30.789 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.789 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.790 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.790 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.791 DEBUG:boto:Method: GET 2022-09-13T07:39:30.791 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.792 DEBUG:boto:Data: 2022-09-13T07:39:30.792 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.793 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.793 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.794 DEBUG:boto:Params: {} 2022-09-13T07:39:30.795 DEBUG:boto:Token: None 2022-09-13T07:39:30.795 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.796 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:K2xtZDiKrFTnYZdUTVfuEEc2PH8= 2022-09-13T07:39:30.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:K2xtZDiKrFTnYZdUTVfuEEc2PH8=', 'Content-Length': '0'} 2022-09-13T07:39:30.799 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8f7b233f7cd8e12-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.800 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.801 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.801 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.802 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.803 DEBUG:boto:Method: GET 2022-09-13T07:39:30.803 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.804 DEBUG:boto:Data: 2022-09-13T07:39:30.805 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.805 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.806 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.807 DEBUG:boto:Params: {} 2022-09-13T07:39:30.808 DEBUG:boto:Token: None 2022-09-13T07:39:30.809 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:30.810 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:K2xtZDiKrFTnYZdUTVfuEEc2PH8= 2022-09-13T07:39:30.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:K2xtZDiKrFTnYZdUTVfuEEc2PH8=', 'Content-Length': '0'} 2022-09-13T07:39:30.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7bcb08feae82767-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.815 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.816 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.816 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.817 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.817 DEBUG:boto:Method: GET 2022-09-13T07:39:30.818 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.818 DEBUG:boto:Data: 2022-09-13T07:39:30.819 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.819 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.820 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.820 DEBUG:boto:Params: {} 2022-09-13T07:39:30.821 DEBUG:boto:Token: None 2022-09-13T07:39:30.821 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/ 2022-09-13T07:39:30.823 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw= 2022-09-13T07:39:30.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw=', 'Content-Length': '0'} 2022-09-13T07:39:30.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045381e61b9c5fa47-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.874 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:30.876 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.876 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.877 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.878 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.879 DEBUG:boto:Method: GET 2022-09-13T07:39:30.880 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.881 DEBUG:boto:Data: 2022-09-13T07:39:30.882 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.882 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.883 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.884 DEBUG:boto:Params: {} 2022-09-13T07:39:30.885 DEBUG:boto:Token: None 2022-09-13T07:39:30.886 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/ 2022-09-13T07:39:30.887 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw= 2022-09-13T07:39:30.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw=', 'Content-Length': '0'} 2022-09-13T07:39:30.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c319cb88989f0791-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.893 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:30.894 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.894 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.895 DEBUG:boto:Method: HEAD 2022-09-13T07:39:30.895 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:30.896 DEBUG:boto:Data: 2022-09-13T07:39:30.896 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.897 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.897 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.898 DEBUG:boto:Params: {} 2022-09-13T07:39:30.899 DEBUG:boto:Token: None 2022-09-13T07:39:30.899 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj 2022-09-13T07:39:30.900 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0= 2022-09-13T07:39:30.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0=', 'Content-Length': '0'} 2022-09-13T07:39:30.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc4961721a16e2fc-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.904 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.905 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.905 DEBUG:boto:Method: HEAD 2022-09-13T07:39:30.906 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:30.906 DEBUG:boto:Data: 2022-09-13T07:39:30.907 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.908 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.908 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.909 DEBUG:boto:Params: {} 2022-09-13T07:39:30.909 DEBUG:boto:Token: None 2022-09-13T07:39:30.910 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj 2022-09-13T07:39:30.910 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0= 2022-09-13T07:39:30.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zxsM9APu/ALeM/Zu1HqTwWpNP+0=', 'Content-Length': '0'} 2022-09-13T07:39:30.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079659e04973dbf90-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:30.916 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.917 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.917 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.918 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.919 DEBUG:boto:Method: GET 2022-09-13T07:39:30.919 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.920 DEBUG:boto:Data: 2022-09-13T07:39:30.920 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.921 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.921 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.922 DEBUG:boto:Params: {} 2022-09-13T07:39:30.922 DEBUG:boto:Token: None 2022-09-13T07:39:30.923 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.923 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM= 2022-09-13T07:39:30.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM=', 'Content-Length': '0'} 2022-09-13T07:39:30.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e813c847354b355-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.927 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:30.927 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:30.928 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.928 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.929 DEBUG:boto:Method: GET 2022-09-13T07:39:30.929 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.930 DEBUG:boto:Data: 2022-09-13T07:39:30.931 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.931 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:30.932 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.932 DEBUG:boto:Params: {} 2022-09-13T07:39:30.933 DEBUG:boto:Token: None 2022-09-13T07:39:30.933 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:30.934 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM= 2022-09-13T07:39:30.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DJcaB0M2Biu+kH+6Pb13J3KpXyM=', 'Content-Length': '0'} 2022-09-13T07:39:30.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013b63405ef4a719a-00632033b2-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.982 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.983 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.984 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.985 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.987 DEBUG:boto:Method: GET 2022-09-13T07:39:30.988 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.988 DEBUG:boto:Data: 2022-09-13T07:39:30.989 DEBUG:boto:Headers: {} 2022-09-13T07:39:30.989 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:30.990 DEBUG:boto:Port: 8000 2022-09-13T07:39:30.990 DEBUG:boto:Params: {} 2022-09-13T07:39:30.991 DEBUG:boto:Token: None 2022-09-13T07:39:30.991 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:30 GMT /yxvvuh-19/ 2022-09-13T07:39:30.992 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw= 2022-09-13T07:39:30.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XaxsUaxrJaqKLzM4HypMxkCXNbw=', 'Content-Length': '0'} 2022-09-13T07:39:30.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003042887d7f146193-00632033b2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:30.996 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:30.997 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:30.998 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:30.999 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:30.999 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.000 DEBUG:boto:Method: GET 2022-09-13T07:39:31.000 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.001 DEBUG:boto:Data: 2022-09-13T07:39:31.002 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.002 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.003 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.003 DEBUG:boto:Params: {} 2022-09-13T07:39:31.004 DEBUG:boto:Token: None 2022-09-13T07:39:31.004 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/ 2022-09-13T07:39:31.005 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E= 2022-09-13T07:39:31.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E=', 'Content-Length': '0'} 2022-09-13T07:39:31.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012eb760375743c91-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.010 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:31.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:31.012 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.012 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.013 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.013 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.014 DEBUG:boto:Method: GET 2022-09-13T07:39:31.014 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.015 DEBUG:boto:Data: 2022-09-13T07:39:31.016 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.016 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.017 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.017 DEBUG:boto:Params: {} 2022-09-13T07:39:31.018 DEBUG:boto:Token: None 2022-09-13T07:39:31.018 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.019 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KvHm9iEBJeQxI7tLfIyt3dJPbq4= 2022-09-13T07:39:31.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KvHm9iEBJeQxI7tLfIyt3dJPbq4=', 'Content-Length': '0'} 2022-09-13T07:39:31.022 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a779cf743e610c81-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.066 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.067 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.067 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.068 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.068 DEBUG:boto:Method: GET 2022-09-13T07:39:31.069 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.069 DEBUG:boto:Data: 2022-09-13T07:39:31.070 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.070 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.071 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.071 DEBUG:boto:Params: {} 2022-09-13T07:39:31.072 DEBUG:boto:Token: None 2022-09-13T07:39:31.073 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.073 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KvHm9iEBJeQxI7tLfIyt3dJPbq4= 2022-09-13T07:39:31.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KvHm9iEBJeQxI7tLfIyt3dJPbq4=', 'Content-Length': '0'} 2022-09-13T07:39:31.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a721f77cb183fef-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.077 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.078 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.078 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.079 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.079 DEBUG:boto:Method: HEAD 2022-09-13T07:39:31.080 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.080 DEBUG:boto:Data: 2022-09-13T07:39:31.081 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.081 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.081 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.082 DEBUG:boto:Params: {} 2022-09-13T07:39:31.083 DEBUG:boto:Token: None 2022-09-13T07:39:31.083 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.084 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:M66XyRma4bRRcJvjw0jQUAYm6e4= 2022-09-13T07:39:31.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:M66XyRma4bRRcJvjw0jQUAYm6e4=', 'Content-Length': '0'} 2022-09-13T07:39:31.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9383c69f12f1387-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.087 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.087 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.088 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.089 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.089 DEBUG:boto:Method: HEAD 2022-09-13T07:39:31.090 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.090 DEBUG:boto:Data: 2022-09-13T07:39:31.091 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.091 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.092 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.092 DEBUG:boto:Params: {} 2022-09-13T07:39:31.093 DEBUG:boto:Token: None 2022-09-13T07:39:31.094 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.094 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:M66XyRma4bRRcJvjw0jQUAYm6e4= 2022-09-13T07:39:31.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:M66XyRma4bRRcJvjw0jQUAYm6e4=', 'Content-Length': '0'} 2022-09-13T07:39:31.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0925d2cf000bee8-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:31.098 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.099 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.099 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.100 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.100 DEBUG:boto:Method: GET 2022-09-13T07:39:31.101 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.101 DEBUG:boto:Data: 2022-09-13T07:39:31.102 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.102 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.103 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.103 DEBUG:boto:Params: {} 2022-09-13T07:39:31.104 DEBUG:boto:Token: None 2022-09-13T07:39:31.105 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.107 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KvHm9iEBJeQxI7tLfIyt3dJPbq4= 2022-09-13T07:39:31.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KvHm9iEBJeQxI7tLfIyt3dJPbq4=', 'Content-Length': '0'} 2022-09-13T07:39:31.110 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074e4225b07883f72-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.112 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.112 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.113 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.114 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.114 DEBUG:boto:Method: GET 2022-09-13T07:39:31.115 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.116 DEBUG:boto:Data: 2022-09-13T07:39:31.117 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.118 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.119 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.120 DEBUG:boto:Params: {} 2022-09-13T07:39:31.121 DEBUG:boto:Token: None 2022-09-13T07:39:31.122 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:31.123 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KvHm9iEBJeQxI7tLfIyt3dJPbq4= 2022-09-13T07:39:31.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KvHm9iEBJeQxI7tLfIyt3dJPbq4=', 'Content-Length': '0'} 2022-09-13T07:39:31.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000683f25fb72b2605f-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.127 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:31.128 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:31.129 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.130 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.131 DEBUG:boto:Method: GET 2022-09-13T07:39:31.132 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.133 DEBUG:boto:Data: 2022-09-13T07:39:31.134 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.134 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.135 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.135 DEBUG:boto:Params: {} 2022-09-13T07:39:31.136 DEBUG:boto:Token: None 2022-09-13T07:39:31.137 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/ 2022-09-13T07:39:31.138 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E= 2022-09-13T07:39:31.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E=', 'Content-Length': '0'} 2022-09-13T07:39:31.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030987df2948834bd-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.186 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:31.187 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:31.187 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:31.188 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.188 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.189 DEBUG:boto:Method: GET 2022-09-13T07:39:31.189 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.190 DEBUG:boto:Data: 2022-09-13T07:39:31.190 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.191 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.191 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.192 DEBUG:boto:Params: {} 2022-09-13T07:39:31.192 DEBUG:boto:Token: None 2022-09-13T07:39:31.193 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/ 2022-09-13T07:39:31.194 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E= 2022-09-13T07:39:31.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E=', 'Content-Length': '0'} 2022-09-13T07:39:31.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ed296374bbf8c76-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.198 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:31.199 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.199 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.200 DEBUG:boto:Method: HEAD 2022-09-13T07:39:31.200 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:31.201 DEBUG:boto:Data: 2022-09-13T07:39:31.201 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.202 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.203 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.203 DEBUG:boto:Params: {} 2022-09-13T07:39:31.210 DEBUG:boto:Token: None 2022-09-13T07:39:31.211 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj 2022-09-13T07:39:31.211 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gdVMjXd4mKLhTf3B0YdRBVsMJtY= 2022-09-13T07:39:31.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gdVMjXd4mKLhTf3B0YdRBVsMJtY=', 'Content-Length': '0'} 2022-09-13T07:39:31.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ef1c2760864e916-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.217 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.218 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.219 DEBUG:boto:Method: HEAD 2022-09-13T07:39:31.220 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:31.221 DEBUG:boto:Data: 2022-09-13T07:39:31.222 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.223 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.224 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.224 DEBUG:boto:Params: {} 2022-09-13T07:39:31.226 DEBUG:boto:Token: None 2022-09-13T07:39:31.227 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj 2022-09-13T07:39:31.228 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gdVMjXd4mKLhTf3B0YdRBVsMJtY= 2022-09-13T07:39:31.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gdVMjXd4mKLhTf3B0YdRBVsMJtY=', 'Content-Length': '0'} 2022-09-13T07:39:31.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098f86445d31f36af-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:31.235 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.236 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.237 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.238 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.239 DEBUG:boto:Method: GET 2022-09-13T07:39:31.240 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.241 DEBUG:boto:Data: 2022-09-13T07:39:31.242 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.243 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.244 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.245 DEBUG:boto:Params: {} 2022-09-13T07:39:31.246 DEBUG:boto:Token: None 2022-09-13T07:39:31.247 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.248 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gFnw9tg8T65jeFYk0jDzjo9CHxM= 2022-09-13T07:39:31.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gFnw9tg8T65jeFYk0jDzjo9CHxM=', 'Content-Length': '0'} 2022-09-13T07:39:31.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011d4440bc62e2368-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.254 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.255 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.256 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.257 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.258 DEBUG:boto:Method: GET 2022-09-13T07:39:31.259 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.260 DEBUG:boto:Data: 2022-09-13T07:39:31.261 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.262 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.263 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.264 DEBUG:boto:Params: {} 2022-09-13T07:39:31.265 DEBUG:boto:Token: None 2022-09-13T07:39:31.266 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.267 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gFnw9tg8T65jeFYk0jDzjo9CHxM= 2022-09-13T07:39:31.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gFnw9tg8T65jeFYk0jDzjo9CHxM=', 'Content-Length': '0'} 2022-09-13T07:39:31.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d14ca7f0ba1c0431-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.273 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:31.274 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:31.275 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.276 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.277 DEBUG:boto:Method: GET 2022-09-13T07:39:31.278 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.279 DEBUG:boto:Data: 2022-09-13T07:39:31.280 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.281 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.282 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.283 DEBUG:boto:Params: {} 2022-09-13T07:39:31.284 DEBUG:boto:Token: None 2022-09-13T07:39:31.285 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/ 2022-09-13T07:39:31.286 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E= 2022-09-13T07:39:31.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E=', 'Content-Length': '0'} 2022-09-13T07:39:31.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fa789b1c1e53bc1-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.292 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:31.293 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:31.294 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:31.295 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.296 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.297 DEBUG:boto:Method: GET 2022-09-13T07:39:31.298 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.299 DEBUG:boto:Data: 2022-09-13T07:39:31.300 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.301 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.302 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.303 DEBUG:boto:Params: {} 2022-09-13T07:39:31.304 DEBUG:boto:Token: None 2022-09-13T07:39:31.305 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/ 2022-09-13T07:39:31.307 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E= 2022-09-13T07:39:31.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E=', 'Content-Length': '0'} 2022-09-13T07:39:31.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d86c989c219a8e4-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.313 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:31.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:31.316 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.317 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.317 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.318 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.319 DEBUG:boto:Method: GET 2022-09-13T07:39:31.320 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.321 DEBUG:boto:Data: 2022-09-13T07:39:31.322 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.323 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.324 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.325 DEBUG:boto:Params: {} 2022-09-13T07:39:31.326 DEBUG:boto:Token: None 2022-09-13T07:39:31.327 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.328 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gY3s1l8OTbgxNDAfyibcMiI4VFc= 2022-09-13T07:39:31.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gY3s1l8OTbgxNDAfyibcMiI4VFc=', 'Content-Length': '0'} 2022-09-13T07:39:31.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081736b09fb255f37-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.334 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.335 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.335 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.336 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.337 DEBUG:boto:Method: GET 2022-09-13T07:39:31.338 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.339 DEBUG:boto:Data: 2022-09-13T07:39:31.340 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.341 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.342 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.343 DEBUG:boto:Params: {} 2022-09-13T07:39:31.344 DEBUG:boto:Token: None 2022-09-13T07:39:31.345 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.346 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gY3s1l8OTbgxNDAfyibcMiI4VFc= 2022-09-13T07:39:31.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gY3s1l8OTbgxNDAfyibcMiI4VFc=', 'Content-Length': '0'} 2022-09-13T07:39:31.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe075f0833c6a8a9-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.351 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.352 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.353 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.354 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.355 DEBUG:boto:Method: HEAD 2022-09-13T07:39:31.356 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.357 DEBUG:boto:Data: 2022-09-13T07:39:31.358 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.359 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.360 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.361 DEBUG:boto:Params: {} 2022-09-13T07:39:31.362 DEBUG:boto:Token: None 2022-09-13T07:39:31.363 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.364 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OQaa9+/VOW0Ge/p2XmQFvpL4Kpo= 2022-09-13T07:39:31.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OQaa9+/VOW0Ge/p2XmQFvpL4Kpo=', 'Content-Length': '0'} 2022-09-13T07:39:31.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002646231b743de301-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.368 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.369 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.369 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.370 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.370 DEBUG:boto:Method: HEAD 2022-09-13T07:39:31.371 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.371 DEBUG:boto:Data: 2022-09-13T07:39:31.372 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.372 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.373 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.373 DEBUG:boto:Params: {} 2022-09-13T07:39:31.374 DEBUG:boto:Token: None 2022-09-13T07:39:31.374 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.375 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OQaa9+/VOW0Ge/p2XmQFvpL4Kpo= 2022-09-13T07:39:31.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OQaa9+/VOW0Ge/p2XmQFvpL4Kpo=', 'Content-Length': '0'} 2022-09-13T07:39:31.378 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9545a2a3c889b13-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:31.379 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.380 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.380 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.381 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.382 DEBUG:boto:Method: GET 2022-09-13T07:39:31.383 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.384 DEBUG:boto:Data: 2022-09-13T07:39:31.385 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.387 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.388 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.389 DEBUG:boto:Params: {} 2022-09-13T07:39:31.390 DEBUG:boto:Token: None 2022-09-13T07:39:31.391 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.393 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gY3s1l8OTbgxNDAfyibcMiI4VFc= 2022-09-13T07:39:31.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gY3s1l8OTbgxNDAfyibcMiI4VFc=', 'Content-Length': '0'} 2022-09-13T07:39:31.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002204d7f1cfe7a67f-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.403 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.404 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.404 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.405 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.405 DEBUG:boto:Method: GET 2022-09-13T07:39:31.406 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.406 DEBUG:boto:Data: 2022-09-13T07:39:31.407 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.408 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.408 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.409 DEBUG:boto:Params: {} 2022-09-13T07:39:31.410 DEBUG:boto:Token: None 2022-09-13T07:39:31.411 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:31.411 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gY3s1l8OTbgxNDAfyibcMiI4VFc= 2022-09-13T07:39:31.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gY3s1l8OTbgxNDAfyibcMiI4VFc=', 'Content-Length': '0'} 2022-09-13T07:39:31.414 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef13cd24bdb6c16a-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.458 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:31.459 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:31.459 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.460 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.461 DEBUG:boto:Method: GET 2022-09-13T07:39:31.461 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.462 DEBUG:boto:Data: 2022-09-13T07:39:31.463 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.464 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.464 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.465 DEBUG:boto:Params: {} 2022-09-13T07:39:31.466 DEBUG:boto:Token: None 2022-09-13T07:39:31.467 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/ 2022-09-13T07:39:31.468 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E= 2022-09-13T07:39:31.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E=', 'Content-Length': '0'} 2022-09-13T07:39:31.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b33df7c885b7fe80-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.474 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:31.475 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:31.475 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:31.476 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.476 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.477 DEBUG:boto:Method: GET 2022-09-13T07:39:31.477 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.478 DEBUG:boto:Data: 2022-09-13T07:39:31.478 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.479 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.479 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.480 DEBUG:boto:Params: {} 2022-09-13T07:39:31.481 DEBUG:boto:Token: None 2022-09-13T07:39:31.481 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/ 2022-09-13T07:39:31.482 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E= 2022-09-13T07:39:31.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E=', 'Content-Length': '0'} 2022-09-13T07:39:31.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd567479789bad9d-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.487 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:31.487 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.488 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.488 DEBUG:boto:Method: HEAD 2022-09-13T07:39:31.489 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:31.489 DEBUG:boto:Data: 2022-09-13T07:39:31.490 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.491 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.491 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.492 DEBUG:boto:Params: {} 2022-09-13T07:39:31.493 DEBUG:boto:Token: None 2022-09-13T07:39:31.493 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj 2022-09-13T07:39:31.494 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gdVMjXd4mKLhTf3B0YdRBVsMJtY= 2022-09-13T07:39:31.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gdVMjXd4mKLhTf3B0YdRBVsMJtY=', 'Content-Length': '0'} 2022-09-13T07:39:31.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8762ad1ca15a7b2-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.499 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.500 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.500 DEBUG:boto:Method: HEAD 2022-09-13T07:39:31.501 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:31.501 DEBUG:boto:Data: 2022-09-13T07:39:31.502 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.502 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.504 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.504 DEBUG:boto:Params: {} 2022-09-13T07:39:31.505 DEBUG:boto:Token: None 2022-09-13T07:39:31.506 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj 2022-09-13T07:39:31.506 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gdVMjXd4mKLhTf3B0YdRBVsMJtY= 2022-09-13T07:39:31.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gdVMjXd4mKLhTf3B0YdRBVsMJtY=', 'Content-Length': '0'} 2022-09-13T07:39:31.511 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000250108ce39ce9ecc-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:31.512 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.512 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.513 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.514 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.514 DEBUG:boto:Method: GET 2022-09-13T07:39:31.515 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.515 DEBUG:boto:Data: 2022-09-13T07:39:31.516 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.516 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.517 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.517 DEBUG:boto:Params: {} 2022-09-13T07:39:31.518 DEBUG:boto:Token: None 2022-09-13T07:39:31.519 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.519 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gFnw9tg8T65jeFYk0jDzjo9CHxM= 2022-09-13T07:39:31.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gFnw9tg8T65jeFYk0jDzjo9CHxM=', 'Content-Length': '0'} 2022-09-13T07:39:31.522 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090fd2f9ff8910483-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.566 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.567 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.567 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.568 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.568 DEBUG:boto:Method: GET 2022-09-13T07:39:31.569 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.569 DEBUG:boto:Data: 2022-09-13T07:39:31.570 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.570 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.571 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.572 DEBUG:boto:Params: {} 2022-09-13T07:39:31.572 DEBUG:boto:Token: None 2022-09-13T07:39:31.573 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.574 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gFnw9tg8T65jeFYk0jDzjo9CHxM= 2022-09-13T07:39:31.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gFnw9tg8T65jeFYk0jDzjo9CHxM=', 'Content-Length': '0'} 2022-09-13T07:39:31.577 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9bb50108a216890-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.578 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:31.578 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:31.579 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.580 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.580 DEBUG:boto:Method: GET 2022-09-13T07:39:31.581 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.581 DEBUG:boto:Data: 2022-09-13T07:39:31.582 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.582 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.583 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.583 DEBUG:boto:Params: {} 2022-09-13T07:39:31.584 DEBUG:boto:Token: None 2022-09-13T07:39:31.584 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/ 2022-09-13T07:39:31.585 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E= 2022-09-13T07:39:31.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E=', 'Content-Length': '0'} 2022-09-13T07:39:31.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045b89d0e9fbd1fcd-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.589 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:31.590 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:31.591 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:31.591 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.592 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.592 DEBUG:boto:Method: GET 2022-09-13T07:39:31.593 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.594 DEBUG:boto:Data: 2022-09-13T07:39:31.595 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.596 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.597 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.598 DEBUG:boto:Params: {} 2022-09-13T07:39:31.599 DEBUG:boto:Token: None 2022-09-13T07:39:31.600 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/ 2022-09-13T07:39:31.601 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E= 2022-09-13T07:39:31.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E=', 'Content-Length': '0'} 2022-09-13T07:39:31.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ff813b8cfcf1992-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.607 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:31.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:31.610 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.611 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.612 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.613 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.613 DEBUG:boto:Method: GET 2022-09-13T07:39:31.614 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.615 DEBUG:boto:Data: 2022-09-13T07:39:31.615 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.616 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.616 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.617 DEBUG:boto:Params: {} 2022-09-13T07:39:31.617 DEBUG:boto:Token: None 2022-09-13T07:39:31.618 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.619 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bjn8O1RIfTBOeBdpIIkbZZhlCLo= 2022-09-13T07:39:31.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjn8O1RIfTBOeBdpIIkbZZhlCLo=', 'Content-Length': '0'} 2022-09-13T07:39:31.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a6804173c2b4d28-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.624 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.625 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.626 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.627 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.627 DEBUG:boto:Method: GET 2022-09-13T07:39:31.628 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.629 DEBUG:boto:Data: 2022-09-13T07:39:31.630 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.630 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.631 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.631 DEBUG:boto:Params: {} 2022-09-13T07:39:31.632 DEBUG:boto:Token: None 2022-09-13T07:39:31.633 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.633 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bjn8O1RIfTBOeBdpIIkbZZhlCLo= 2022-09-13T07:39:31.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjn8O1RIfTBOeBdpIIkbZZhlCLo=', 'Content-Length': '0'} 2022-09-13T07:39:31.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e59ae34c2119559e-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.682 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.683 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.683 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.684 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.684 DEBUG:boto:Method: HEAD 2022-09-13T07:39:31.685 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.685 DEBUG:boto:Data: 2022-09-13T07:39:31.686 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.687 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.687 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.688 DEBUG:boto:Params: {} 2022-09-13T07:39:31.688 DEBUG:boto:Token: None 2022-09-13T07:39:31.689 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.690 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:P+j6vp2cgEoXw6iQLwInw8h/x24= 2022-09-13T07:39:31.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:P+j6vp2cgEoXw6iQLwInw8h/x24=', 'Content-Length': '0'} 2022-09-13T07:39:31.692 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036450c01851af7e5-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.693 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.693 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.694 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.694 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.695 DEBUG:boto:Method: HEAD 2022-09-13T07:39:31.695 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.696 DEBUG:boto:Data: 2022-09-13T07:39:31.696 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.697 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.697 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.698 DEBUG:boto:Params: {} 2022-09-13T07:39:31.698 DEBUG:boto:Token: None 2022-09-13T07:39:31.699 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.700 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:P+j6vp2cgEoXw6iQLwInw8h/x24= 2022-09-13T07:39:31.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:P+j6vp2cgEoXw6iQLwInw8h/x24=', 'Content-Length': '0'} 2022-09-13T07:39:31.703 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b93211442756b5c-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:31.705 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.705 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.706 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.706 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.707 DEBUG:boto:Method: GET 2022-09-13T07:39:31.707 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.708 DEBUG:boto:Data: 2022-09-13T07:39:31.708 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.709 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.709 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.710 DEBUG:boto:Params: {} 2022-09-13T07:39:31.710 DEBUG:boto:Token: None 2022-09-13T07:39:31.711 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.712 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bjn8O1RIfTBOeBdpIIkbZZhlCLo= 2022-09-13T07:39:31.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjn8O1RIfTBOeBdpIIkbZZhlCLo=', 'Content-Length': '0'} 2022-09-13T07:39:31.715 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000774a0db1606b8c69-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.717 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.717 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.718 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.719 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.719 DEBUG:boto:Method: GET 2022-09-13T07:39:31.720 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.720 DEBUG:boto:Data: 2022-09-13T07:39:31.721 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.721 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.722 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.723 DEBUG:boto:Params: {} 2022-09-13T07:39:31.723 DEBUG:boto:Token: None 2022-09-13T07:39:31.724 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:31.725 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bjn8O1RIfTBOeBdpIIkbZZhlCLo= 2022-09-13T07:39:31.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjn8O1RIfTBOeBdpIIkbZZhlCLo=', 'Content-Length': '0'} 2022-09-13T07:39:31.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000953f0881e5956035-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.729 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:31.729 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:31.730 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.730 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.731 DEBUG:boto:Method: GET 2022-09-13T07:39:31.732 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.732 DEBUG:boto:Data: 2022-09-13T07:39:31.733 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.733 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.734 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.734 DEBUG:boto:Params: {} 2022-09-13T07:39:31.735 DEBUG:boto:Token: None 2022-09-13T07:39:31.736 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/ 2022-09-13T07:39:31.736 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E= 2022-09-13T07:39:31.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E=', 'Content-Length': '0'} 2022-09-13T07:39:31.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc5e76ba0a12f24b-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.741 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:31.742 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:31.742 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:31.743 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.743 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.744 DEBUG:boto:Method: GET 2022-09-13T07:39:31.745 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.745 DEBUG:boto:Data: 2022-09-13T07:39:31.746 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.746 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.747 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.747 DEBUG:boto:Params: {} 2022-09-13T07:39:31.748 DEBUG:boto:Token: None 2022-09-13T07:39:31.749 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/ 2022-09-13T07:39:31.749 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E= 2022-09-13T07:39:31.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E=', 'Content-Length': '0'} 2022-09-13T07:39:31.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0ea0877e5ffa3f0-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.754 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:31.755 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.756 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.756 DEBUG:boto:Method: HEAD 2022-09-13T07:39:31.757 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:31.758 DEBUG:boto:Data: 2022-09-13T07:39:31.758 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.759 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.759 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.760 DEBUG:boto:Params: {} 2022-09-13T07:39:31.760 DEBUG:boto:Token: None 2022-09-13T07:39:31.761 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj 2022-09-13T07:39:31.762 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gdVMjXd4mKLhTf3B0YdRBVsMJtY= 2022-09-13T07:39:31.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gdVMjXd4mKLhTf3B0YdRBVsMJtY=', 'Content-Length': '0'} 2022-09-13T07:39:31.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7f8d8b904d81a5f-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.767 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.767 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.768 DEBUG:boto:Method: HEAD 2022-09-13T07:39:31.768 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:31.769 DEBUG:boto:Data: 2022-09-13T07:39:31.769 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.770 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.770 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.771 DEBUG:boto:Params: {} 2022-09-13T07:39:31.771 DEBUG:boto:Token: None 2022-09-13T07:39:31.772 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj 2022-09-13T07:39:31.772 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gdVMjXd4mKLhTf3B0YdRBVsMJtY= 2022-09-13T07:39:31.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gdVMjXd4mKLhTf3B0YdRBVsMJtY=', 'Content-Length': '0'} 2022-09-13T07:39:31.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db8bfdca19596db9-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:31.778 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.779 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.779 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.780 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.780 DEBUG:boto:Method: GET 2022-09-13T07:39:31.781 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.782 DEBUG:boto:Data: 2022-09-13T07:39:31.782 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.783 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.784 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.785 DEBUG:boto:Params: {} 2022-09-13T07:39:31.786 DEBUG:boto:Token: None 2022-09-13T07:39:31.787 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.788 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gFnw9tg8T65jeFYk0jDzjo9CHxM= 2022-09-13T07:39:31.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gFnw9tg8T65jeFYk0jDzjo9CHxM=', 'Content-Length': '0'} 2022-09-13T07:39:31.792 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c052622fc02b692-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.793 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.794 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.795 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.796 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.797 DEBUG:boto:Method: GET 2022-09-13T07:39:31.798 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.799 DEBUG:boto:Data: 2022-09-13T07:39:31.800 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.801 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.802 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.802 DEBUG:boto:Params: {} 2022-09-13T07:39:31.803 DEBUG:boto:Token: None 2022-09-13T07:39:31.803 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:31.804 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gFnw9tg8T65jeFYk0jDzjo9CHxM= 2022-09-13T07:39:31.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gFnw9tg8T65jeFYk0jDzjo9CHxM=', 'Content-Length': '0'} 2022-09-13T07:39:31.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000daf783fbf41908a-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.854 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:31.855 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:31.855 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.856 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.857 DEBUG:boto:Method: GET 2022-09-13T07:39:31.857 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.858 DEBUG:boto:Data: 2022-09-13T07:39:31.858 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.859 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.859 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.860 DEBUG:boto:Params: {} 2022-09-13T07:39:31.860 DEBUG:boto:Token: None 2022-09-13T07:39:31.861 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/ 2022-09-13T07:39:31.862 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E= 2022-09-13T07:39:31.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E=', 'Content-Length': '0'} 2022-09-13T07:39:31.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6375ff7822132bb-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.867 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:31.869 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:31.870 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:31.871 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.872 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.873 DEBUG:boto:Method: GET 2022-09-13T07:39:31.874 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:31.875 DEBUG:boto:Data: 2022-09-13T07:39:31.876 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.877 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.878 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.879 DEBUG:boto:Params: {} 2022-09-13T07:39:31.881 DEBUG:boto:Token: None 2022-09-13T07:39:31.882 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/ 2022-09-13T07:39:31.883 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E= 2022-09-13T07:39:31.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2/OuP2P1pmRP+9F6KkHZhL5K01E=', 'Content-Length': '0'} 2022-09-13T07:39:31.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d96c59b74b57a83-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.889 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:31.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:31.891 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.891 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.892 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.892 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.893 DEBUG:boto:Method: GET 2022-09-13T07:39:31.893 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.894 DEBUG:boto:Data: 2022-09-13T07:39:31.894 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.895 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.896 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.896 DEBUG:boto:Params: {} 2022-09-13T07:39:31.897 DEBUG:boto:Token: None 2022-09-13T07:39:31.898 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.898 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GM5c1GGY9JDnysoagfhHbVsIvRw= 2022-09-13T07:39:31.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GM5c1GGY9JDnysoagfhHbVsIvRw=', 'Content-Length': '0'} 2022-09-13T07:39:31.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049cfe4eeb83f8abf-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.902 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.903 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.903 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.904 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.905 DEBUG:boto:Method: GET 2022-09-13T07:39:31.914 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.915 DEBUG:boto:Data: 2022-09-13T07:39:31.915 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.916 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.917 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.917 DEBUG:boto:Params: {} 2022-09-13T07:39:31.918 DEBUG:boto:Token: None 2022-09-13T07:39:31.918 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.919 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GM5c1GGY9JDnysoagfhHbVsIvRw= 2022-09-13T07:39:31.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GM5c1GGY9JDnysoagfhHbVsIvRw=', 'Content-Length': '0'} 2022-09-13T07:39:31.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd2e1ba0477af3ba-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.922 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.923 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.923 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.924 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.924 DEBUG:boto:Method: HEAD 2022-09-13T07:39:31.925 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.925 DEBUG:boto:Data: 2022-09-13T07:39:31.926 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.926 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.927 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.927 DEBUG:boto:Params: {} 2022-09-13T07:39:31.928 DEBUG:boto:Token: None 2022-09-13T07:39:31.929 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.929 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:69PwK0KKsust0j14FHMPcuIu0qE= 2022-09-13T07:39:31.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:69PwK0KKsust0j14FHMPcuIu0qE=', 'Content-Length': '0'} 2022-09-13T07:39:31.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071dbe7e3713c2f4b-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.933 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.933 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.934 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.934 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.935 DEBUG:boto:Method: HEAD 2022-09-13T07:39:31.935 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.936 DEBUG:boto:Data: 2022-09-13T07:39:31.936 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.937 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.937 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.938 DEBUG:boto:Params: {} 2022-09-13T07:39:31.939 DEBUG:boto:Token: None 2022-09-13T07:39:31.939 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.940 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:69PwK0KKsust0j14FHMPcuIu0qE= 2022-09-13T07:39:31.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:69PwK0KKsust0j14FHMPcuIu0qE=', 'Content-Length': '0'} 2022-09-13T07:39:31.943 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6c84bcb94b2781f-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:31.944 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.944 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.945 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.945 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.946 DEBUG:boto:Method: GET 2022-09-13T07:39:31.946 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.947 DEBUG:boto:Data: 2022-09-13T07:39:31.948 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.948 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:31.949 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.949 DEBUG:boto:Params: {} 2022-09-13T07:39:31.950 DEBUG:boto:Token: None 2022-09-13T07:39:31.950 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.951 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GM5c1GGY9JDnysoagfhHbVsIvRw= 2022-09-13T07:39:31.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GM5c1GGY9JDnysoagfhHbVsIvRw=', 'Content-Length': '0'} 2022-09-13T07:39:31.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004bc823f25240cfed-00632033b3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:31.955 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:31.955 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:31.956 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.956 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.957 DEBUG:boto:Method: GET 2022-09-13T07:39:31.957 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.958 DEBUG:boto:Data: 2022-09-13T07:39:31.958 DEBUG:boto:Headers: {} 2022-09-13T07:39:31.959 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:31.959 DEBUG:boto:Port: 8000 2022-09-13T07:39:31.960 DEBUG:boto:Params: {} 2022-09-13T07:39:31.961 DEBUG:boto:Token: None 2022-09-13T07:39:31.961 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:31 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:31.962 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GM5c1GGY9JDnysoagfhHbVsIvRw= 2022-09-13T07:39:31.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GM5c1GGY9JDnysoagfhHbVsIvRw=', 'Content-Length': '0'} 2022-09-13T07:39:31.964 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccf0b4daea23bb02-00632033b3-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.006 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.007 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.007 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.008 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.010 DEBUG:boto:Method: GET 2022-09-13T07:39:32.011 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.011 DEBUG:boto:Data: 2022-09-13T07:39:32.012 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.012 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.013 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.013 DEBUG:boto:Params: {} 2022-09-13T07:39:32.014 DEBUG:boto:Token: None 2022-09-13T07:39:32.014 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.015 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d3a014de1700ca5-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.020 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.021 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.022 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.022 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.023 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.023 DEBUG:boto:Method: GET 2022-09-13T07:39:32.024 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.024 DEBUG:boto:Data: 2022-09-13T07:39:32.025 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.025 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.026 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.026 DEBUG:boto:Params: {} 2022-09-13T07:39:32.027 DEBUG:boto:Token: None 2022-09-13T07:39:32.028 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.028 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006756b2d73c956963-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.033 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.034 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.034 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.035 DEBUG:boto:Method: HEAD 2022-09-13T07:39:32.035 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:32.036 DEBUG:boto:Data: 2022-09-13T07:39:32.036 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.037 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.037 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.038 DEBUG:boto:Params: {} 2022-09-13T07:39:32.038 DEBUG:boto:Token: None 2022-09-13T07:39:32.039 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj 2022-09-13T07:39:32.039 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk= 2022-09-13T07:39:32.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk=', 'Content-Length': '0'} 2022-09-13T07:39:32.043 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e13ef71a54a974b1-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.044 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.045 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.045 DEBUG:boto:Method: HEAD 2022-09-13T07:39:32.046 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:32.046 DEBUG:boto:Data: 2022-09-13T07:39:32.047 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.047 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.048 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.048 DEBUG:boto:Params: {} 2022-09-13T07:39:32.049 DEBUG:boto:Token: None 2022-09-13T07:39:32.049 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj 2022-09-13T07:39:32.050 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk= 2022-09-13T07:39:32.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk=', 'Content-Length': '0'} 2022-09-13T07:39:32.055 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002006c846ba729c06-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:32.058 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.059 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.060 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.061 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.062 DEBUG:boto:Method: GET 2022-09-13T07:39:32.063 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.064 DEBUG:boto:Data: 2022-09-13T07:39:32.064 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.065 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.067 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.068 DEBUG:boto:Params: {} 2022-09-13T07:39:32.068 DEBUG:boto:Token: None 2022-09-13T07:39:32.069 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.070 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU= 2022-09-13T07:39:32.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU=', 'Content-Length': '0'} 2022-09-13T07:39:32.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b30ea60bc136e853-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.114 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.115 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.115 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.116 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.117 DEBUG:boto:Method: GET 2022-09-13T07:39:32.117 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.118 DEBUG:boto:Data: 2022-09-13T07:39:32.118 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.119 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.120 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.120 DEBUG:boto:Params: {} 2022-09-13T07:39:32.121 DEBUG:boto:Token: None 2022-09-13T07:39:32.122 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.122 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU= 2022-09-13T07:39:32.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU=', 'Content-Length': '0'} 2022-09-13T07:39:32.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9c58a3228d443c8-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.127 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.128 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.128 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.129 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.129 DEBUG:boto:Method: GET 2022-09-13T07:39:32.130 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.130 DEBUG:boto:Data: 2022-09-13T07:39:32.131 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.132 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.133 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.133 DEBUG:boto:Params: {} 2022-09-13T07:39:32.134 DEBUG:boto:Token: None 2022-09-13T07:39:32.135 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.135 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000434e685b141206f0-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.141 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.142 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.143 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.144 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.145 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.146 DEBUG:boto:Method: GET 2022-09-13T07:39:32.147 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.148 DEBUG:boto:Data: 2022-09-13T07:39:32.149 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.149 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.150 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.151 DEBUG:boto:Params: {} 2022-09-13T07:39:32.151 DEBUG:boto:Token: None 2022-09-13T07:39:32.152 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.153 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b62feabd2efb9b58-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.198 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:32.203 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.204 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.204 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.205 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.205 DEBUG:boto:Method: GET 2022-09-13T07:39:32.206 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.206 DEBUG:boto:Data: 2022-09-13T07:39:32.207 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.207 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.208 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.208 DEBUG:boto:Params: {} 2022-09-13T07:39:32.209 DEBUG:boto:Token: None 2022-09-13T07:39:32.210 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.210 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2NqWNJhDfLsjVUgNi1ifrr9K/DQ= 2022-09-13T07:39:32.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2NqWNJhDfLsjVUgNi1ifrr9K/DQ=', 'Content-Length': '0'} 2022-09-13T07:39:32.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001564f47458469ec5-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.216 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.217 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.217 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.218 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.219 DEBUG:boto:Method: GET 2022-09-13T07:39:32.220 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.220 DEBUG:boto:Data: 2022-09-13T07:39:32.221 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.221 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.222 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.222 DEBUG:boto:Params: {} 2022-09-13T07:39:32.223 DEBUG:boto:Token: None 2022-09-13T07:39:32.223 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.224 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2NqWNJhDfLsjVUgNi1ifrr9K/DQ= 2022-09-13T07:39:32.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2NqWNJhDfLsjVUgNi1ifrr9K/DQ=', 'Content-Length': '0'} 2022-09-13T07:39:32.227 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087a511c4add106fa-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.227 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.228 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.229 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.229 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.230 DEBUG:boto:Method: HEAD 2022-09-13T07:39:32.230 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.231 DEBUG:boto:Data: 2022-09-13T07:39:32.231 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.231 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.232 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.232 DEBUG:boto:Params: {} 2022-09-13T07:39:32.233 DEBUG:boto:Token: None 2022-09-13T07:39:32.233 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.234 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S7c1SqZNJsLlYwpSiDX3fj677F4= 2022-09-13T07:39:32.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S7c1SqZNJsLlYwpSiDX3fj677F4=', 'Content-Length': '0'} 2022-09-13T07:39:32.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccb69179993a9c17-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.238 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.239 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.240 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.241 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.241 DEBUG:boto:Method: HEAD 2022-09-13T07:39:32.242 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.243 DEBUG:boto:Data: 2022-09-13T07:39:32.243 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.244 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.245 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.246 DEBUG:boto:Params: {} 2022-09-13T07:39:32.247 DEBUG:boto:Token: None 2022-09-13T07:39:32.248 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.248 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S7c1SqZNJsLlYwpSiDX3fj677F4= 2022-09-13T07:39:32.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S7c1SqZNJsLlYwpSiDX3fj677F4=', 'Content-Length': '0'} 2022-09-13T07:39:32.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5f4ad634d76d117-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:32.254 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.255 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.255 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.256 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.257 DEBUG:boto:Method: GET 2022-09-13T07:39:32.257 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.258 DEBUG:boto:Data: 2022-09-13T07:39:32.258 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.259 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.259 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.260 DEBUG:boto:Params: {} 2022-09-13T07:39:32.260 DEBUG:boto:Token: None 2022-09-13T07:39:32.261 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.262 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2NqWNJhDfLsjVUgNi1ifrr9K/DQ= 2022-09-13T07:39:32.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2NqWNJhDfLsjVUgNi1ifrr9K/DQ=', 'Content-Length': '0'} 2022-09-13T07:39:32.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9f4016f9d47864e-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.265 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.266 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.266 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.267 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.267 DEBUG:boto:Method: GET 2022-09-13T07:39:32.268 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.268 DEBUG:boto:Data: 2022-09-13T07:39:32.269 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.270 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.270 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.271 DEBUG:boto:Params: {} 2022-09-13T07:39:32.272 DEBUG:boto:Token: None 2022-09-13T07:39:32.272 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:32.273 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2NqWNJhDfLsjVUgNi1ifrr9K/DQ= 2022-09-13T07:39:32.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2NqWNJhDfLsjVUgNi1ifrr9K/DQ=', 'Content-Length': '0'} 2022-09-13T07:39:32.276 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000deddf041fe1cae2b-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.277 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.277 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.278 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.279 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.279 DEBUG:boto:Method: GET 2022-09-13T07:39:32.280 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.280 DEBUG:boto:Data: 2022-09-13T07:39:32.281 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.281 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.282 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.283 DEBUG:boto:Params: {} 2022-09-13T07:39:32.283 DEBUG:boto:Token: None 2022-09-13T07:39:32.284 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.285 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6854bded85e8867-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.289 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.290 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.291 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.291 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.292 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.292 DEBUG:boto:Method: GET 2022-09-13T07:39:32.293 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.293 DEBUG:boto:Data: 2022-09-13T07:39:32.294 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.294 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.295 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.295 DEBUG:boto:Params: {} 2022-09-13T07:39:32.296 DEBUG:boto:Token: None 2022-09-13T07:39:32.297 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.297 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d038432bfaa654c9-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.302 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.303 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.303 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.304 DEBUG:boto:Method: HEAD 2022-09-13T07:39:32.304 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:32.305 DEBUG:boto:Data: 2022-09-13T07:39:32.305 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.306 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.306 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.307 DEBUG:boto:Params: {} 2022-09-13T07:39:32.307 DEBUG:boto:Token: None 2022-09-13T07:39:32.308 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj 2022-09-13T07:39:32.308 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk= 2022-09-13T07:39:32.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk=', 'Content-Length': '0'} 2022-09-13T07:39:32.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003badc6f52d0be3fa-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.313 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.315 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.316 DEBUG:boto:Method: HEAD 2022-09-13T07:39:32.316 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:32.317 DEBUG:boto:Data: 2022-09-13T07:39:32.318 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.319 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.320 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.320 DEBUG:boto:Params: {} 2022-09-13T07:39:32.321 DEBUG:boto:Token: None 2022-09-13T07:39:32.322 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj 2022-09-13T07:39:32.323 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk= 2022-09-13T07:39:32.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk=', 'Content-Length': '0'} 2022-09-13T07:39:32.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9d1cfa2f3e9297f-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:32.331 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.332 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.333 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.334 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.335 DEBUG:boto:Method: GET 2022-09-13T07:39:32.336 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.336 DEBUG:boto:Data: 2022-09-13T07:39:32.337 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.338 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.339 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.340 DEBUG:boto:Params: {} 2022-09-13T07:39:32.341 DEBUG:boto:Token: None 2022-09-13T07:39:32.342 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.343 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU= 2022-09-13T07:39:32.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU=', 'Content-Length': '0'} 2022-09-13T07:39:32.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007477f65c8dc04bf2-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.348 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.349 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.350 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.351 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.352 DEBUG:boto:Method: GET 2022-09-13T07:39:32.353 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.354 DEBUG:boto:Data: 2022-09-13T07:39:32.355 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.355 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.356 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.357 DEBUG:boto:Params: {} 2022-09-13T07:39:32.358 DEBUG:boto:Token: None 2022-09-13T07:39:32.358 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.359 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU= 2022-09-13T07:39:32.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU=', 'Content-Length': '0'} 2022-09-13T07:39:32.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cee5e258714a1bcb-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.362 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.363 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.363 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.364 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.364 DEBUG:boto:Method: GET 2022-09-13T07:39:32.365 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.365 DEBUG:boto:Data: 2022-09-13T07:39:32.366 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.367 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.367 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.367 DEBUG:boto:Params: {} 2022-09-13T07:39:32.368 DEBUG:boto:Token: None 2022-09-13T07:39:32.369 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.369 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bb24403ba582067-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.414 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.415 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.416 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.416 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.417 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.417 DEBUG:boto:Method: GET 2022-09-13T07:39:32.418 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.418 DEBUG:boto:Data: 2022-09-13T07:39:32.419 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.419 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.420 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.420 DEBUG:boto:Params: {} 2022-09-13T07:39:32.421 DEBUG:boto:Token: None 2022-09-13T07:39:32.422 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.422 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb9cf6c50da9f82b-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.427 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:32.428 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.429 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.429 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.430 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.430 DEBUG:boto:Method: GET 2022-09-13T07:39:32.431 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.431 DEBUG:boto:Data: 2022-09-13T07:39:32.432 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.432 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.433 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.433 DEBUG:boto:Params: {} 2022-09-13T07:39:32.434 DEBUG:boto:Token: None 2022-09-13T07:39:32.435 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.435 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IpWlLyHwDCYqrurBiiLfRzA/Y8c= 2022-09-13T07:39:32.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IpWlLyHwDCYqrurBiiLfRzA/Y8c=', 'Content-Length': '0'} 2022-09-13T07:39:32.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9a5b8cf577c1409-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.439 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.439 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.440 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.440 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.441 DEBUG:boto:Method: GET 2022-09-13T07:39:32.442 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.442 DEBUG:boto:Data: 2022-09-13T07:39:32.443 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.443 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.444 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.444 DEBUG:boto:Params: {} 2022-09-13T07:39:32.445 DEBUG:boto:Token: None 2022-09-13T07:39:32.445 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.446 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IpWlLyHwDCYqrurBiiLfRzA/Y8c= 2022-09-13T07:39:32.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IpWlLyHwDCYqrurBiiLfRzA/Y8c=', 'Content-Length': '0'} 2022-09-13T07:39:32.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d6509f2b34c990b-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.490 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.491 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.492 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.493 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.494 DEBUG:boto:Method: HEAD 2022-09-13T07:39:32.495 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.495 DEBUG:boto:Data: 2022-09-13T07:39:32.496 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.497 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.498 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.499 DEBUG:boto:Params: {} 2022-09-13T07:39:32.500 DEBUG:boto:Token: None 2022-09-13T07:39:32.501 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.502 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3u8xtXHe4utrNivScHF/wdo6VPA= 2022-09-13T07:39:32.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3u8xtXHe4utrNivScHF/wdo6VPA=', 'Content-Length': '0'} 2022-09-13T07:39:32.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000beea2c3e8457d83b-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.507 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.507 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.508 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.509 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.510 DEBUG:boto:Method: HEAD 2022-09-13T07:39:32.511 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.512 DEBUG:boto:Data: 2022-09-13T07:39:32.513 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.513 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.514 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.515 DEBUG:boto:Params: {} 2022-09-13T07:39:32.516 DEBUG:boto:Token: None 2022-09-13T07:39:32.517 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.518 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3u8xtXHe4utrNivScHF/wdo6VPA= 2022-09-13T07:39:32.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3u8xtXHe4utrNivScHF/wdo6VPA=', 'Content-Length': '0'} 2022-09-13T07:39:32.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9e66060ffed4118-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:32.523 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.523 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.524 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.524 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.525 DEBUG:boto:Method: GET 2022-09-13T07:39:32.526 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.527 DEBUG:boto:Data: 2022-09-13T07:39:32.527 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.528 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.529 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.530 DEBUG:boto:Params: {} 2022-09-13T07:39:32.531 DEBUG:boto:Token: None 2022-09-13T07:39:32.532 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.533 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IpWlLyHwDCYqrurBiiLfRzA/Y8c= 2022-09-13T07:39:32.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IpWlLyHwDCYqrurBiiLfRzA/Y8c=', 'Content-Length': '0'} 2022-09-13T07:39:32.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6876fd8b1657a35-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.538 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.539 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.540 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.541 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.542 DEBUG:boto:Method: GET 2022-09-13T07:39:32.543 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.544 DEBUG:boto:Data: 2022-09-13T07:39:32.545 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.545 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.546 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.547 DEBUG:boto:Params: {} 2022-09-13T07:39:32.548 DEBUG:boto:Token: None 2022-09-13T07:39:32.549 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:32.550 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IpWlLyHwDCYqrurBiiLfRzA/Y8c= 2022-09-13T07:39:32.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IpWlLyHwDCYqrurBiiLfRzA/Y8c=', 'Content-Length': '0'} 2022-09-13T07:39:32.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e483cc1b7f8ecefb-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.554 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.555 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.555 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.556 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.557 DEBUG:boto:Method: GET 2022-09-13T07:39:32.558 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.559 DEBUG:boto:Data: 2022-09-13T07:39:32.560 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.560 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.561 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.561 DEBUG:boto:Params: {} 2022-09-13T07:39:32.562 DEBUG:boto:Token: None 2022-09-13T07:39:32.563 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.563 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a840afa9bcbb588-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.568 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.569 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.569 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.570 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.571 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.571 DEBUG:boto:Method: GET 2022-09-13T07:39:32.572 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.572 DEBUG:boto:Data: 2022-09-13T07:39:32.573 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.573 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.574 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.574 DEBUG:boto:Params: {} 2022-09-13T07:39:32.575 DEBUG:boto:Token: None 2022-09-13T07:39:32.575 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.576 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d1d3ae4ff96e6c4-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.582 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.583 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.585 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.586 DEBUG:boto:Method: HEAD 2022-09-13T07:39:32.587 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:32.588 DEBUG:boto:Data: 2022-09-13T07:39:32.589 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.590 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.591 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.592 DEBUG:boto:Params: {} 2022-09-13T07:39:32.593 DEBUG:boto:Token: None 2022-09-13T07:39:32.594 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj 2022-09-13T07:39:32.595 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk= 2022-09-13T07:39:32.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk=', 'Content-Length': '0'} 2022-09-13T07:39:32.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e4ce9fd7dfab948-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.601 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.602 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.603 DEBUG:boto:Method: HEAD 2022-09-13T07:39:32.604 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:32.605 DEBUG:boto:Data: 2022-09-13T07:39:32.605 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.606 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.607 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.608 DEBUG:boto:Params: {} 2022-09-13T07:39:32.609 DEBUG:boto:Token: None 2022-09-13T07:39:32.610 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj 2022-09-13T07:39:32.612 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk= 2022-09-13T07:39:32.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk=', 'Content-Length': '0'} 2022-09-13T07:39:32.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b618d742595808b2-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:32.619 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.620 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.621 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.622 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.623 DEBUG:boto:Method: GET 2022-09-13T07:39:32.624 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.625 DEBUG:boto:Data: 2022-09-13T07:39:32.626 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.626 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.628 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.629 DEBUG:boto:Params: {} 2022-09-13T07:39:32.630 DEBUG:boto:Token: None 2022-09-13T07:39:32.632 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.632 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU= 2022-09-13T07:39:32.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU=', 'Content-Length': '0'} 2022-09-13T07:39:32.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000253c633f2a3e96b3-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.682 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.683 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.683 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.684 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.685 DEBUG:boto:Method: GET 2022-09-13T07:39:32.685 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.686 DEBUG:boto:Data: 2022-09-13T07:39:32.686 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.687 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.688 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.688 DEBUG:boto:Params: {} 2022-09-13T07:39:32.689 DEBUG:boto:Token: None 2022-09-13T07:39:32.689 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.690 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU= 2022-09-13T07:39:32.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU=', 'Content-Length': '0'} 2022-09-13T07:39:32.693 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d3919402a6fbaf5-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.694 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.695 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.695 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.696 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.696 DEBUG:boto:Method: GET 2022-09-13T07:39:32.697 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.697 DEBUG:boto:Data: 2022-09-13T07:39:32.698 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.698 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.699 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.699 DEBUG:boto:Params: {} 2022-09-13T07:39:32.700 DEBUG:boto:Token: None 2022-09-13T07:39:32.700 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.701 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a27bab124058ac2-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.705 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.706 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.707 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.707 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.708 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.708 DEBUG:boto:Method: GET 2022-09-13T07:39:32.709 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.709 DEBUG:boto:Data: 2022-09-13T07:39:32.710 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.710 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.711 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.711 DEBUG:boto:Params: {} 2022-09-13T07:39:32.712 DEBUG:boto:Token: None 2022-09-13T07:39:32.713 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.713 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e2f54ced215c7a1-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.718 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:32.719 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.720 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.720 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.721 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.722 DEBUG:boto:Method: GET 2022-09-13T07:39:32.722 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.723 DEBUG:boto:Data: 2022-09-13T07:39:32.724 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.724 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.725 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.725 DEBUG:boto:Params: {} 2022-09-13T07:39:32.726 DEBUG:boto:Token: None 2022-09-13T07:39:32.726 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.727 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Sa63iiuLg8xa+J1M+w95vepl0Zk= 2022-09-13T07:39:32.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Sa63iiuLg8xa+J1M+w95vepl0Zk=', 'Content-Length': '0'} 2022-09-13T07:39:32.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d949d47e9717976f-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.731 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.731 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.732 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.732 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.733 DEBUG:boto:Method: GET 2022-09-13T07:39:32.733 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.734 DEBUG:boto:Data: 2022-09-13T07:39:32.734 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.735 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.735 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.736 DEBUG:boto:Params: {} 2022-09-13T07:39:32.737 DEBUG:boto:Token: None 2022-09-13T07:39:32.737 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.738 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Sa63iiuLg8xa+J1M+w95vepl0Zk= 2022-09-13T07:39:32.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Sa63iiuLg8xa+J1M+w95vepl0Zk=', 'Content-Length': '0'} 2022-09-13T07:39:32.741 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e532720490f10008-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.742 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.742 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.743 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.743 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.744 DEBUG:boto:Method: HEAD 2022-09-13T07:39:32.744 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.745 DEBUG:boto:Data: 2022-09-13T07:39:32.745 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.746 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.746 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.747 DEBUG:boto:Params: {} 2022-09-13T07:39:32.747 DEBUG:boto:Token: None 2022-09-13T07:39:32.748 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.748 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oenTy4kdWWh6TPxyrWYOn9eJ//E= 2022-09-13T07:39:32.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oenTy4kdWWh6TPxyrWYOn9eJ//E=', 'Content-Length': '0'} 2022-09-13T07:39:32.751 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086ec063f476e6125-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.751 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.752 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.752 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.753 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.753 DEBUG:boto:Method: HEAD 2022-09-13T07:39:32.754 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.754 DEBUG:boto:Data: 2022-09-13T07:39:32.755 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.755 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.756 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.756 DEBUG:boto:Params: {} 2022-09-13T07:39:32.757 DEBUG:boto:Token: None 2022-09-13T07:39:32.757 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.758 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oenTy4kdWWh6TPxyrWYOn9eJ//E= 2022-09-13T07:39:32.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oenTy4kdWWh6TPxyrWYOn9eJ//E=', 'Content-Length': '0'} 2022-09-13T07:39:32.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002977a54bb62c1827-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:32.762 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.762 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.763 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.763 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.764 DEBUG:boto:Method: GET 2022-09-13T07:39:32.764 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.765 DEBUG:boto:Data: 2022-09-13T07:39:32.765 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.766 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.766 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.767 DEBUG:boto:Params: {} 2022-09-13T07:39:32.767 DEBUG:boto:Token: None 2022-09-13T07:39:32.768 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.769 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Sa63iiuLg8xa+J1M+w95vepl0Zk= 2022-09-13T07:39:32.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Sa63iiuLg8xa+J1M+w95vepl0Zk=', 'Content-Length': '0'} 2022-09-13T07:39:32.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a2a8aa2153ebe61-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.772 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.773 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.773 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.774 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.774 DEBUG:boto:Method: GET 2022-09-13T07:39:32.775 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.775 DEBUG:boto:Data: 2022-09-13T07:39:32.776 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.776 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.777 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.777 DEBUG:boto:Params: {} 2022-09-13T07:39:32.778 DEBUG:boto:Token: None 2022-09-13T07:39:32.778 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:32.779 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Sa63iiuLg8xa+J1M+w95vepl0Zk= 2022-09-13T07:39:32.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Sa63iiuLg8xa+J1M+w95vepl0Zk=', 'Content-Length': '0'} 2022-09-13T07:39:32.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c66e785a6388340a-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.782 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.783 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.783 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.784 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.784 DEBUG:boto:Method: GET 2022-09-13T07:39:32.785 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.785 DEBUG:boto:Data: 2022-09-13T07:39:32.786 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.786 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.787 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.787 DEBUG:boto:Params: {} 2022-09-13T07:39:32.788 DEBUG:boto:Token: None 2022-09-13T07:39:32.788 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.789 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028b65fba17b486d2-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.793 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.794 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.795 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.795 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.795 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.796 DEBUG:boto:Method: GET 2022-09-13T07:39:32.796 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.797 DEBUG:boto:Data: 2022-09-13T07:39:32.797 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.798 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.798 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.799 DEBUG:boto:Params: {} 2022-09-13T07:39:32.799 DEBUG:boto:Token: None 2022-09-13T07:39:32.800 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.801 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa90f4ce7b2f2b56-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.805 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.806 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.806 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.807 DEBUG:boto:Method: HEAD 2022-09-13T07:39:32.807 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:32.808 DEBUG:boto:Data: 2022-09-13T07:39:32.808 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.809 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.809 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.810 DEBUG:boto:Params: {} 2022-09-13T07:39:32.810 DEBUG:boto:Token: None 2022-09-13T07:39:32.811 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj 2022-09-13T07:39:32.811 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk= 2022-09-13T07:39:32.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk=', 'Content-Length': '0'} 2022-09-13T07:39:32.815 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002df6b06a0595ef15-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.816 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.816 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.817 DEBUG:boto:Method: HEAD 2022-09-13T07:39:32.817 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:32.818 DEBUG:boto:Data: 2022-09-13T07:39:32.818 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.819 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.819 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.820 DEBUG:boto:Params: {} 2022-09-13T07:39:32.820 DEBUG:boto:Token: None 2022-09-13T07:39:32.821 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj 2022-09-13T07:39:32.821 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk= 2022-09-13T07:39:32.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TZ2kzebZEnhaKG6LVtTCxthSWYk=', 'Content-Length': '0'} 2022-09-13T07:39:32.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049d9830f26f1e1ae-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:32.827 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.827 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.828 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.828 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.829 DEBUG:boto:Method: GET 2022-09-13T07:39:32.829 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.830 DEBUG:boto:Data: 2022-09-13T07:39:32.830 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.831 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.831 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.832 DEBUG:boto:Params: {} 2022-09-13T07:39:32.832 DEBUG:boto:Token: None 2022-09-13T07:39:32.833 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.834 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU= 2022-09-13T07:39:32.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU=', 'Content-Length': '0'} 2022-09-13T07:39:32.836 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000146fce051ebe730e-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.882 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.883 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.884 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.884 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.885 DEBUG:boto:Method: GET 2022-09-13T07:39:32.885 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.886 DEBUG:boto:Data: 2022-09-13T07:39:32.887 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.887 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.888 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.888 DEBUG:boto:Params: {} 2022-09-13T07:39:32.889 DEBUG:boto:Token: None 2022-09-13T07:39:32.889 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:32.890 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU= 2022-09-13T07:39:32.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:h5F0k4VD8xREpVB1/t9DcpSXPsU=', 'Content-Length': '0'} 2022-09-13T07:39:32.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4aa6acf6dedd4fc-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.894 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.894 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.895 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.895 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.896 DEBUG:boto:Method: GET 2022-09-13T07:39:32.896 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.897 DEBUG:boto:Data: 2022-09-13T07:39:32.897 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.898 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.898 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.899 DEBUG:boto:Params: {} 2022-09-13T07:39:32.899 DEBUG:boto:Token: None 2022-09-13T07:39:32.900 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.901 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee982347d20bd1af-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.905 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.906 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:32.907 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:32.907 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.908 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.908 DEBUG:boto:Method: GET 2022-09-13T07:39:32.909 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:32.909 DEBUG:boto:Data: 2022-09-13T07:39:32.910 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.910 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.911 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.911 DEBUG:boto:Params: {} 2022-09-13T07:39:32.912 DEBUG:boto:Token: None 2022-09-13T07:39:32.913 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/ 2022-09-13T07:39:32.914 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU= 2022-09-13T07:39:32.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gn1Kf7r59rqzt4Rk4NW+2+0c/jU=', 'Content-Length': '0'} 2022-09-13T07:39:32.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000415bc2f4d94b86c-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.920 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:32.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:32.922 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.923 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.923 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:32.924 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:32.924 DEBUG:boto:Method: GET 2022-09-13T07:39:32.925 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:32.926 DEBUG:boto:Data: 2022-09-13T07:39:32.926 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.927 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:32.928 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.929 DEBUG:boto:Params: {} 2022-09-13T07:39:32.930 DEBUG:boto:Token: None 2022-09-13T07:39:32.931 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:32.932 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xrQKn6mTv0R6mNl0hB9fGR8mwJM= 2022-09-13T07:39:32.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xrQKn6mTv0R6mNl0hB9fGR8mwJM=', 'Content-Length': '0'} 2022-09-13T07:39:32.936 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9bbd496ae1838e3-00632033b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.937 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.938 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.938 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:32.939 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:32.939 DEBUG:boto:Method: GET 2022-09-13T07:39:32.940 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:32.940 DEBUG:boto:Data: 2022-09-13T07:39:32.941 DEBUG:boto:Headers: {} 2022-09-13T07:39:32.941 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:32.942 DEBUG:boto:Port: 8000 2022-09-13T07:39:32.942 DEBUG:boto:Params: {} 2022-09-13T07:39:32.943 DEBUG:boto:Token: None 2022-09-13T07:39:32.944 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:32 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:32.945 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xrQKn6mTv0R6mNl0hB9fGR8mwJM= 2022-09-13T07:39:32.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xrQKn6mTv0R6mNl0hB9fGR8mwJM=', 'Content-Length': '0'} 2022-09-13T07:39:32.950 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b4c941173e80ecc-00632033b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:32.993 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:32.994 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:32.995 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:32.996 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:32.997 DEBUG:boto:Method: HEAD 2022-09-13T07:39:32.998 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:32.999 DEBUG:boto:Data: 2022-09-13T07:39:33.000 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.000 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.001 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.002 DEBUG:boto:Params: {} 2022-09-13T07:39:33.003 DEBUG:boto:Token: None 2022-09-13T07:39:33.005 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:33.006 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VTkwT+EySCsww2fLExUPx+Woes8= 2022-09-13T07:39:33.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VTkwT+EySCsww2fLExUPx+Woes8=', 'Content-Length': '0'} 2022-09-13T07:39:33.009 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de932b7225c99b64-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.010 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.011 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.011 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:33.012 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:33.012 DEBUG:boto:Method: HEAD 2022-09-13T07:39:33.013 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:33.013 DEBUG:boto:Data: 2022-09-13T07:39:33.014 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.014 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.015 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.015 DEBUG:boto:Params: {} 2022-09-13T07:39:33.016 DEBUG:boto:Token: None 2022-09-13T07:39:33.016 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:33.017 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VTkwT+EySCsww2fLExUPx+Woes8= 2022-09-13T07:39:33.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VTkwT+EySCsww2fLExUPx+Woes8=', 'Content-Length': '0'} 2022-09-13T07:39:33.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076da0ade6e98c1f8-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:33.021 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.022 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.022 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:33.023 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:33.024 DEBUG:boto:Method: GET 2022-09-13T07:39:33.024 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:33.025 DEBUG:boto:Data: 2022-09-13T07:39:33.026 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.027 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.027 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.028 DEBUG:boto:Params: {} 2022-09-13T07:39:33.029 DEBUG:boto:Token: None 2022-09-13T07:39:33.029 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:33.030 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NmxYJqgioJEQmlxBdFzujwGtrac= 2022-09-13T07:39:33.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NmxYJqgioJEQmlxBdFzujwGtrac=', 'Content-Length': '0'} 2022-09-13T07:39:33.033 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000887d7093a4e05cae-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.034 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.035 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.035 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:33.036 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:33.037 DEBUG:boto:Method: GET 2022-09-13T07:39:33.038 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:33.038 DEBUG:boto:Data: 2022-09-13T07:39:33.039 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.039 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.040 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.040 DEBUG:boto:Params: {} 2022-09-13T07:39:33.041 DEBUG:boto:Token: None 2022-09-13T07:39:33.041 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:33.042 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NmxYJqgioJEQmlxBdFzujwGtrac= 2022-09-13T07:39:33.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NmxYJqgioJEQmlxBdFzujwGtrac=', 'Content-Length': '0'} 2022-09-13T07:39:33.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ec63e539734ca90-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.046 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:33.046 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:33.047 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.047 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.048 DEBUG:boto:Method: GET 2022-09-13T07:39:33.048 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.049 DEBUG:boto:Data: 2022-09-13T07:39:33.049 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.051 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.052 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.052 DEBUG:boto:Params: {} 2022-09-13T07:39:33.054 DEBUG:boto:Token: None 2022-09-13T07:39:33.054 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/ 2022-09-13T07:39:33.055 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY= 2022-09-13T07:39:33.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY=', 'Content-Length': '0'} 2022-09-13T07:39:33.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ade53e389320778-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.061 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:33.062 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:33.063 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:33.063 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.064 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.064 DEBUG:boto:Method: GET 2022-09-13T07:39:33.065 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.065 DEBUG:boto:Data: 2022-09-13T07:39:33.066 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.066 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.067 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.068 DEBUG:boto:Params: {} 2022-09-13T07:39:33.068 DEBUG:boto:Token: None 2022-09-13T07:39:33.069 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/ 2022-09-13T07:39:33.069 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY= 2022-09-13T07:39:33.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY=', 'Content-Length': '0'} 2022-09-13T07:39:33.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f093fea3c6195050-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.074 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:33.075 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.075 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.076 DEBUG:boto:Method: HEAD 2022-09-13T07:39:33.076 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:33.077 DEBUG:boto:Data: 2022-09-13T07:39:33.077 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.078 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.078 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.079 DEBUG:boto:Params: {} 2022-09-13T07:39:33.079 DEBUG:boto:Token: None 2022-09-13T07:39:33.080 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj 2022-09-13T07:39:33.080 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860= 2022-09-13T07:39:33.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860=', 'Content-Length': '0'} 2022-09-13T07:39:33.084 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfa99a0ad0dd8dad-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.085 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.085 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.086 DEBUG:boto:Method: HEAD 2022-09-13T07:39:33.087 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:33.087 DEBUG:boto:Data: 2022-09-13T07:39:33.088 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.088 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.089 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.089 DEBUG:boto:Params: {} 2022-09-13T07:39:33.090 DEBUG:boto:Token: None 2022-09-13T07:39:33.090 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj 2022-09-13T07:39:33.091 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860= 2022-09-13T07:39:33.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860=', 'Content-Length': '0'} 2022-09-13T07:39:33.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e395309b21031961-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:33.099 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.099 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.100 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.101 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.102 DEBUG:boto:Method: GET 2022-09-13T07:39:33.103 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.103 DEBUG:boto:Data: 2022-09-13T07:39:33.104 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.105 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.105 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.106 DEBUG:boto:Params: {} 2022-09-13T07:39:33.106 DEBUG:boto:Token: None 2022-09-13T07:39:33.107 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.107 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc= 2022-09-13T07:39:33.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc=', 'Content-Length': '0'} 2022-09-13T07:39:33.110 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031ed4e96ddd0b328-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.111 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.112 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.112 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.113 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.113 DEBUG:boto:Method: GET 2022-09-13T07:39:33.114 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.114 DEBUG:boto:Data: 2022-09-13T07:39:33.115 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.115 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.116 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.116 DEBUG:boto:Params: {} 2022-09-13T07:39:33.117 DEBUG:boto:Token: None 2022-09-13T07:39:33.118 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.119 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc= 2022-09-13T07:39:33.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc=', 'Content-Length': '0'} 2022-09-13T07:39:33.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002620941e3136d513-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.122 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:33.123 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:33.123 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.124 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.124 DEBUG:boto:Method: GET 2022-09-13T07:39:33.125 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.125 DEBUG:boto:Data: 2022-09-13T07:39:33.126 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.127 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.128 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.129 DEBUG:boto:Params: {} 2022-09-13T07:39:33.130 DEBUG:boto:Token: None 2022-09-13T07:39:33.131 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/ 2022-09-13T07:39:33.132 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY= 2022-09-13T07:39:33.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY=', 'Content-Length': '0'} 2022-09-13T07:39:33.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ad715f3f269ea5d-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.178 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:33.179 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:33.180 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:33.181 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.182 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.183 DEBUG:boto:Method: GET 2022-09-13T07:39:33.183 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.184 DEBUG:boto:Data: 2022-09-13T07:39:33.185 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.186 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.187 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.188 DEBUG:boto:Params: {} 2022-09-13T07:39:33.189 DEBUG:boto:Token: None 2022-09-13T07:39:33.190 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/ 2022-09-13T07:39:33.191 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY= 2022-09-13T07:39:33.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY=', 'Content-Length': '0'} 2022-09-13T07:39:33.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000745bdb3323ba3556-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.197 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:33.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:33.199 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.200 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.200 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.201 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.201 DEBUG:boto:Method: GET 2022-09-13T07:39:33.202 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.202 DEBUG:boto:Data: 2022-09-13T07:39:33.203 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.203 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.204 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.205 DEBUG:boto:Params: {} 2022-09-13T07:39:33.205 DEBUG:boto:Token: None 2022-09-13T07:39:33.206 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.206 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7qCr8becZyI+HI71yMMzV/HAjPU= 2022-09-13T07:39:33.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7qCr8becZyI+HI71yMMzV/HAjPU=', 'Content-Length': '0'} 2022-09-13T07:39:33.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb065172ab84e5ce-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.210 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.211 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.212 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.212 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.213 DEBUG:boto:Method: GET 2022-09-13T07:39:33.213 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.214 DEBUG:boto:Data: 2022-09-13T07:39:33.214 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.215 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.215 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.216 DEBUG:boto:Params: {} 2022-09-13T07:39:33.216 DEBUG:boto:Token: None 2022-09-13T07:39:33.217 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.218 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7qCr8becZyI+HI71yMMzV/HAjPU= 2022-09-13T07:39:33.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7qCr8becZyI+HI71yMMzV/HAjPU=', 'Content-Length': '0'} 2022-09-13T07:39:33.222 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005685cb94f9db776a-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.266 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.267 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.268 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.268 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.269 DEBUG:boto:Method: HEAD 2022-09-13T07:39:33.270 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.271 DEBUG:boto:Data: 2022-09-13T07:39:33.272 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.273 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.274 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.275 DEBUG:boto:Params: {} 2022-09-13T07:39:33.276 DEBUG:boto:Token: None 2022-09-13T07:39:33.277 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.278 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wN732OFxXotrwpo+Wy9lVYMRcwA= 2022-09-13T07:39:33.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wN732OFxXotrwpo+Wy9lVYMRcwA=', 'Content-Length': '0'} 2022-09-13T07:39:33.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000667c69ce4a1cb0e9-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.281 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.282 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.282 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.283 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.284 DEBUG:boto:Method: HEAD 2022-09-13T07:39:33.285 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.285 DEBUG:boto:Data: 2022-09-13T07:39:33.286 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.286 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.287 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.287 DEBUG:boto:Params: {} 2022-09-13T07:39:33.288 DEBUG:boto:Token: None 2022-09-13T07:39:33.289 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.289 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wN732OFxXotrwpo+Wy9lVYMRcwA= 2022-09-13T07:39:33.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wN732OFxXotrwpo+Wy9lVYMRcwA=', 'Content-Length': '0'} 2022-09-13T07:39:33.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cebbe6c8e96cf5a5-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:33.294 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.294 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.295 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.295 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.296 DEBUG:boto:Method: GET 2022-09-13T07:39:33.296 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.297 DEBUG:boto:Data: 2022-09-13T07:39:33.297 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.298 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.299 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.299 DEBUG:boto:Params: {} 2022-09-13T07:39:33.300 DEBUG:boto:Token: None 2022-09-13T07:39:33.300 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.301 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7qCr8becZyI+HI71yMMzV/HAjPU= 2022-09-13T07:39:33.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7qCr8becZyI+HI71yMMzV/HAjPU=', 'Content-Length': '0'} 2022-09-13T07:39:33.304 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f845fd8314f921aa-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.304 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.305 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.306 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.306 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.307 DEBUG:boto:Method: GET 2022-09-13T07:39:33.307 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.308 DEBUG:boto:Data: 2022-09-13T07:39:33.308 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.309 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.309 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.310 DEBUG:boto:Params: {} 2022-09-13T07:39:33.310 DEBUG:boto:Token: None 2022-09-13T07:39:33.311 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:33.311 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7qCr8becZyI+HI71yMMzV/HAjPU= 2022-09-13T07:39:33.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7qCr8becZyI+HI71yMMzV/HAjPU=', 'Content-Length': '0'} 2022-09-13T07:39:33.316 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e633472f48435ed8-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.317 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:33.318 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:33.319 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.320 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.321 DEBUG:boto:Method: GET 2022-09-13T07:39:33.322 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.323 DEBUG:boto:Data: 2022-09-13T07:39:33.324 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.325 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.326 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.327 DEBUG:boto:Params: {} 2022-09-13T07:39:33.328 DEBUG:boto:Token: None 2022-09-13T07:39:33.329 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/ 2022-09-13T07:39:33.330 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY= 2022-09-13T07:39:33.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY=', 'Content-Length': '0'} 2022-09-13T07:39:33.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000feaa83745e927488-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.336 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:33.337 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:33.338 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:33.339 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.340 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.340 DEBUG:boto:Method: GET 2022-09-13T07:39:33.341 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.341 DEBUG:boto:Data: 2022-09-13T07:39:33.342 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.342 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.343 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.343 DEBUG:boto:Params: {} 2022-09-13T07:39:33.344 DEBUG:boto:Token: None 2022-09-13T07:39:33.345 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/ 2022-09-13T07:39:33.345 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY= 2022-09-13T07:39:33.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY=', 'Content-Length': '0'} 2022-09-13T07:39:33.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3ec63ddd972f273-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.350 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:33.351 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.351 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.352 DEBUG:boto:Method: HEAD 2022-09-13T07:39:33.352 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:33.353 DEBUG:boto:Data: 2022-09-13T07:39:33.353 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.354 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.354 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.355 DEBUG:boto:Params: {} 2022-09-13T07:39:33.356 DEBUG:boto:Token: None 2022-09-13T07:39:33.356 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj 2022-09-13T07:39:33.357 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860= 2022-09-13T07:39:33.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860=', 'Content-Length': '0'} 2022-09-13T07:39:33.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d40d8597b748cff0-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.362 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.362 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.363 DEBUG:boto:Method: HEAD 2022-09-13T07:39:33.363 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:33.364 DEBUG:boto:Data: 2022-09-13T07:39:33.364 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.365 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.365 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.366 DEBUG:boto:Params: {} 2022-09-13T07:39:33.366 DEBUG:boto:Token: None 2022-09-13T07:39:33.367 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj 2022-09-13T07:39:33.368 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860= 2022-09-13T07:39:33.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860=', 'Content-Length': '0'} 2022-09-13T07:39:33.372 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1580c11b1a3243e-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:33.373 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.374 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.374 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.375 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.375 DEBUG:boto:Method: GET 2022-09-13T07:39:33.376 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.376 DEBUG:boto:Data: 2022-09-13T07:39:33.377 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.377 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.378 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.378 DEBUG:boto:Params: {} 2022-09-13T07:39:33.379 DEBUG:boto:Token: None 2022-09-13T07:39:33.380 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.380 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc= 2022-09-13T07:39:33.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc=', 'Content-Length': '0'} 2022-09-13T07:39:33.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002e4de1ff3d29838-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.384 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.384 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.385 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.385 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.386 DEBUG:boto:Method: GET 2022-09-13T07:39:33.386 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.387 DEBUG:boto:Data: 2022-09-13T07:39:33.388 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.388 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.389 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.389 DEBUG:boto:Params: {} 2022-09-13T07:39:33.390 DEBUG:boto:Token: None 2022-09-13T07:39:33.390 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.391 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc= 2022-09-13T07:39:33.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc=', 'Content-Length': '0'} 2022-09-13T07:39:33.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e412fbaea44ac805-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.395 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:33.396 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:33.396 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.397 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.397 DEBUG:boto:Method: GET 2022-09-13T07:39:33.398 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.399 DEBUG:boto:Data: 2022-09-13T07:39:33.399 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.400 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.400 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.401 DEBUG:boto:Params: {} 2022-09-13T07:39:33.401 DEBUG:boto:Token: None 2022-09-13T07:39:33.402 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/ 2022-09-13T07:39:33.403 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY= 2022-09-13T07:39:33.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY=', 'Content-Length': '0'} 2022-09-13T07:39:33.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd236451c893c4e6-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.454 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:33.455 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:33.456 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:33.456 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.457 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.457 DEBUG:boto:Method: GET 2022-09-13T07:39:33.458 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.459 DEBUG:boto:Data: 2022-09-13T07:39:33.459 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.460 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.461 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.461 DEBUG:boto:Params: {} 2022-09-13T07:39:33.462 DEBUG:boto:Token: None 2022-09-13T07:39:33.463 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/ 2022-09-13T07:39:33.463 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY= 2022-09-13T07:39:33.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY=', 'Content-Length': '0'} 2022-09-13T07:39:33.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1e3022d95bead4e-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.468 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:33.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:33.470 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.471 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.472 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.473 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.473 DEBUG:boto:Method: GET 2022-09-13T07:39:33.474 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.474 DEBUG:boto:Data: 2022-09-13T07:39:33.475 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.475 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.476 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.476 DEBUG:boto:Params: {} 2022-09-13T07:39:33.477 DEBUG:boto:Token: None 2022-09-13T07:39:33.478 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.479 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jGqExzdntFgY43kMnlGbdxC1qNA= 2022-09-13T07:39:33.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jGqExzdntFgY43kMnlGbdxC1qNA=', 'Content-Length': '0'} 2022-09-13T07:39:33.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a9a8b8ee6173355-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.483 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.483 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.484 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.484 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.485 DEBUG:boto:Method: GET 2022-09-13T07:39:33.485 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.486 DEBUG:boto:Data: 2022-09-13T07:39:33.487 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.487 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.488 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.488 DEBUG:boto:Params: {} 2022-09-13T07:39:33.489 DEBUG:boto:Token: None 2022-09-13T07:39:33.490 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.490 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jGqExzdntFgY43kMnlGbdxC1qNA= 2022-09-13T07:39:33.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jGqExzdntFgY43kMnlGbdxC1qNA=', 'Content-Length': '0'} 2022-09-13T07:39:33.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056d53509b3c6d28e-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.538 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.539 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.539 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.540 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.540 DEBUG:boto:Method: HEAD 2022-09-13T07:39:33.541 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.542 DEBUG:boto:Data: 2022-09-13T07:39:33.543 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.544 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.545 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.546 DEBUG:boto:Params: {} 2022-09-13T07:39:33.547 DEBUG:boto:Token: None 2022-09-13T07:39:33.548 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.549 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fBdfH/zTVTOoUNrRNwnrRsPEQV0= 2022-09-13T07:39:33.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fBdfH/zTVTOoUNrRNwnrRsPEQV0=', 'Content-Length': '0'} 2022-09-13T07:39:33.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff1909228ced64eb-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.554 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.554 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.555 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.556 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.557 DEBUG:boto:Method: HEAD 2022-09-13T07:39:33.558 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.559 DEBUG:boto:Data: 2022-09-13T07:39:33.560 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.561 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.562 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.563 DEBUG:boto:Params: {} 2022-09-13T07:39:33.564 DEBUG:boto:Token: None 2022-09-13T07:39:33.565 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.566 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fBdfH/zTVTOoUNrRNwnrRsPEQV0= 2022-09-13T07:39:33.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fBdfH/zTVTOoUNrRNwnrRsPEQV0=', 'Content-Length': '0'} 2022-09-13T07:39:33.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ae9266b160f816c-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:33.570 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.571 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.572 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.572 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.573 DEBUG:boto:Method: GET 2022-09-13T07:39:33.573 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.574 DEBUG:boto:Data: 2022-09-13T07:39:33.574 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.575 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.575 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.576 DEBUG:boto:Params: {} 2022-09-13T07:39:33.576 DEBUG:boto:Token: None 2022-09-13T07:39:33.577 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.577 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jGqExzdntFgY43kMnlGbdxC1qNA= 2022-09-13T07:39:33.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jGqExzdntFgY43kMnlGbdxC1qNA=', 'Content-Length': '0'} 2022-09-13T07:39:33.580 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004402bb1d8e8b7b62-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.581 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.582 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.582 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.583 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.583 DEBUG:boto:Method: GET 2022-09-13T07:39:33.584 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.584 DEBUG:boto:Data: 2022-09-13T07:39:33.585 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.585 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.586 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.586 DEBUG:boto:Params: {} 2022-09-13T07:39:33.587 DEBUG:boto:Token: None 2022-09-13T07:39:33.587 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:33.588 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jGqExzdntFgY43kMnlGbdxC1qNA= 2022-09-13T07:39:33.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jGqExzdntFgY43kMnlGbdxC1qNA=', 'Content-Length': '0'} 2022-09-13T07:39:33.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4594fd255ad971f-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.591 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:33.592 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:33.592 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.593 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.594 DEBUG:boto:Method: GET 2022-09-13T07:39:33.594 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.595 DEBUG:boto:Data: 2022-09-13T07:39:33.595 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.596 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.596 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.597 DEBUG:boto:Params: {} 2022-09-13T07:39:33.597 DEBUG:boto:Token: None 2022-09-13T07:39:33.598 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/ 2022-09-13T07:39:33.598 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY= 2022-09-13T07:39:33.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY=', 'Content-Length': '0'} 2022-09-13T07:39:33.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fb6fcff4e5c4289-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.602 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:33.603 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:33.604 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:33.604 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.605 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.606 DEBUG:boto:Method: GET 2022-09-13T07:39:33.606 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.607 DEBUG:boto:Data: 2022-09-13T07:39:33.607 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.608 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.608 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.609 DEBUG:boto:Params: {} 2022-09-13T07:39:33.609 DEBUG:boto:Token: None 2022-09-13T07:39:33.610 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/ 2022-09-13T07:39:33.611 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY= 2022-09-13T07:39:33.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY=', 'Content-Length': '0'} 2022-09-13T07:39:33.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e30d20ab59ac5fc-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.615 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:33.616 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.617 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.617 DEBUG:boto:Method: HEAD 2022-09-13T07:39:33.618 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:33.618 DEBUG:boto:Data: 2022-09-13T07:39:33.619 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.619 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.620 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.620 DEBUG:boto:Params: {} 2022-09-13T07:39:33.621 DEBUG:boto:Token: None 2022-09-13T07:39:33.622 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj 2022-09-13T07:39:33.622 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860= 2022-09-13T07:39:33.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860=', 'Content-Length': '0'} 2022-09-13T07:39:33.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084b145f9120ef8a0-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.628 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.629 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.630 DEBUG:boto:Method: HEAD 2022-09-13T07:39:33.630 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:33.631 DEBUG:boto:Data: 2022-09-13T07:39:33.631 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.632 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.632 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.633 DEBUG:boto:Params: {} 2022-09-13T07:39:33.634 DEBUG:boto:Token: None 2022-09-13T07:39:33.635 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj 2022-09-13T07:39:33.636 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860= 2022-09-13T07:39:33.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860=', 'Content-Length': '0'} 2022-09-13T07:39:33.640 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e303519051cdd7e-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:33.641 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.642 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.642 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.644 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.645 DEBUG:boto:Method: GET 2022-09-13T07:39:33.647 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.647 DEBUG:boto:Data: 2022-09-13T07:39:33.648 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.648 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.649 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.649 DEBUG:boto:Params: {} 2022-09-13T07:39:33.651 DEBUG:boto:Token: None 2022-09-13T07:39:33.652 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.652 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc= 2022-09-13T07:39:33.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc=', 'Content-Length': '0'} 2022-09-13T07:39:33.655 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099f7b70dc4497d85-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.656 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.657 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.657 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.658 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.658 DEBUG:boto:Method: GET 2022-09-13T07:39:33.659 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.659 DEBUG:boto:Data: 2022-09-13T07:39:33.660 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.660 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.661 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.661 DEBUG:boto:Params: {} 2022-09-13T07:39:33.662 DEBUG:boto:Token: None 2022-09-13T07:39:33.663 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.663 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc= 2022-09-13T07:39:33.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc=', 'Content-Length': '0'} 2022-09-13T07:39:33.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a53de9de7297fea-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.714 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:33.715 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:33.715 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.716 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.716 DEBUG:boto:Method: GET 2022-09-13T07:39:33.717 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.717 DEBUG:boto:Data: 2022-09-13T07:39:33.718 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.719 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.719 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.720 DEBUG:boto:Params: {} 2022-09-13T07:39:33.720 DEBUG:boto:Token: None 2022-09-13T07:39:33.721 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/ 2022-09-13T07:39:33.722 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY= 2022-09-13T07:39:33.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY=', 'Content-Length': '0'} 2022-09-13T07:39:33.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038aa2dc2769c7f24-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.726 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:33.727 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:33.728 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:33.728 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.729 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.729 DEBUG:boto:Method: GET 2022-09-13T07:39:33.730 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.730 DEBUG:boto:Data: 2022-09-13T07:39:33.731 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.732 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.733 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.733 DEBUG:boto:Params: {} 2022-09-13T07:39:33.734 DEBUG:boto:Token: None 2022-09-13T07:39:33.735 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/ 2022-09-13T07:39:33.735 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY= 2022-09-13T07:39:33.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY=', 'Content-Length': '0'} 2022-09-13T07:39:33.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005812e1b71ef744d5-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.740 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:33.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:33.741 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.741 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.742 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.742 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.743 DEBUG:boto:Method: GET 2022-09-13T07:39:33.743 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.744 DEBUG:boto:Data: 2022-09-13T07:39:33.744 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.745 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.746 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.746 DEBUG:boto:Params: {} 2022-09-13T07:39:33.747 DEBUG:boto:Token: None 2022-09-13T07:39:33.747 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.748 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:m/w4qDEzXAgDzZSYw6wITPVLjbM= 2022-09-13T07:39:33.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:m/w4qDEzXAgDzZSYw6wITPVLjbM=', 'Content-Length': '0'} 2022-09-13T07:39:33.751 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eaf8aa312afc0851-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.752 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.752 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.753 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.753 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.754 DEBUG:boto:Method: GET 2022-09-13T07:39:33.754 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.755 DEBUG:boto:Data: 2022-09-13T07:39:33.756 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.756 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.757 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.757 DEBUG:boto:Params: {} 2022-09-13T07:39:33.758 DEBUG:boto:Token: None 2022-09-13T07:39:33.759 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.759 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:m/w4qDEzXAgDzZSYw6wITPVLjbM= 2022-09-13T07:39:33.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:m/w4qDEzXAgDzZSYw6wITPVLjbM=', 'Content-Length': '0'} 2022-09-13T07:39:33.763 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001dd85014d1824555-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.764 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.765 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.766 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.766 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.766 DEBUG:boto:Method: HEAD 2022-09-13T07:39:33.767 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.767 DEBUG:boto:Data: 2022-09-13T07:39:33.768 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.768 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.769 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.770 DEBUG:boto:Params: {} 2022-09-13T07:39:33.770 DEBUG:boto:Token: None 2022-09-13T07:39:33.771 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.771 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GgSKhJpgiU2zanKaM0Gob56x9/0= 2022-09-13T07:39:33.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GgSKhJpgiU2zanKaM0Gob56x9/0=', 'Content-Length': '0'} 2022-09-13T07:39:33.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8f3b1f9ca877d3b-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.775 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.775 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.776 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.776 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.777 DEBUG:boto:Method: HEAD 2022-09-13T07:39:33.777 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.778 DEBUG:boto:Data: 2022-09-13T07:39:33.778 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.779 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.779 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.780 DEBUG:boto:Params: {} 2022-09-13T07:39:33.780 DEBUG:boto:Token: None 2022-09-13T07:39:33.781 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.781 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GgSKhJpgiU2zanKaM0Gob56x9/0= 2022-09-13T07:39:33.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GgSKhJpgiU2zanKaM0Gob56x9/0=', 'Content-Length': '0'} 2022-09-13T07:39:33.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba64e99b84b8e5d4-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:33.785 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.786 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.786 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.787 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.787 DEBUG:boto:Method: GET 2022-09-13T07:39:33.788 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.788 DEBUG:boto:Data: 2022-09-13T07:39:33.789 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.789 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.790 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.791 DEBUG:boto:Params: {} 2022-09-13T07:39:33.791 DEBUG:boto:Token: None 2022-09-13T07:39:33.792 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.792 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:m/w4qDEzXAgDzZSYw6wITPVLjbM= 2022-09-13T07:39:33.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:m/w4qDEzXAgDzZSYw6wITPVLjbM=', 'Content-Length': '0'} 2022-09-13T07:39:33.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000aaaccacfc1bfd99-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.838 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.839 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.839 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.841 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.842 DEBUG:boto:Method: GET 2022-09-13T07:39:33.843 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.844 DEBUG:boto:Data: 2022-09-13T07:39:33.844 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.845 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.846 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.847 DEBUG:boto:Params: {} 2022-09-13T07:39:33.848 DEBUG:boto:Token: None 2022-09-13T07:39:33.849 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:33.851 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:m/w4qDEzXAgDzZSYw6wITPVLjbM= 2022-09-13T07:39:33.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:m/w4qDEzXAgDzZSYw6wITPVLjbM=', 'Content-Length': '0'} 2022-09-13T07:39:33.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016fda310efc32f25-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.857 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:33.858 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:33.859 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.860 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.861 DEBUG:boto:Method: GET 2022-09-13T07:39:33.862 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.863 DEBUG:boto:Data: 2022-09-13T07:39:33.864 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.865 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.866 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.867 DEBUG:boto:Params: {} 2022-09-13T07:39:33.868 DEBUG:boto:Token: None 2022-09-13T07:39:33.869 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/ 2022-09-13T07:39:33.870 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY= 2022-09-13T07:39:33.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY=', 'Content-Length': '0'} 2022-09-13T07:39:33.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fb49b9e35d8907e-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.876 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:33.877 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:33.878 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:33.879 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.880 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.880 DEBUG:boto:Method: GET 2022-09-13T07:39:33.881 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.882 DEBUG:boto:Data: 2022-09-13T07:39:33.883 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.884 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.885 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.886 DEBUG:boto:Params: {} 2022-09-13T07:39:33.887 DEBUG:boto:Token: None 2022-09-13T07:39:33.888 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/ 2022-09-13T07:39:33.888 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY= 2022-09-13T07:39:33.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fWvoSR17OaBbzqFH1FHuz6a//aY=', 'Content-Length': '0'} 2022-09-13T07:39:33.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005460091c886d94e1-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.893 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:33.893 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.894 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.894 DEBUG:boto:Method: HEAD 2022-09-13T07:39:33.895 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:33.895 DEBUG:boto:Data: 2022-09-13T07:39:33.896 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.896 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.897 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.898 DEBUG:boto:Params: {} 2022-09-13T07:39:33.899 DEBUG:boto:Token: None 2022-09-13T07:39:33.899 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj 2022-09-13T07:39:33.900 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860= 2022-09-13T07:39:33.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860=', 'Content-Length': '0'} 2022-09-13T07:39:33.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066cc62a7cd91ec9a-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.904 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.905 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.905 DEBUG:boto:Method: HEAD 2022-09-13T07:39:33.906 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:33.907 DEBUG:boto:Data: 2022-09-13T07:39:33.908 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.908 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.908 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.909 DEBUG:boto:Params: {} 2022-09-13T07:39:33.909 DEBUG:boto:Token: None 2022-09-13T07:39:33.910 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj 2022-09-13T07:39:33.911 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860= 2022-09-13T07:39:33.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tTwXYjWo+xeMqlgGYpZyOae+860=', 'Content-Length': '0'} 2022-09-13T07:39:33.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000896b8c9142072d69-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:33.916 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.917 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.917 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.918 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.918 DEBUG:boto:Method: GET 2022-09-13T07:39:33.919 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.919 DEBUG:boto:Data: 2022-09-13T07:39:33.920 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.920 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:33.921 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.922 DEBUG:boto:Params: {} 2022-09-13T07:39:33.923 DEBUG:boto:Token: None 2022-09-13T07:39:33.923 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.924 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc= 2022-09-13T07:39:33.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc=', 'Content-Length': '0'} 2022-09-13T07:39:33.927 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bc743665f173cdc-00632033b5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.971 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:33.974 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:33.975 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.976 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.977 DEBUG:boto:Method: GET 2022-09-13T07:39:33.978 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.980 DEBUG:boto:Data: 2022-09-13T07:39:33.981 DEBUG:boto:Headers: {} 2022-09-13T07:39:33.982 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:33.983 DEBUG:boto:Port: 8000 2022-09-13T07:39:33.984 DEBUG:boto:Params: {} 2022-09-13T07:39:33.985 DEBUG:boto:Token: None 2022-09-13T07:39:33.986 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:33 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:33.988 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc= 2022-09-13T07:39:33.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:33 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qBjBj7D7kn5/I6JWuzRpxHOCUEc=', 'Content-Length': '0'} 2022-09-13T07:39:33.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000956efb3bfca6b82a-00632033b5-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:33.998 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:33.998 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:33.999 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:33.999 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.000 DEBUG:boto:Method: GET 2022-09-13T07:39:34.001 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.002 DEBUG:boto:Data: 2022-09-13T07:39:34.003 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.003 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.004 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.005 DEBUG:boto:Params: {} 2022-09-13T07:39:34.006 DEBUG:boto:Token: None 2022-09-13T07:39:34.006 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.007 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o= 2022-09-13T07:39:34.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o=', 'Content-Length': '0'} 2022-09-13T07:39:34.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000292fff29f772edaf-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.012 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.014 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.015 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.016 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.017 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.018 DEBUG:boto:Method: GET 2022-09-13T07:39:34.019 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.020 DEBUG:boto:Data: 2022-09-13T07:39:34.021 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.022 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.023 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.024 DEBUG:boto:Params: {} 2022-09-13T07:39:34.024 DEBUG:boto:Token: None 2022-09-13T07:39:34.025 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.025 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o= 2022-09-13T07:39:34.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o=', 'Content-Length': '0'} 2022-09-13T07:39:34.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b60473d764938aaf-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.031 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:34.034 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.035 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.036 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.037 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.038 DEBUG:boto:Method: GET 2022-09-13T07:39:34.039 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.040 DEBUG:boto:Data: 2022-09-13T07:39:34.040 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.041 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.042 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.043 DEBUG:boto:Params: {} 2022-09-13T07:39:34.044 DEBUG:boto:Token: None 2022-09-13T07:39:34.046 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.047 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:o5E/XH0Y3oQZ96rarVNz46QOfVk= 2022-09-13T07:39:34.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:o5E/XH0Y3oQZ96rarVNz46QOfVk=', 'Content-Length': '0'} 2022-09-13T07:39:34.051 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a126192e13e883c8-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.052 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.053 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.054 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.055 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.056 DEBUG:boto:Method: GET 2022-09-13T07:39:34.057 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.058 DEBUG:boto:Data: 2022-09-13T07:39:34.060 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.061 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.061 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.062 DEBUG:boto:Params: {} 2022-09-13T07:39:34.064 DEBUG:boto:Token: None 2022-09-13T07:39:34.065 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.066 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:o5E/XH0Y3oQZ96rarVNz46QOfVk= 2022-09-13T07:39:34.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:o5E/XH0Y3oQZ96rarVNz46QOfVk=', 'Content-Length': '0'} 2022-09-13T07:39:34.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ae8bd77203cb15c-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.072 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.073 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.074 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.075 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.076 DEBUG:boto:Method: HEAD 2022-09-13T07:39:34.077 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.078 DEBUG:boto:Data: 2022-09-13T07:39:34.079 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.080 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.081 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.081 DEBUG:boto:Params: {} 2022-09-13T07:39:34.082 DEBUG:boto:Token: None 2022-09-13T07:39:34.084 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.086 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zAJ/oee2zhyfQGDwbZWfVDDhAcA= 2022-09-13T07:39:34.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zAJ/oee2zhyfQGDwbZWfVDDhAcA=', 'Content-Length': '0'} 2022-09-13T07:39:34.090 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000478f664190c8636e-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.091 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.092 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.093 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.094 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.095 DEBUG:boto:Method: HEAD 2022-09-13T07:39:34.096 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.097 DEBUG:boto:Data: 2022-09-13T07:39:34.098 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.099 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.100 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.101 DEBUG:boto:Params: {} 2022-09-13T07:39:34.102 DEBUG:boto:Token: None 2022-09-13T07:39:34.103 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.104 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zAJ/oee2zhyfQGDwbZWfVDDhAcA= 2022-09-13T07:39:34.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zAJ/oee2zhyfQGDwbZWfVDDhAcA=', 'Content-Length': '0'} 2022-09-13T07:39:34.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000828ddd04551d824-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:34.111 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.112 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.113 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.114 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.115 DEBUG:boto:Method: GET 2022-09-13T07:39:34.116 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.117 DEBUG:boto:Data: 2022-09-13T07:39:34.118 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.119 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.120 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.122 DEBUG:boto:Params: {} 2022-09-13T07:39:34.123 DEBUG:boto:Token: None 2022-09-13T07:39:34.124 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.125 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:o5E/XH0Y3oQZ96rarVNz46QOfVk= 2022-09-13T07:39:34.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:o5E/XH0Y3oQZ96rarVNz46QOfVk=', 'Content-Length': '0'} 2022-09-13T07:39:34.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cd54b423b2b8a6b-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.174 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.175 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.176 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.177 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.178 DEBUG:boto:Method: GET 2022-09-13T07:39:34.179 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.179 DEBUG:boto:Data: 2022-09-13T07:39:34.180 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.181 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.181 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.182 DEBUG:boto:Params: {} 2022-09-13T07:39:34.183 DEBUG:boto:Token: None 2022-09-13T07:39:34.183 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:34.184 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:o5E/XH0Y3oQZ96rarVNz46QOfVk= 2022-09-13T07:39:34.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:o5E/XH0Y3oQZ96rarVNz46QOfVk=', 'Content-Length': '0'} 2022-09-13T07:39:34.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ec22c0d20abb38b-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.188 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.188 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.189 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.189 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.190 DEBUG:boto:Method: GET 2022-09-13T07:39:34.191 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.192 DEBUG:boto:Data: 2022-09-13T07:39:34.192 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.193 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.194 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.195 DEBUG:boto:Params: {} 2022-09-13T07:39:34.195 DEBUG:boto:Token: None 2022-09-13T07:39:34.196 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.196 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o= 2022-09-13T07:39:34.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o=', 'Content-Length': '0'} 2022-09-13T07:39:34.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005330ef7216ade8e1-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.201 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.203 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.204 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.204 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.205 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.205 DEBUG:boto:Method: GET 2022-09-13T07:39:34.206 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.207 DEBUG:boto:Data: 2022-09-13T07:39:34.208 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.208 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.209 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.210 DEBUG:boto:Params: {} 2022-09-13T07:39:34.211 DEBUG:boto:Token: None 2022-09-13T07:39:34.212 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.213 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o= 2022-09-13T07:39:34.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o=', 'Content-Length': '0'} 2022-09-13T07:39:34.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f270d82dd34420f-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.258 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.259 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.260 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.260 DEBUG:boto:Method: HEAD 2022-09-13T07:39:34.261 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:34.261 DEBUG:boto:Data: 2022-09-13T07:39:34.262 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.262 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.263 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.263 DEBUG:boto:Params: {} 2022-09-13T07:39:34.264 DEBUG:boto:Token: None 2022-09-13T07:39:34.264 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj 2022-09-13T07:39:34.265 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SG3sQ7+7CS5OqshagBt2f1cHFXo= 2022-09-13T07:39:34.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SG3sQ7+7CS5OqshagBt2f1cHFXo=', 'Content-Length': '0'} 2022-09-13T07:39:34.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005030cc9526ccbaeb-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.270 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.270 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.271 DEBUG:boto:Method: HEAD 2022-09-13T07:39:34.271 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:34.272 DEBUG:boto:Data: 2022-09-13T07:39:34.272 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.273 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.273 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.274 DEBUG:boto:Params: {} 2022-09-13T07:39:34.274 DEBUG:boto:Token: None 2022-09-13T07:39:34.275 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj 2022-09-13T07:39:34.275 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SG3sQ7+7CS5OqshagBt2f1cHFXo= 2022-09-13T07:39:34.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SG3sQ7+7CS5OqshagBt2f1cHFXo=', 'Content-Length': '0'} 2022-09-13T07:39:34.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001089b2683e195e52-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:34.281 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.281 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.282 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.282 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.283 DEBUG:boto:Method: GET 2022-09-13T07:39:34.283 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.284 DEBUG:boto:Data: 2022-09-13T07:39:34.284 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.285 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.285 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.286 DEBUG:boto:Params: {} 2022-09-13T07:39:34.287 DEBUG:boto:Token: None 2022-09-13T07:39:34.288 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.289 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Fos0hon8fOXhefHsuYVvxKa1z7U= 2022-09-13T07:39:34.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Fos0hon8fOXhefHsuYVvxKa1z7U=', 'Content-Length': '0'} 2022-09-13T07:39:34.293 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007347011ea81aaccd-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.293 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.294 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.295 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.296 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.297 DEBUG:boto:Method: GET 2022-09-13T07:39:34.298 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.299 DEBUG:boto:Data: 2022-09-13T07:39:34.300 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.301 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.302 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.302 DEBUG:boto:Params: {} 2022-09-13T07:39:34.303 DEBUG:boto:Token: None 2022-09-13T07:39:34.304 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.305 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Fos0hon8fOXhefHsuYVvxKa1z7U= 2022-09-13T07:39:34.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Fos0hon8fOXhefHsuYVvxKa1z7U=', 'Content-Length': '0'} 2022-09-13T07:39:34.308 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016bd86ed8ed4e220-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.309 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.309 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.310 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.311 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.311 DEBUG:boto:Method: GET 2022-09-13T07:39:34.312 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.312 DEBUG:boto:Data: 2022-09-13T07:39:34.313 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.313 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.314 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.315 DEBUG:boto:Params: {} 2022-09-13T07:39:34.315 DEBUG:boto:Token: None 2022-09-13T07:39:34.316 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.317 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o= 2022-09-13T07:39:34.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o=', 'Content-Length': '0'} 2022-09-13T07:39:34.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e7a1ec6ad0cc960-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.321 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.322 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.323 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.324 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.325 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.326 DEBUG:boto:Method: GET 2022-09-13T07:39:34.327 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.328 DEBUG:boto:Data: 2022-09-13T07:39:34.328 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.329 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.329 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.330 DEBUG:boto:Params: {} 2022-09-13T07:39:34.330 DEBUG:boto:Token: None 2022-09-13T07:39:34.331 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.332 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o= 2022-09-13T07:39:34.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o=', 'Content-Length': '0'} 2022-09-13T07:39:34.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d5922f198a7564d-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.336 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:34.337 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.338 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.338 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.339 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.339 DEBUG:boto:Method: GET 2022-09-13T07:39:34.340 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.340 DEBUG:boto:Data: 2022-09-13T07:39:34.341 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.341 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.342 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.342 DEBUG:boto:Params: {} 2022-09-13T07:39:34.343 DEBUG:boto:Token: None 2022-09-13T07:39:34.343 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.344 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IjfMYH6DYNks/cHSrRoalR2yJgg= 2022-09-13T07:39:34.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IjfMYH6DYNks/cHSrRoalR2yJgg=', 'Content-Length': '0'} 2022-09-13T07:39:34.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5ba32d8efdb1908-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.348 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.348 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.349 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.349 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.350 DEBUG:boto:Method: GET 2022-09-13T07:39:34.350 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.351 DEBUG:boto:Data: 2022-09-13T07:39:34.351 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.352 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.352 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.353 DEBUG:boto:Params: {} 2022-09-13T07:39:34.353 DEBUG:boto:Token: None 2022-09-13T07:39:34.354 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.354 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IjfMYH6DYNks/cHSrRoalR2yJgg= 2022-09-13T07:39:34.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IjfMYH6DYNks/cHSrRoalR2yJgg=', 'Content-Length': '0'} 2022-09-13T07:39:34.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2842ecd5e2bc644-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.398 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.399 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.399 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.400 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.400 DEBUG:boto:Method: HEAD 2022-09-13T07:39:34.401 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.402 DEBUG:boto:Data: 2022-09-13T07:39:34.402 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.403 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.403 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.404 DEBUG:boto:Params: {} 2022-09-13T07:39:34.405 DEBUG:boto:Token: None 2022-09-13T07:39:34.405 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.406 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GumCWZeYjNoyZMuIdREd/O/BXgM= 2022-09-13T07:39:34.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GumCWZeYjNoyZMuIdREd/O/BXgM=', 'Content-Length': '0'} 2022-09-13T07:39:34.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000333c54f6aeebac4-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.410 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.410 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.411 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.412 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.413 DEBUG:boto:Method: HEAD 2022-09-13T07:39:34.414 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.415 DEBUG:boto:Data: 2022-09-13T07:39:34.416 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.417 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.418 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.419 DEBUG:boto:Params: {} 2022-09-13T07:39:34.420 DEBUG:boto:Token: None 2022-09-13T07:39:34.421 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.422 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GumCWZeYjNoyZMuIdREd/O/BXgM= 2022-09-13T07:39:34.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GumCWZeYjNoyZMuIdREd/O/BXgM=', 'Content-Length': '0'} 2022-09-13T07:39:34.425 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a1197164966eedd-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:34.427 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.427 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.428 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.428 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.429 DEBUG:boto:Method: GET 2022-09-13T07:39:34.429 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.430 DEBUG:boto:Data: 2022-09-13T07:39:34.430 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.431 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.431 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.432 DEBUG:boto:Params: {} 2022-09-13T07:39:34.433 DEBUG:boto:Token: None 2022-09-13T07:39:34.433 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.434 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IjfMYH6DYNks/cHSrRoalR2yJgg= 2022-09-13T07:39:34.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IjfMYH6DYNks/cHSrRoalR2yJgg=', 'Content-Length': '0'} 2022-09-13T07:39:34.437 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ec009630851b6fd-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.438 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.438 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.439 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.439 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.440 DEBUG:boto:Method: GET 2022-09-13T07:39:34.441 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.441 DEBUG:boto:Data: 2022-09-13T07:39:34.442 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.442 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.443 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.444 DEBUG:boto:Params: {} 2022-09-13T07:39:34.444 DEBUG:boto:Token: None 2022-09-13T07:39:34.445 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:34.446 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IjfMYH6DYNks/cHSrRoalR2yJgg= 2022-09-13T07:39:34.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IjfMYH6DYNks/cHSrRoalR2yJgg=', 'Content-Length': '0'} 2022-09-13T07:39:34.448 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aae65031419efda6-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.449 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.450 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.450 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.451 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.451 DEBUG:boto:Method: GET 2022-09-13T07:39:34.452 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.453 DEBUG:boto:Data: 2022-09-13T07:39:34.453 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.454 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.454 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.455 DEBUG:boto:Params: {} 2022-09-13T07:39:34.455 DEBUG:boto:Token: None 2022-09-13T07:39:34.456 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.456 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o= 2022-09-13T07:39:34.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o=', 'Content-Length': '0'} 2022-09-13T07:39:34.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065099b749f553d0e-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.461 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.462 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.462 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.463 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.463 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.464 DEBUG:boto:Method: GET 2022-09-13T07:39:34.464 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.465 DEBUG:boto:Data: 2022-09-13T07:39:34.465 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.466 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.466 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.467 DEBUG:boto:Params: {} 2022-09-13T07:39:34.468 DEBUG:boto:Token: None 2022-09-13T07:39:34.469 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.469 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o= 2022-09-13T07:39:34.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o=', 'Content-Length': '0'} 2022-09-13T07:39:34.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069822305978ba48e-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.474 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.475 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.476 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.477 DEBUG:boto:Method: HEAD 2022-09-13T07:39:34.478 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:34.479 DEBUG:boto:Data: 2022-09-13T07:39:34.480 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.480 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.481 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.482 DEBUG:boto:Params: {} 2022-09-13T07:39:34.483 DEBUG:boto:Token: None 2022-09-13T07:39:34.485 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj 2022-09-13T07:39:34.486 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SG3sQ7+7CS5OqshagBt2f1cHFXo= 2022-09-13T07:39:34.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SG3sQ7+7CS5OqshagBt2f1cHFXo=', 'Content-Length': '0'} 2022-09-13T07:39:34.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bbf673017231aa3-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.492 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.493 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.494 DEBUG:boto:Method: HEAD 2022-09-13T07:39:34.495 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:34.496 DEBUG:boto:Data: 2022-09-13T07:39:34.497 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.498 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.499 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.500 DEBUG:boto:Params: {} 2022-09-13T07:39:34.501 DEBUG:boto:Token: None 2022-09-13T07:39:34.502 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj 2022-09-13T07:39:34.503 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SG3sQ7+7CS5OqshagBt2f1cHFXo= 2022-09-13T07:39:34.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SG3sQ7+7CS5OqshagBt2f1cHFXo=', 'Content-Length': '0'} 2022-09-13T07:39:34.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cb3eb6fdd7dc40f-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:34.511 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.512 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.512 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.513 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.513 DEBUG:boto:Method: GET 2022-09-13T07:39:34.514 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.514 DEBUG:boto:Data: 2022-09-13T07:39:34.515 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.516 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.517 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.518 DEBUG:boto:Params: {} 2022-09-13T07:39:34.519 DEBUG:boto:Token: None 2022-09-13T07:39:34.520 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.521 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Fos0hon8fOXhefHsuYVvxKa1z7U= 2022-09-13T07:39:34.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Fos0hon8fOXhefHsuYVvxKa1z7U=', 'Content-Length': '0'} 2022-09-13T07:39:34.525 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b004c74153624ef-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.525 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.526 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.527 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.527 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.528 DEBUG:boto:Method: GET 2022-09-13T07:39:34.528 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.529 DEBUG:boto:Data: 2022-09-13T07:39:34.529 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.530 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.530 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.531 DEBUG:boto:Params: {} 2022-09-13T07:39:34.531 DEBUG:boto:Token: None 2022-09-13T07:39:34.532 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.533 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Fos0hon8fOXhefHsuYVvxKa1z7U= 2022-09-13T07:39:34.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Fos0hon8fOXhefHsuYVvxKa1z7U=', 'Content-Length': '0'} 2022-09-13T07:39:34.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077335426b8e80ac9-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.536 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.537 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.538 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.539 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.540 DEBUG:boto:Method: GET 2022-09-13T07:39:34.541 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.542 DEBUG:boto:Data: 2022-09-13T07:39:34.543 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.544 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.545 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.545 DEBUG:boto:Params: {} 2022-09-13T07:39:34.546 DEBUG:boto:Token: None 2022-09-13T07:39:34.547 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.548 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o= 2022-09-13T07:39:34.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o=', 'Content-Length': '0'} 2022-09-13T07:39:34.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000288d29d37535fe87-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.553 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.554 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.555 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.556 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.557 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.558 DEBUG:boto:Method: GET 2022-09-13T07:39:34.559 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.560 DEBUG:boto:Data: 2022-09-13T07:39:34.561 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.561 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.562 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.563 DEBUG:boto:Params: {} 2022-09-13T07:39:34.564 DEBUG:boto:Token: None 2022-09-13T07:39:34.564 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.565 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o= 2022-09-13T07:39:34.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o=', 'Content-Length': '0'} 2022-09-13T07:39:34.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000352ed84dd83b2a65-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.569 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:34.571 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.571 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.572 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.572 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.573 DEBUG:boto:Method: GET 2022-09-13T07:39:34.573 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.574 DEBUG:boto:Data: 2022-09-13T07:39:34.574 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.575 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.575 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.576 DEBUG:boto:Params: {} 2022-09-13T07:39:34.576 DEBUG:boto:Token: None 2022-09-13T07:39:34.577 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.577 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xwfH7AYEYa2Kcg5hfJrq6tzjGzw= 2022-09-13T07:39:34.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xwfH7AYEYa2Kcg5hfJrq6tzjGzw=', 'Content-Length': '0'} 2022-09-13T07:39:34.580 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd0ef609ce3d0c01-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.622 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.623 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.623 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.624 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.624 DEBUG:boto:Method: GET 2022-09-13T07:39:34.625 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.625 DEBUG:boto:Data: 2022-09-13T07:39:34.626 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.626 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.627 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.627 DEBUG:boto:Params: {} 2022-09-13T07:39:34.628 DEBUG:boto:Token: None 2022-09-13T07:39:34.629 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.629 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xwfH7AYEYa2Kcg5hfJrq6tzjGzw= 2022-09-13T07:39:34.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xwfH7AYEYa2Kcg5hfJrq6tzjGzw=', 'Content-Length': '0'} 2022-09-13T07:39:34.633 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfa707de7f52874d-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.634 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.634 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.635 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.635 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.636 DEBUG:boto:Method: HEAD 2022-09-13T07:39:34.636 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.637 DEBUG:boto:Data: 2022-09-13T07:39:34.637 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.638 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.638 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.639 DEBUG:boto:Params: {} 2022-09-13T07:39:34.639 DEBUG:boto:Token: None 2022-09-13T07:39:34.640 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.640 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/rBhdjY+Atv7JF2sogX82fHFhkg= 2022-09-13T07:39:34.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/rBhdjY+Atv7JF2sogX82fHFhkg=', 'Content-Length': '0'} 2022-09-13T07:39:34.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048a449f811589acb-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.643 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.644 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.644 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.645 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.645 DEBUG:boto:Method: HEAD 2022-09-13T07:39:34.646 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.646 DEBUG:boto:Data: 2022-09-13T07:39:34.647 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.647 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.648 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.648 DEBUG:boto:Params: {} 2022-09-13T07:39:34.649 DEBUG:boto:Token: None 2022-09-13T07:39:34.650 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.650 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/rBhdjY+Atv7JF2sogX82fHFhkg= 2022-09-13T07:39:34.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/rBhdjY+Atv7JF2sogX82fHFhkg=', 'Content-Length': '0'} 2022-09-13T07:39:34.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000064c58cee3839f4c-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:34.654 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.654 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.655 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.655 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.656 DEBUG:boto:Method: GET 2022-09-13T07:39:34.656 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.657 DEBUG:boto:Data: 2022-09-13T07:39:34.657 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.658 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.658 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.659 DEBUG:boto:Params: {} 2022-09-13T07:39:34.660 DEBUG:boto:Token: None 2022-09-13T07:39:34.660 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.661 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xwfH7AYEYa2Kcg5hfJrq6tzjGzw= 2022-09-13T07:39:34.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xwfH7AYEYa2Kcg5hfJrq6tzjGzw=', 'Content-Length': '0'} 2022-09-13T07:39:34.664 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3a0da7bfc500568-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.664 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.665 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.665 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.666 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.666 DEBUG:boto:Method: GET 2022-09-13T07:39:34.667 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.667 DEBUG:boto:Data: 2022-09-13T07:39:34.668 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.668 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.669 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.669 DEBUG:boto:Params: {} 2022-09-13T07:39:34.670 DEBUG:boto:Token: None 2022-09-13T07:39:34.671 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:34.671 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xwfH7AYEYa2Kcg5hfJrq6tzjGzw= 2022-09-13T07:39:34.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xwfH7AYEYa2Kcg5hfJrq6tzjGzw=', 'Content-Length': '0'} 2022-09-13T07:39:34.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d3b12b80da7e068-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.675 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.676 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.676 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.677 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.678 DEBUG:boto:Method: GET 2022-09-13T07:39:34.679 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.680 DEBUG:boto:Data: 2022-09-13T07:39:34.681 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.682 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.683 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.684 DEBUG:boto:Params: {} 2022-09-13T07:39:34.686 DEBUG:boto:Token: None 2022-09-13T07:39:34.687 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.688 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o= 2022-09-13T07:39:34.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o=', 'Content-Length': '0'} 2022-09-13T07:39:34.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026afbf7f1dcc02b0-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.694 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.695 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.696 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.697 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.698 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.698 DEBUG:boto:Method: GET 2022-09-13T07:39:34.699 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.699 DEBUG:boto:Data: 2022-09-13T07:39:34.700 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.700 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.701 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.701 DEBUG:boto:Params: {} 2022-09-13T07:39:34.702 DEBUG:boto:Token: None 2022-09-13T07:39:34.702 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.703 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o= 2022-09-13T07:39:34.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o=', 'Content-Length': '0'} 2022-09-13T07:39:34.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090dfd0556850595c-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.750 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.752 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.752 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.753 DEBUG:boto:Method: HEAD 2022-09-13T07:39:34.753 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:34.754 DEBUG:boto:Data: 2022-09-13T07:39:34.755 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.755 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.756 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.756 DEBUG:boto:Params: {} 2022-09-13T07:39:34.757 DEBUG:boto:Token: None 2022-09-13T07:39:34.758 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj 2022-09-13T07:39:34.759 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SG3sQ7+7CS5OqshagBt2f1cHFXo= 2022-09-13T07:39:34.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SG3sQ7+7CS5OqshagBt2f1cHFXo=', 'Content-Length': '0'} 2022-09-13T07:39:34.763 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053db67c28385e255-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.765 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.765 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.766 DEBUG:boto:Method: HEAD 2022-09-13T07:39:34.767 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:34.768 DEBUG:boto:Data: 2022-09-13T07:39:34.768 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.769 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.770 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.771 DEBUG:boto:Params: {} 2022-09-13T07:39:34.772 DEBUG:boto:Token: None 2022-09-13T07:39:34.773 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj 2022-09-13T07:39:34.774 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SG3sQ7+7CS5OqshagBt2f1cHFXo= 2022-09-13T07:39:34.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SG3sQ7+7CS5OqshagBt2f1cHFXo=', 'Content-Length': '0'} 2022-09-13T07:39:34.779 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e510ec385fc8ad48-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:34.781 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.782 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.783 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.784 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.785 DEBUG:boto:Method: GET 2022-09-13T07:39:34.785 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.786 DEBUG:boto:Data: 2022-09-13T07:39:34.787 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.788 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.788 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.789 DEBUG:boto:Params: {} 2022-09-13T07:39:34.790 DEBUG:boto:Token: None 2022-09-13T07:39:34.791 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.792 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Fos0hon8fOXhefHsuYVvxKa1z7U= 2022-09-13T07:39:34.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Fos0hon8fOXhefHsuYVvxKa1z7U=', 'Content-Length': '0'} 2022-09-13T07:39:34.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000775cf8dd143ac98f-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.842 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:34.843 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:34.844 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.844 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.845 DEBUG:boto:Method: GET 2022-09-13T07:39:34.845 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.846 DEBUG:boto:Data: 2022-09-13T07:39:34.846 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.847 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.847 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.848 DEBUG:boto:Params: {} 2022-09-13T07:39:34.848 DEBUG:boto:Token: None 2022-09-13T07:39:34.849 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:34.850 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Fos0hon8fOXhefHsuYVvxKa1z7U= 2022-09-13T07:39:34.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Fos0hon8fOXhefHsuYVvxKa1z7U=', 'Content-Length': '0'} 2022-09-13T07:39:34.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000363f324e05032f7c-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.855 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.855 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.856 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.856 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.857 DEBUG:boto:Method: GET 2022-09-13T07:39:34.857 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.858 DEBUG:boto:Data: 2022-09-13T07:39:34.858 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.859 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:34.859 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.860 DEBUG:boto:Params: {} 2022-09-13T07:39:34.861 DEBUG:boto:Token: None 2022-09-13T07:39:34.861 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.862 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o= 2022-09-13T07:39:34.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o=', 'Content-Length': '0'} 2022-09-13T07:39:34.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005795b81aee7e693d-00632033b6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.866 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.867 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.868 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.868 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.869 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.869 DEBUG:boto:Method: GET 2022-09-13T07:39:34.870 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:34.870 DEBUG:boto:Data: 2022-09-13T07:39:34.871 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.871 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.872 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.872 DEBUG:boto:Params: {} 2022-09-13T07:39:34.873 DEBUG:boto:Token: None 2022-09-13T07:39:34.874 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.874 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o= 2022-09-13T07:39:34.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HHHR7JktN/IYgenzEhQJz+XN99o=', 'Content-Length': '0'} 2022-09-13T07:39:34.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000270bfbc66b4e47e-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.879 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.880 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-19 zones={test-zone2, test-zone1} 2022-09-13T07:39:34.880 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-19 zones={test-zone3, test-zone1} 2022-09-13T07:39:34.881 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.881 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.882 DEBUG:boto:Method: HEAD 2022-09-13T07:39:34.882 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:39:34.883 DEBUG:boto:Data: 2022-09-13T07:39:34.883 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.884 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:34.884 DEBUG:boto:Port: 8001 2022-09-13T07:39:34.885 DEBUG:boto:Params: {} 2022-09-13T07:39:34.885 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-13T07:39:34.886 DEBUG:boto:Token: None 2022-09-13T07:39:34.886 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.887 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zRsh/pN6zPAOK+RRI4QycddJEgs= 2022-09-13T07:39:34.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zRsh/pN6zPAOK+RRI4QycddJEgs=', 'Content-Length': '0'} 2022-09-13T07:39:34.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000489a1244d3e3a65d-00632033b6-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.892 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.893 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.893 DEBUG:boto:Method: HEAD 2022-09-13T07:39:34.894 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:39:34.894 DEBUG:boto:Data: 2022-09-13T07:39:34.895 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.895 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.896 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.896 DEBUG:boto:Params: {} 2022-09-13T07:39:34.897 DEBUG:boto:Token: None 2022-09-13T07:39:34.897 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/ 2022-09-13T07:39:34.898 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zRsh/pN6zPAOK+RRI4QycddJEgs= 2022-09-13T07:39:34.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zRsh/pN6zPAOK+RRI4QycddJEgs=', 'Content-Length': '0'} 2022-09-13T07:39:34.902 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036c67d04675a2c48-00632033b6-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.903 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:39:34.903 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.904 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.905 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:34.906 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:34.907 DEBUG:boto:Method: GET 2022-09-13T07:39:34.908 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:34.909 DEBUG:boto:Data: 2022-09-13T07:39:34.909 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.910 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:34.911 DEBUG:boto:Port: 8001 2022-09-13T07:39:34.912 DEBUG:boto:Params: {} 2022-09-13T07:39:34.914 DEBUG:boto:Token: None 2022-09-13T07:39:34.915 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/?versions 2022-09-13T07:39:34.916 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VlWswTyw2Axz/cSzKf9NwedJS/s= 2022-09-13T07:39:34.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VlWswTyw2Axz/cSzKf9NwedJS/s=', 'Content-Length': '0'} 2022-09-13T07:39:34.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a81e74bc2e8cabfa-00632033b6-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.924 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEn7MftyiCP0pYMPcqn5sySJi4UkpaiZtrue2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmwBtbzZwRR4A0HEc28dvfcEWdzuIZDfalse2022-09-13T07:37:30.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7false2022-09-13T07:37:30.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTL9xVVVkIODKu69Lcj5HlhVjbyjGE3ffalse2022-09-13T07:37:30.515Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU.elqMxtYPda6ZsKEKDP833Q93.DHGxfalse2022-09-13T07:37:30.547Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji8qJKwt5yOnRkOjqtadujgkvccCo7V6false2022-09-13T07:37:30.592Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WBfalse2022-09-13T07:37:30.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtLDy1U4ZTxWriIUMx30zfLE7zPcVUt5false2022-09-13T07:37:30.424Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjulPqaKM9GsnZBkMYt.uabOf.VJ5P.5mfalse2022-09-13T07:37:30.465Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eAfalse2022-09-13T07:37:30.308Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnEgHn6dtpr-DjlqQutDDilFO8cCU9tVfalse2022-09-13T07:37:30.261Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzbAEfbqMIp9ReKOn2t8gGBeNLMCc26hfalse2022-09-13T07:37:30.341Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkKoYsAdAKf7lBJBYkG7TA7YHa0Qh7adfalse2022-09-13T07:37:30.152Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1false2022-09-13T07:37:30.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxwhfwIrEIbdFug3DCFDiVhftQi2CUm6false2022-09-13T07:37:30.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN1ijg0L2Sa1qSFcwgB7aYlf3fysHL.ofalse2022-09-13T07:37:30.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6pfalse2022-09-13T07:37:30.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjr3j0LS1bjhh8xQBCJCWMQPOkDqpqjuMfalse2022-09-13T07:37:30.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUFIYwFqhiCTkSFbmDIDHjWDiTsQToDGfalse2022-09-13T07:37:30.001Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjei7viyB1s5vEJjb-DgvRL3K6yeA94ngfalse2022-09-13T07:37:29.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHnhTbZbus4hq.TBh5.hecwSTeKVCaRfalse2022-09-13T07:37:30.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8bP-bkYz6JpHUV25JyCvc0fDB8aZwslfalse2022-09-13T07:37:29.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcPIo6H3tnk3IHLM4mBy4H8IXyB8CUPCfalse2022-09-13T07:37:29.842Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-false2022-09-13T07:37:29.815Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3VqUFgxLYLsisMjs5Rrmm15ZQudhjyfalse2022-09-13T07:37:29.688Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4false2022-09-13T07:37:29.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxshrAtab2xKYxtLaiLBmYWJxRHGcH2rfalse2022-09-13T07:37:29.770Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.Ifalse2022-09-13T07:37:29.573Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC-XByO2K9qUOxsnzB35XnbS7CZIHKxHfalse2022-09-13T07:37:29.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaErb35kohBMBuUIVZDyjKh5q.cT.bAJfalse2022-09-13T07:37:29.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.927 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:39:34.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.942 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:39:34.943 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.943 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.944 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:34.944 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:34.945 DEBUG:boto:Method: GET 2022-09-13T07:39:34.945 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:34.946 DEBUG:boto:Data: 2022-09-13T07:39:34.946 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.947 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.947 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.948 DEBUG:boto:Params: {} 2022-09-13T07:39:34.948 DEBUG:boto:Token: None 2022-09-13T07:39:34.949 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/?versions 2022-09-13T07:39:34.950 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VlWswTyw2Axz/cSzKf9NwedJS/s= 2022-09-13T07:39:34.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VlWswTyw2Axz/cSzKf9NwedJS/s=', 'Content-Length': '0'} 2022-09-13T07:39:34.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7880c218b8c65cf-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.955 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEn7MftyiCP0pYMPcqn5sySJi4UkpaiZtrue2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmwBtbzZwRR4A0HEc28dvfcEWdzuIZDfalse2022-09-13T07:37:30.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7false2022-09-13T07:37:30.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTL9xVVVkIODKu69Lcj5HlhVjbyjGE3ffalse2022-09-13T07:37:30.515Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU.elqMxtYPda6ZsKEKDP833Q93.DHGxfalse2022-09-13T07:37:30.547Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji8qJKwt5yOnRkOjqtadujgkvccCo7V6false2022-09-13T07:37:30.592Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WBfalse2022-09-13T07:37:30.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtLDy1U4ZTxWriIUMx30zfLE7zPcVUt5false2022-09-13T07:37:30.424Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjulPqaKM9GsnZBkMYt.uabOf.VJ5P.5mfalse2022-09-13T07:37:30.465Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eAfalse2022-09-13T07:37:30.308Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnEgHn6dtpr-DjlqQutDDilFO8cCU9tVfalse2022-09-13T07:37:30.261Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzbAEfbqMIp9ReKOn2t8gGBeNLMCc26hfalse2022-09-13T07:37:30.341Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkKoYsAdAKf7lBJBYkG7TA7YHa0Qh7adfalse2022-09-13T07:37:30.152Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1false2022-09-13T07:37:30.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxwhfwIrEIbdFug3DCFDiVhftQi2CUm6false2022-09-13T07:37:30.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN1ijg0L2Sa1qSFcwgB7aYlf3fysHL.ofalse2022-09-13T07:37:30.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6pfalse2022-09-13T07:37:30.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjr3j0LS1bjhh8xQBCJCWMQPOkDqpqjuMfalse2022-09-13T07:37:30.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUFIYwFqhiCTkSFbmDIDHjWDiTsQToDGfalse2022-09-13T07:37:30.001Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjei7viyB1s5vEJjb-DgvRL3K6yeA94ngfalse2022-09-13T07:37:29.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHnhTbZbus4hq.TBh5.hecwSTeKVCaRfalse2022-09-13T07:37:30.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8bP-bkYz6JpHUV25JyCvc0fDB8aZwslfalse2022-09-13T07:37:29.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcPIo6H3tnk3IHLM4mBy4H8IXyB8CUPCfalse2022-09-13T07:37:29.842Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-false2022-09-13T07:37:29.815Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3VqUFgxLYLsisMjs5Rrmm15ZQudhjyfalse2022-09-13T07:37:29.688Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4false2022-09-13T07:37:29.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxshrAtab2xKYxtLaiLBmYWJxRHGcH2rfalse2022-09-13T07:37:29.770Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.Ifalse2022-09-13T07:37:29.573Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC-XByO2K9qUOxsnzB35XnbS7CZIHKxHfalse2022-09-13T07:37:29.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaErb35kohBMBuUIVZDyjKh5q.cT.bAJfalse2022-09-13T07:37:29.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.957 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:39:34.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:34.972 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.972 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.973 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:34.973 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:34.974 DEBUG:boto:Method: GET 2022-09-13T07:39:34.974 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:34.975 DEBUG:boto:Data: 2022-09-13T07:39:34.976 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.976 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:34.977 DEBUG:boto:Port: 8001 2022-09-13T07:39:34.977 DEBUG:boto:Params: {} 2022-09-13T07:39:34.978 DEBUG:boto:Token: None 2022-09-13T07:39:34.978 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/?versions 2022-09-13T07:39:34.979 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VlWswTyw2Axz/cSzKf9NwedJS/s= 2022-09-13T07:39:34.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VlWswTyw2Axz/cSzKf9NwedJS/s=', 'Content-Length': '0'} 2022-09-13T07:39:34.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a084a8a0c8a591bb-00632033b6-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:34.986 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEn7MftyiCP0pYMPcqn5sySJi4UkpaiZtrue2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmwBtbzZwRR4A0HEc28dvfcEWdzuIZDfalse2022-09-13T07:37:30.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7false2022-09-13T07:37:30.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTL9xVVVkIODKu69Lcj5HlhVjbyjGE3ffalse2022-09-13T07:37:30.515Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU.elqMxtYPda6ZsKEKDP833Q93.DHGxfalse2022-09-13T07:37:30.547Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji8qJKwt5yOnRkOjqtadujgkvccCo7V6false2022-09-13T07:37:30.592Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WBfalse2022-09-13T07:37:30.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtLDy1U4ZTxWriIUMx30zfLE7zPcVUt5false2022-09-13T07:37:30.424Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjulPqaKM9GsnZBkMYt.uabOf.VJ5P.5mfalse2022-09-13T07:37:30.465Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eAfalse2022-09-13T07:37:30.308Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnEgHn6dtpr-DjlqQutDDilFO8cCU9tVfalse2022-09-13T07:37:30.261Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzbAEfbqMIp9ReKOn2t8gGBeNLMCc26hfalse2022-09-13T07:37:30.341Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkKoYsAdAKf7lBJBYkG7TA7YHa0Qh7adfalse2022-09-13T07:37:30.152Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1false2022-09-13T07:37:30.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxwhfwIrEIbdFug3DCFDiVhftQi2CUm6false2022-09-13T07:37:30.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN1ijg0L2Sa1qSFcwgB7aYlf3fysHL.ofalse2022-09-13T07:37:30.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6pfalse2022-09-13T07:37:30.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjr3j0LS1bjhh8xQBCJCWMQPOkDqpqjuMfalse2022-09-13T07:37:30.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUFIYwFqhiCTkSFbmDIDHjWDiTsQToDGfalse2022-09-13T07:37:30.001Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjei7viyB1s5vEJjb-DgvRL3K6yeA94ngfalse2022-09-13T07:37:29.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHnhTbZbus4hq.TBh5.hecwSTeKVCaRfalse2022-09-13T07:37:30.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8bP-bkYz6JpHUV25JyCvc0fDB8aZwslfalse2022-09-13T07:37:29.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcPIo6H3tnk3IHLM4mBy4H8IXyB8CUPCfalse2022-09-13T07:37:29.842Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-false2022-09-13T07:37:29.815Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3VqUFgxLYLsisMjs5Rrmm15ZQudhjyfalse2022-09-13T07:37:29.688Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4false2022-09-13T07:37:29.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxshrAtab2xKYxtLaiLBmYWJxRHGcH2rfalse2022-09-13T07:37:29.770Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.Ifalse2022-09-13T07:37:29.573Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC-XByO2K9qUOxsnzB35XnbS7CZIHKxHfalse2022-09-13T07:37:29.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaErb35kohBMBuUIVZDyjKh5q.cT.bAJfalse2022-09-13T07:37:29.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:34.989 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:34.990 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:34.990 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:34.991 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:34.992 DEBUG:boto:Method: GET 2022-09-13T07:39:34.992 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:34.993 DEBUG:boto:Data: 2022-09-13T07:39:34.993 DEBUG:boto:Headers: {} 2022-09-13T07:39:34.994 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:34.994 DEBUG:boto:Port: 8000 2022-09-13T07:39:34.995 DEBUG:boto:Params: {} 2022-09-13T07:39:34.996 DEBUG:boto:Token: None 2022-09-13T07:39:34.996 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:34 GMT /yxvvuh-19/?versions 2022-09-13T07:39:34.997 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VlWswTyw2Axz/cSzKf9NwedJS/s= 2022-09-13T07:39:34.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:34 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VlWswTyw2Axz/cSzKf9NwedJS/s=', 'Content-Length': '0'} 2022-09-13T07:39:35.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac8fea26866e20d2-00632033b6-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.002 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEn7MftyiCP0pYMPcqn5sySJi4UkpaiZtrue2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmwBtbzZwRR4A0HEc28dvfcEWdzuIZDfalse2022-09-13T07:37:30.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7false2022-09-13T07:37:30.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTL9xVVVkIODKu69Lcj5HlhVjbyjGE3ffalse2022-09-13T07:37:30.515Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU.elqMxtYPda6ZsKEKDP833Q93.DHGxfalse2022-09-13T07:37:30.547Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji8qJKwt5yOnRkOjqtadujgkvccCo7V6false2022-09-13T07:37:30.592Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WBfalse2022-09-13T07:37:30.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtLDy1U4ZTxWriIUMx30zfLE7zPcVUt5false2022-09-13T07:37:30.424Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjulPqaKM9GsnZBkMYt.uabOf.VJ5P.5mfalse2022-09-13T07:37:30.465Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eAfalse2022-09-13T07:37:30.308Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnEgHn6dtpr-DjlqQutDDilFO8cCU9tVfalse2022-09-13T07:37:30.261Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzbAEfbqMIp9ReKOn2t8gGBeNLMCc26hfalse2022-09-13T07:37:30.341Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkKoYsAdAKf7lBJBYkG7TA7YHa0Qh7adfalse2022-09-13T07:37:30.152Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1false2022-09-13T07:37:30.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxwhfwIrEIbdFug3DCFDiVhftQi2CUm6false2022-09-13T07:37:30.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN1ijg0L2Sa1qSFcwgB7aYlf3fysHL.ofalse2022-09-13T07:37:30.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6pfalse2022-09-13T07:37:30.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjr3j0LS1bjhh8xQBCJCWMQPOkDqpqjuMfalse2022-09-13T07:37:30.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUFIYwFqhiCTkSFbmDIDHjWDiTsQToDGfalse2022-09-13T07:37:30.001Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjei7viyB1s5vEJjb-DgvRL3K6yeA94ngfalse2022-09-13T07:37:29.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHnhTbZbus4hq.TBh5.hecwSTeKVCaRfalse2022-09-13T07:37:30.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8bP-bkYz6JpHUV25JyCvc0fDB8aZwslfalse2022-09-13T07:37:29.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcPIo6H3tnk3IHLM4mBy4H8IXyB8CUPCfalse2022-09-13T07:37:29.842Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-false2022-09-13T07:37:29.815Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3VqUFgxLYLsisMjs5Rrmm15ZQudhjyfalse2022-09-13T07:37:29.688Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4false2022-09-13T07:37:29.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxshrAtab2xKYxtLaiLBmYWJxRHGcH2rfalse2022-09-13T07:37:29.770Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.Ifalse2022-09-13T07:37:29.573Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC-XByO2K9qUOxsnzB35XnbS7CZIHKxHfalse2022-09-13T07:37:29.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaErb35kohBMBuUIVZDyjKh5q.cT.bAJfalse2022-09-13T07:37:29.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:35.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:39:35.006 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:35.007 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:35.007 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.008 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.008 DEBUG:boto:Method: GET 2022-09-13T07:39:35.009 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.009 DEBUG:boto:Data: 2022-09-13T07:39:35.010 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.011 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.011 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.012 DEBUG:boto:Params: {} 2022-09-13T07:39:35.012 DEBUG:boto:Token: None 2022-09-13T07:39:35.013 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.014 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:otXpmnBQnClhJOqC35uKDBn8+os= 2022-09-13T07:39:35.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:otXpmnBQnClhJOqC35uKDBn8+os=', 'Content-Length': '0'} 2022-09-13T07:39:35.021 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000067911951afe7b888-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.023 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:35.023 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:35.024 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.025 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.026 DEBUG:boto:Method: GET 2022-09-13T07:39:35.026 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.027 DEBUG:boto:Data: 2022-09-13T07:39:35.027 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.028 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.029 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.029 DEBUG:boto:Params: {} 2022-09-13T07:39:35.030 DEBUG:boto:Token: None 2022-09-13T07:39:35.030 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.031 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:otXpmnBQnClhJOqC35uKDBn8+os= 2022-09-13T07:39:35.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:otXpmnBQnClhJOqC35uKDBn8+os=', 'Content-Length': '0'} 2022-09-13T07:39:35.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d0d772ec9e2e6a1-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.035 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:35.035 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:35.036 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.036 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.037 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.038 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.038 DEBUG:boto:Data: 2022-09-13T07:39:35.039 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.039 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.044 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.044 DEBUG:boto:Params: {} 2022-09-13T07:39:35.045 DEBUG:boto:Token: None 2022-09-13T07:39:35.045 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.047 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aijiwCyNl9+qe+sh6HCt/b2dGi8= 2022-09-13T07:39:35.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aijiwCyNl9+qe+sh6HCt/b2dGi8=', 'Content-Length': '0'} 2022-09-13T07:39:35.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c5d33025c0b276be-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.050 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:35.051 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:35.051 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.054 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.054 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.055 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.055 DEBUG:boto:Data: 2022-09-13T07:39:35.056 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.056 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.057 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.057 DEBUG:boto:Params: {} 2022-09-13T07:39:35.058 DEBUG:boto:Token: None 2022-09-13T07:39:35.058 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.059 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aijiwCyNl9+qe+sh6HCt/b2dGi8= 2022-09-13T07:39:35.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aijiwCyNl9+qe+sh6HCt/b2dGi8=', 'Content-Length': '0'} 2022-09-13T07:39:35.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f985c6a39554e7d7-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:39:35.065 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:35.066 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:35.067 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.068 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.069 DEBUG:boto:Method: GET 2022-09-13T07:39:35.070 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.071 DEBUG:boto:Data: 2022-09-13T07:39:35.072 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.073 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.074 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.075 DEBUG:boto:Params: {} 2022-09-13T07:39:35.076 DEBUG:boto:Token: None 2022-09-13T07:39:35.077 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.079 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:otXpmnBQnClhJOqC35uKDBn8+os= 2022-09-13T07:39:35.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:otXpmnBQnClhJOqC35uKDBn8+os=', 'Content-Length': '0'} 2022-09-13T07:39:35.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007e2879fc2e214223-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.083 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:35.083 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:35.084 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.084 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.085 DEBUG:boto:Method: GET 2022-09-13T07:39:35.085 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.086 DEBUG:boto:Data: 2022-09-13T07:39:35.086 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.087 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.088 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.088 DEBUG:boto:Params: {} 2022-09-13T07:39:35.089 DEBUG:boto:Token: None 2022-09-13T07:39:35.089 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.090 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:otXpmnBQnClhJOqC35uKDBn8+os= 2022-09-13T07:39:35.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:otXpmnBQnClhJOqC35uKDBn8+os=', 'Content-Length': '0'} 2022-09-13T07:39:35.093 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000166eaa4e1d237d34-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.094 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:35.094 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:35.095 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:35.095 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:35.096 DEBUG:boto:Method: GET 2022-09-13T07:39:35.096 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:39:35.097 DEBUG:boto:Data: 2022-09-13T07:39:35.097 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.098 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.098 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.099 DEBUG:boto:Params: {} 2022-09-13T07:39:35.100 DEBUG:boto:Token: None 2022-09-13T07:39:35.100 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/ 2022-09-13T07:39:35.101 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc= 2022-09-13T07:39:35.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc=', 'Content-Length': '0'} 2022-09-13T07:39:35.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052e088545aac0188-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.105 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:35.106 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:35.107 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:35.107 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:35.108 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:35.108 DEBUG:boto:Method: GET 2022-09-13T07:39:35.109 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:39:35.109 DEBUG:boto:Data: 2022-09-13T07:39:35.110 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.111 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.111 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.112 DEBUG:boto:Params: {} 2022-09-13T07:39:35.112 DEBUG:boto:Token: None 2022-09-13T07:39:35.113 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/ 2022-09-13T07:39:35.114 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc= 2022-09-13T07:39:35.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc=', 'Content-Length': '0'} 2022-09-13T07:39:35.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073cf0ebaee65b216-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.118 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:35.119 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:35.120 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:35.120 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.121 DEBUG:boto:Path: /yxvvuh-19/dummy 2022-09-13T07:39:35.121 DEBUG:boto:Data: 2022-09-13T07:39:35.122 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.122 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.123 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.123 DEBUG:boto:Params: {} 2022-09-13T07:39:35.124 DEBUG:boto:Token: None 2022-09-13T07:39:35.124 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/dummy 2022-09-13T07:39:35.125 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9wM2v+4o5kOtxno58THtsfi70Ck= 2022-09-13T07:39:35.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9wM2v+4o5kOtxno58THtsfi70Ck=', 'Content-Length': '0'} 2022-09-13T07:39:35.128 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006873bb3c6a45c85f-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.129 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:35.130 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:35.131 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.132 DEBUG:boto:Path: /yxvvuh-19/dummy 2022-09-13T07:39:35.133 DEBUG:boto:Data: 2022-09-13T07:39:35.134 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.135 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.136 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.137 DEBUG:boto:Params: {} 2022-09-13T07:39:35.139 DEBUG:boto:Token: None 2022-09-13T07:39:35.140 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/dummy 2022-09-13T07:39:35.141 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9wM2v+4o5kOtxno58THtsfi70Ck= 2022-09-13T07:39:35.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9wM2v+4o5kOtxno58THtsfi70Ck=', 'Content-Length': '0'} 2022-09-13T07:39:35.145 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000067f9fbc9fa1cd949-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:39:35.147 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:35.148 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:35.149 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.149 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.150 DEBUG:boto:Method: GET 2022-09-13T07:39:35.151 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.152 DEBUG:boto:Data: 2022-09-13T07:39:35.153 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.154 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.155 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.156 DEBUG:boto:Params: {} 2022-09-13T07:39:35.157 DEBUG:boto:Token: None 2022-09-13T07:39:35.158 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.159 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:otXpmnBQnClhJOqC35uKDBn8+os= 2022-09-13T07:39:35.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:otXpmnBQnClhJOqC35uKDBn8+os=', 'Content-Length': '0'} 2022-09-13T07:39:35.163 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba166852c1df6d0f-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.165 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:35.166 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:35.166 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.167 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.168 DEBUG:boto:Method: GET 2022-09-13T07:39:35.169 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.170 DEBUG:boto:Data: 2022-09-13T07:39:35.170 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.171 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.172 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.173 DEBUG:boto:Params: {} 2022-09-13T07:39:35.174 DEBUG:boto:Token: None 2022-09-13T07:39:35.175 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:35.177 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:otXpmnBQnClhJOqC35uKDBn8+os= 2022-09-13T07:39:35.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:otXpmnBQnClhJOqC35uKDBn8+os=', 'Content-Length': '0'} 2022-09-13T07:39:35.180 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f6b22257f714c49e-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.181 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:35.182 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:35.183 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:35.184 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:35.185 DEBUG:boto:Method: GET 2022-09-13T07:39:35.185 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:39:35.186 DEBUG:boto:Data: 2022-09-13T07:39:35.186 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.187 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.187 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.188 DEBUG:boto:Params: {} 2022-09-13T07:39:35.188 DEBUG:boto:Token: None 2022-09-13T07:39:35.189 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/ 2022-09-13T07:39:35.189 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc= 2022-09-13T07:39:35.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc=', 'Content-Length': '0'} 2022-09-13T07:39:35.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c0c0586024ef237-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.234 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:35.236 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:35.237 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:35.238 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:35.239 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:35.240 DEBUG:boto:Method: GET 2022-09-13T07:39:35.241 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:39:35.242 DEBUG:boto:Data: 2022-09-13T07:39:35.243 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.244 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.245 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.246 DEBUG:boto:Params: {} 2022-09-13T07:39:35.247 DEBUG:boto:Token: None 2022-09-13T07:39:35.248 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/ 2022-09-13T07:39:35.249 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc= 2022-09-13T07:39:35.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc=', 'Content-Length': '0'} 2022-09-13T07:39:35.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002affe9156240101e-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.255 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:35.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:35.257 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.258 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.259 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.260 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.261 DEBUG:boto:Method: GET 2022-09-13T07:39:35.262 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.263 DEBUG:boto:Data: 2022-09-13T07:39:35.264 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.265 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.266 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.267 DEBUG:boto:Params: {} 2022-09-13T07:39:35.268 DEBUG:boto:Token: None 2022-09-13T07:39:35.269 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.270 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y= 2022-09-13T07:39:35.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y=', 'Content-Length': '0'} 2022-09-13T07:39:35.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6eed95a1a46aad8-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.276 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.277 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.278 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.279 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.280 DEBUG:boto:Method: GET 2022-09-13T07:39:35.281 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.282 DEBUG:boto:Data: 2022-09-13T07:39:35.283 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.284 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.285 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.286 DEBUG:boto:Params: {} 2022-09-13T07:39:35.287 DEBUG:boto:Token: None 2022-09-13T07:39:35.288 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.289 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y= 2022-09-13T07:39:35.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y=', 'Content-Length': '0'} 2022-09-13T07:39:35.293 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e3cda1d8135de58-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.294 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.294 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.295 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.295 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.296 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.297 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.297 DEBUG:boto:Data: 2022-09-13T07:39:35.298 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.298 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.299 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.299 DEBUG:boto:Params: {} 2022-09-13T07:39:35.300 DEBUG:boto:Token: None 2022-09-13T07:39:35.300 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.301 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/9RcELWQLwFBW1jiyaXAPAxLLs= 2022-09-13T07:39:35.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/9RcELWQLwFBW1jiyaXAPAxLLs=', 'Content-Length': '0'} 2022-09-13T07:39:35.304 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe86367740aebc41-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.305 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.305 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.306 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.306 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.307 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.307 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.308 DEBUG:boto:Data: 2022-09-13T07:39:35.309 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.309 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.310 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.310 DEBUG:boto:Params: {} 2022-09-13T07:39:35.311 DEBUG:boto:Token: None 2022-09-13T07:39:35.311 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.312 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:b/9RcELWQLwFBW1jiyaXAPAxLLs= 2022-09-13T07:39:35.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:b/9RcELWQLwFBW1jiyaXAPAxLLs=', 'Content-Length': '0'} 2022-09-13T07:39:35.315 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004a52f5e5efd1195-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:35.316 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.316 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.317 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.318 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.318 DEBUG:boto:Method: GET 2022-09-13T07:39:35.319 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.319 DEBUG:boto:Data: 2022-09-13T07:39:35.320 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.320 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.321 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.321 DEBUG:boto:Params: {} 2022-09-13T07:39:35.322 DEBUG:boto:Token: None 2022-09-13T07:39:35.323 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.323 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y= 2022-09-13T07:39:35.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y=', 'Content-Length': '0'} 2022-09-13T07:39:35.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5fb354886e437e6-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.327 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.328 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.328 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.329 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.329 DEBUG:boto:Method: GET 2022-09-13T07:39:35.330 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.330 DEBUG:boto:Data: 2022-09-13T07:39:35.331 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.331 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.332 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.332 DEBUG:boto:Params: {} 2022-09-13T07:39:35.333 DEBUG:boto:Token: None 2022-09-13T07:39:35.334 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.334 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y= 2022-09-13T07:39:35.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y=', 'Content-Length': '0'} 2022-09-13T07:39:35.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f572f3ff461a1442-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.338 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:35.338 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:35.339 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.339 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.340 DEBUG:boto:Method: GET 2022-09-13T07:39:35.340 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.341 DEBUG:boto:Data: 2022-09-13T07:39:35.341 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.342 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.343 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.343 DEBUG:boto:Params: {} 2022-09-13T07:39:35.344 DEBUG:boto:Token: None 2022-09-13T07:39:35.344 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/ 2022-09-13T07:39:35.345 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc= 2022-09-13T07:39:35.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc=', 'Content-Length': '0'} 2022-09-13T07:39:35.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008037f161b0b2f9be-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.390 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:35.392 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:35.393 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:35.394 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.396 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.397 DEBUG:boto:Method: GET 2022-09-13T07:39:35.397 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.398 DEBUG:boto:Data: 2022-09-13T07:39:35.398 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.399 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.399 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.400 DEBUG:boto:Params: {} 2022-09-13T07:39:35.401 DEBUG:boto:Token: None 2022-09-13T07:39:35.402 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/ 2022-09-13T07:39:35.402 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc= 2022-09-13T07:39:35.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc=', 'Content-Length': '0'} 2022-09-13T07:39:35.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae41e77698814baf-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.407 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:35.408 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.408 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.409 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.409 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:35.410 DEBUG:boto:Data: 2022-09-13T07:39:35.410 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.411 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.412 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.412 DEBUG:boto:Params: {} 2022-09-13T07:39:35.413 DEBUG:boto:Token: None 2022-09-13T07:39:35.413 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj 2022-09-13T07:39:35.414 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+AYCBG0nPp5z0y8+4LXEA5b4p58= 2022-09-13T07:39:35.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+AYCBG0nPp5z0y8+4LXEA5b4p58=', 'Content-Length': '0'} 2022-09-13T07:39:35.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc12e6904976097f-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.419 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.419 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.420 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.420 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:35.421 DEBUG:boto:Data: 2022-09-13T07:39:35.421 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.422 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.422 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.423 DEBUG:boto:Params: {} 2022-09-13T07:39:35.424 DEBUG:boto:Token: None 2022-09-13T07:39:35.424 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj 2022-09-13T07:39:35.425 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+AYCBG0nPp5z0y8+4LXEA5b4p58= 2022-09-13T07:39:35.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+AYCBG0nPp5z0y8+4LXEA5b4p58=', 'Content-Length': '0'} 2022-09-13T07:39:35.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070c0055e5a42a4b6-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:35.430 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.431 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.432 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.432 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.433 DEBUG:boto:Method: GET 2022-09-13T07:39:35.433 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.434 DEBUG:boto:Data: 2022-09-13T07:39:35.434 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.435 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.436 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.436 DEBUG:boto:Params: {} 2022-09-13T07:39:35.437 DEBUG:boto:Token: None 2022-09-13T07:39:35.437 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.438 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y= 2022-09-13T07:39:35.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y=', 'Content-Length': '0'} 2022-09-13T07:39:35.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fecc47852fa9d495-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.443 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.444 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.444 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.445 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.447 DEBUG:boto:Method: GET 2022-09-13T07:39:35.448 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.449 DEBUG:boto:Data: 2022-09-13T07:39:35.450 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.451 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.451 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.452 DEBUG:boto:Params: {} 2022-09-13T07:39:35.453 DEBUG:boto:Token: None 2022-09-13T07:39:35.454 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.455 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y= 2022-09-13T07:39:35.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y=', 'Content-Length': '0'} 2022-09-13T07:39:35.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004783094f39a56dc8-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.502 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:35.504 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:35.505 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.505 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.506 DEBUG:boto:Method: GET 2022-09-13T07:39:35.507 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.508 DEBUG:boto:Data: 2022-09-13T07:39:35.509 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.510 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.511 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.512 DEBUG:boto:Params: {} 2022-09-13T07:39:35.513 DEBUG:boto:Token: None 2022-09-13T07:39:35.514 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/ 2022-09-13T07:39:35.515 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc= 2022-09-13T07:39:35.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc=', 'Content-Length': '0'} 2022-09-13T07:39:35.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004344f9b33db81256-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.521 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:35.522 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:35.523 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:35.524 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.524 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.525 DEBUG:boto:Method: GET 2022-09-13T07:39:35.526 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.526 DEBUG:boto:Data: 2022-09-13T07:39:35.527 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.527 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.528 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.528 DEBUG:boto:Params: {} 2022-09-13T07:39:35.529 DEBUG:boto:Token: None 2022-09-13T07:39:35.529 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/ 2022-09-13T07:39:35.530 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc= 2022-09-13T07:39:35.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc=', 'Content-Length': '0'} 2022-09-13T07:39:35.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000123cb9e858b00b10-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.535 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:35.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:35.536 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.537 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.537 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.538 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.538 DEBUG:boto:Method: GET 2022-09-13T07:39:35.539 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.539 DEBUG:boto:Data: 2022-09-13T07:39:35.540 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.540 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.541 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.541 DEBUG:boto:Params: {} 2022-09-13T07:39:35.542 DEBUG:boto:Token: None 2022-09-13T07:39:35.543 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.544 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BLjyBz+OhbRQMDMZE6OvdOBHfZ4= 2022-09-13T07:39:35.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BLjyBz+OhbRQMDMZE6OvdOBHfZ4=', 'Content-Length': '0'} 2022-09-13T07:39:35.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cdd21e0404374b7-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.548 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.548 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.549 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.549 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.550 DEBUG:boto:Method: GET 2022-09-13T07:39:35.551 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.551 DEBUG:boto:Data: 2022-09-13T07:39:35.552 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.552 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.552 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.553 DEBUG:boto:Params: {} 2022-09-13T07:39:35.554 DEBUG:boto:Token: None 2022-09-13T07:39:35.555 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.555 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BLjyBz+OhbRQMDMZE6OvdOBHfZ4= 2022-09-13T07:39:35.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BLjyBz+OhbRQMDMZE6OvdOBHfZ4=', 'Content-Length': '0'} 2022-09-13T07:39:35.558 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053babe7dc376e6e7-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.559 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.559 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.560 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.561 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.562 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.563 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.563 DEBUG:boto:Data: 2022-09-13T07:39:35.564 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.565 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.566 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.567 DEBUG:boto:Params: {} 2022-09-13T07:39:35.569 DEBUG:boto:Token: None 2022-09-13T07:39:35.569 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.571 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:toj60X5+K7pvUBvXATwhmCz0z7E= 2022-09-13T07:39:35.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:toj60X5+K7pvUBvXATwhmCz0z7E=', 'Content-Length': '0'} 2022-09-13T07:39:35.574 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cee27a09b1feb66-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.575 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.575 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.576 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.576 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.577 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.577 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.578 DEBUG:boto:Data: 2022-09-13T07:39:35.578 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.579 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.579 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.580 DEBUG:boto:Params: {} 2022-09-13T07:39:35.580 DEBUG:boto:Token: None 2022-09-13T07:39:35.581 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.582 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:toj60X5+K7pvUBvXATwhmCz0z7E= 2022-09-13T07:39:35.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:toj60X5+K7pvUBvXATwhmCz0z7E=', 'Content-Length': '0'} 2022-09-13T07:39:35.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb5268dbd17db0ed-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:35.586 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.587 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.588 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.588 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.589 DEBUG:boto:Method: GET 2022-09-13T07:39:35.589 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.590 DEBUG:boto:Data: 2022-09-13T07:39:35.591 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.591 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.592 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.593 DEBUG:boto:Params: {} 2022-09-13T07:39:35.593 DEBUG:boto:Token: None 2022-09-13T07:39:35.594 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.595 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BLjyBz+OhbRQMDMZE6OvdOBHfZ4= 2022-09-13T07:39:35.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BLjyBz+OhbRQMDMZE6OvdOBHfZ4=', 'Content-Length': '0'} 2022-09-13T07:39:35.598 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea121ad46cfa4128-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.638 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.640 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.641 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.642 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.643 DEBUG:boto:Method: GET 2022-09-13T07:39:35.644 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.645 DEBUG:boto:Data: 2022-09-13T07:39:35.646 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.647 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.648 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.649 DEBUG:boto:Params: {} 2022-09-13T07:39:35.651 DEBUG:boto:Token: None 2022-09-13T07:39:35.652 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:35.653 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BLjyBz+OhbRQMDMZE6OvdOBHfZ4= 2022-09-13T07:39:35.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BLjyBz+OhbRQMDMZE6OvdOBHfZ4=', 'Content-Length': '0'} 2022-09-13T07:39:35.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008730b1b6ff065a1c-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.657 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:35.659 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:35.660 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.660 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.661 DEBUG:boto:Method: GET 2022-09-13T07:39:35.661 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.662 DEBUG:boto:Data: 2022-09-13T07:39:35.662 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.663 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.663 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.664 DEBUG:boto:Params: {} 2022-09-13T07:39:35.665 DEBUG:boto:Token: None 2022-09-13T07:39:35.665 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/ 2022-09-13T07:39:35.666 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc= 2022-09-13T07:39:35.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc=', 'Content-Length': '0'} 2022-09-13T07:39:35.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006aeca56c010fcb23-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.670 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:35.671 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:35.672 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:35.672 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.673 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.673 DEBUG:boto:Method: GET 2022-09-13T07:39:35.674 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.675 DEBUG:boto:Data: 2022-09-13T07:39:35.675 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.676 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.676 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.677 DEBUG:boto:Params: {} 2022-09-13T07:39:35.677 DEBUG:boto:Token: None 2022-09-13T07:39:35.678 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/ 2022-09-13T07:39:35.678 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc= 2022-09-13T07:39:35.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc=', 'Content-Length': '0'} 2022-09-13T07:39:35.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000676f62d8e5c95011-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.687 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:35.688 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.689 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.689 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.690 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:35.690 DEBUG:boto:Data: 2022-09-13T07:39:35.691 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.691 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.692 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.692 DEBUG:boto:Params: {} 2022-09-13T07:39:35.693 DEBUG:boto:Token: None 2022-09-13T07:39:35.693 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj 2022-09-13T07:39:35.694 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+AYCBG0nPp5z0y8+4LXEA5b4p58= 2022-09-13T07:39:35.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+AYCBG0nPp5z0y8+4LXEA5b4p58=', 'Content-Length': '0'} 2022-09-13T07:39:35.699 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d56679a61fe2875d-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.699 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.700 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.700 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.701 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:35.701 DEBUG:boto:Data: 2022-09-13T07:39:35.702 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.702 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.703 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.703 DEBUG:boto:Params: {} 2022-09-13T07:39:35.704 DEBUG:boto:Token: None 2022-09-13T07:39:35.704 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj 2022-09-13T07:39:35.705 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+AYCBG0nPp5z0y8+4LXEA5b4p58= 2022-09-13T07:39:35.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+AYCBG0nPp5z0y8+4LXEA5b4p58=', 'Content-Length': '0'} 2022-09-13T07:39:35.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000308553ffc086fd11-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:35.710 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.710 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.711 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.711 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.712 DEBUG:boto:Method: GET 2022-09-13T07:39:35.712 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.713 DEBUG:boto:Data: 2022-09-13T07:39:35.713 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.714 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.714 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.715 DEBUG:boto:Params: {} 2022-09-13T07:39:35.716 DEBUG:boto:Token: None 2022-09-13T07:39:35.716 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.717 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y= 2022-09-13T07:39:35.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y=', 'Content-Length': '0'} 2022-09-13T07:39:35.719 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a2a539899037fa8-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.720 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.721 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.721 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.722 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.722 DEBUG:boto:Method: GET 2022-09-13T07:39:35.723 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.723 DEBUG:boto:Data: 2022-09-13T07:39:35.724 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.724 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.725 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.725 DEBUG:boto:Params: {} 2022-09-13T07:39:35.726 DEBUG:boto:Token: None 2022-09-13T07:39:35.727 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.727 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y= 2022-09-13T07:39:35.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y=', 'Content-Length': '0'} 2022-09-13T07:39:35.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000330eb61bc971afc3-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.731 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:35.731 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:35.732 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.733 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.733 DEBUG:boto:Method: GET 2022-09-13T07:39:35.734 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.734 DEBUG:boto:Data: 2022-09-13T07:39:35.735 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.735 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.736 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.736 DEBUG:boto:Params: {} 2022-09-13T07:39:35.737 DEBUG:boto:Token: None 2022-09-13T07:39:35.738 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/ 2022-09-13T07:39:35.738 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc= 2022-09-13T07:39:35.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc=', 'Content-Length': '0'} 2022-09-13T07:39:35.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e853037122c5921a-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.742 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:35.743 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:35.744 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:35.744 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.745 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.745 DEBUG:boto:Method: GET 2022-09-13T07:39:35.746 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.746 DEBUG:boto:Data: 2022-09-13T07:39:35.747 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.747 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.748 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.748 DEBUG:boto:Params: {} 2022-09-13T07:39:35.749 DEBUG:boto:Token: None 2022-09-13T07:39:35.750 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/ 2022-09-13T07:39:35.751 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc= 2022-09-13T07:39:35.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc=', 'Content-Length': '0'} 2022-09-13T07:39:35.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ece9db5826dc7cd-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.756 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:35.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:35.759 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.760 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.760 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.761 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.761 DEBUG:boto:Method: GET 2022-09-13T07:39:35.762 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.762 DEBUG:boto:Data: 2022-09-13T07:39:35.763 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.764 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.765 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.766 DEBUG:boto:Params: {} 2022-09-13T07:39:35.767 DEBUG:boto:Token: None 2022-09-13T07:39:35.768 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.769 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:erD/Ol9JDqnfx6nSKXl2Ay9Z9x8= 2022-09-13T07:39:35.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:erD/Ol9JDqnfx6nSKXl2Ay9Z9x8=', 'Content-Length': '0'} 2022-09-13T07:39:35.772 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea69139607692df0-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.773 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.774 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.775 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.776 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.777 DEBUG:boto:Method: GET 2022-09-13T07:39:35.778 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.779 DEBUG:boto:Data: 2022-09-13T07:39:35.780 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.780 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.781 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.782 DEBUG:boto:Params: {} 2022-09-13T07:39:35.783 DEBUG:boto:Token: None 2022-09-13T07:39:35.784 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.785 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:erD/Ol9JDqnfx6nSKXl2Ay9Z9x8= 2022-09-13T07:39:35.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:erD/Ol9JDqnfx6nSKXl2Ay9Z9x8=', 'Content-Length': '0'} 2022-09-13T07:39:35.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000724137bfe6631af0-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.834 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.835 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.836 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.837 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.837 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.838 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.839 DEBUG:boto:Data: 2022-09-13T07:39:35.839 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.840 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.840 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.841 DEBUG:boto:Params: {} 2022-09-13T07:39:35.841 DEBUG:boto:Token: None 2022-09-13T07:39:35.842 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.843 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Sbp1PY+SB69KTNzYWIk+Y1oleYc= 2022-09-13T07:39:35.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Sbp1PY+SB69KTNzYWIk+Y1oleYc=', 'Content-Length': '0'} 2022-09-13T07:39:35.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e54f98c18430e9c8-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.846 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.847 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.848 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.848 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.849 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.850 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.850 DEBUG:boto:Data: 2022-09-13T07:39:35.851 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.851 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.852 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.852 DEBUG:boto:Params: {} 2022-09-13T07:39:35.853 DEBUG:boto:Token: None 2022-09-13T07:39:35.853 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.854 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Sbp1PY+SB69KTNzYWIk+Y1oleYc= 2022-09-13T07:39:35.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Sbp1PY+SB69KTNzYWIk+Y1oleYc=', 'Content-Length': '0'} 2022-09-13T07:39:35.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd84469ef8639d23-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:35.858 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.859 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.860 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.860 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.861 DEBUG:boto:Method: GET 2022-09-13T07:39:35.861 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.862 DEBUG:boto:Data: 2022-09-13T07:39:35.862 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.863 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.864 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.864 DEBUG:boto:Params: {} 2022-09-13T07:39:35.865 DEBUG:boto:Token: None 2022-09-13T07:39:35.865 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.866 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:erD/Ol9JDqnfx6nSKXl2Ay9Z9x8= 2022-09-13T07:39:35.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:erD/Ol9JDqnfx6nSKXl2Ay9Z9x8=', 'Content-Length': '0'} 2022-09-13T07:39:35.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2ba47e15ca4a4bc-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.870 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.871 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.871 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.872 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.873 DEBUG:boto:Method: GET 2022-09-13T07:39:35.874 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.874 DEBUG:boto:Data: 2022-09-13T07:39:35.875 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.876 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.877 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.878 DEBUG:boto:Params: {} 2022-09-13T07:39:35.879 DEBUG:boto:Token: None 2022-09-13T07:39:35.880 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:35.880 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:erD/Ol9JDqnfx6nSKXl2Ay9Z9x8= 2022-09-13T07:39:35.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:erD/Ol9JDqnfx6nSKXl2Ay9Z9x8=', 'Content-Length': '0'} 2022-09-13T07:39:35.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025def0e907b9887e-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.885 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:35.886 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:35.887 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.887 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.888 DEBUG:boto:Method: GET 2022-09-13T07:39:35.889 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.890 DEBUG:boto:Data: 2022-09-13T07:39:35.890 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.891 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.891 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.892 DEBUG:boto:Params: {} 2022-09-13T07:39:35.892 DEBUG:boto:Token: None 2022-09-13T07:39:35.894 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/ 2022-09-13T07:39:35.894 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc= 2022-09-13T07:39:35.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc=', 'Content-Length': '0'} 2022-09-13T07:39:35.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001123afc8b744bcaf-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.899 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:35.900 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:35.901 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:35.902 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.902 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.903 DEBUG:boto:Method: GET 2022-09-13T07:39:35.903 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.904 DEBUG:boto:Data: 2022-09-13T07:39:35.904 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.905 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.905 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.906 DEBUG:boto:Params: {} 2022-09-13T07:39:35.906 DEBUG:boto:Token: None 2022-09-13T07:39:35.907 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/ 2022-09-13T07:39:35.908 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc= 2022-09-13T07:39:35.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HubH27M9Og1Yfx7rvVyKm7HY5hc=', 'Content-Length': '0'} 2022-09-13T07:39:35.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a220efe7384b4143-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.912 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:35.913 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.914 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.914 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.915 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:35.915 DEBUG:boto:Data: 2022-09-13T07:39:35.916 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.916 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.917 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.917 DEBUG:boto:Params: {} 2022-09-13T07:39:35.918 DEBUG:boto:Token: None 2022-09-13T07:39:35.919 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj 2022-09-13T07:39:35.920 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+AYCBG0nPp5z0y8+4LXEA5b4p58= 2022-09-13T07:39:35.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+AYCBG0nPp5z0y8+4LXEA5b4p58=', 'Content-Length': '0'} 2022-09-13T07:39:35.924 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000580b3c72332f7181-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.925 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.925 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.926 DEBUG:boto:Method: HEAD 2022-09-13T07:39:35.926 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:35.927 DEBUG:boto:Data: 2022-09-13T07:39:35.927 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.928 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.928 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.929 DEBUG:boto:Params: {} 2022-09-13T07:39:35.929 DEBUG:boto:Token: None 2022-09-13T07:39:35.930 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj 2022-09-13T07:39:35.930 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+AYCBG0nPp5z0y8+4LXEA5b4p58= 2022-09-13T07:39:35.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+AYCBG0nPp5z0y8+4LXEA5b4p58=', 'Content-Length': '0'} 2022-09-13T07:39:35.934 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3e6048cb4d43d7a-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:35.935 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.936 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.936 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.937 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.937 DEBUG:boto:Method: GET 2022-09-13T07:39:35.938 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.938 DEBUG:boto:Data: 2022-09-13T07:39:35.939 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.939 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:35.940 DEBUG:boto:Port: 8001 2022-09-13T07:39:35.940 DEBUG:boto:Params: {} 2022-09-13T07:39:35.941 DEBUG:boto:Token: None 2022-09-13T07:39:35.941 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.942 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y= 2022-09-13T07:39:35.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y=', 'Content-Length': '0'} 2022-09-13T07:39:35.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008459a05d9efa86d3-00632033b7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.986 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:35.987 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:35.987 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.988 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.988 DEBUG:boto:Method: GET 2022-09-13T07:39:35.989 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.989 DEBUG:boto:Data: 2022-09-13T07:39:35.990 DEBUG:boto:Headers: {} 2022-09-13T07:39:35.990 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:35.991 DEBUG:boto:Port: 8000 2022-09-13T07:39:35.991 DEBUG:boto:Params: {} 2022-09-13T07:39:35.992 DEBUG:boto:Token: None 2022-09-13T07:39:35.993 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:35 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:35.993 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y= 2022-09-13T07:39:35.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:35 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z15iKdeJ+zh4wZns5iw7yvUTj1Y=', 'Content-Length': '0'} 2022-09-13T07:39:35.996 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc990abf54c0ef17-00632033b7-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:35.997 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:35.997 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:35.998 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.998 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:35.999 DEBUG:boto:Method: GET 2022-09-13T07:39:36.000 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.000 DEBUG:boto:Data: 2022-09-13T07:39:36.001 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.001 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.002 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.002 DEBUG:boto:Params: {} 2022-09-13T07:39:36.003 DEBUG:boto:Token: None 2022-09-13T07:39:36.003 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.004 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008679a7549fccf167-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.008 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.009 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.009 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.010 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.011 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.011 DEBUG:boto:Method: GET 2022-09-13T07:39:36.012 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.012 DEBUG:boto:Data: 2022-09-13T07:39:36.013 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.013 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.014 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.014 DEBUG:boto:Params: {} 2022-09-13T07:39:36.015 DEBUG:boto:Token: None 2022-09-13T07:39:36.016 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.016 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000159b12cc9973f65-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.020 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:36.022 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.022 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.023 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.023 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.024 DEBUG:boto:Method: GET 2022-09-13T07:39:36.024 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.025 DEBUG:boto:Data: 2022-09-13T07:39:36.025 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.026 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.026 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.027 DEBUG:boto:Params: {} 2022-09-13T07:39:36.028 DEBUG:boto:Token: None 2022-09-13T07:39:36.028 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.029 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FO1aKNi9P2bkbWjwtnUDFHCsoIE= 2022-09-13T07:39:36.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FO1aKNi9P2bkbWjwtnUDFHCsoIE=', 'Content-Length': '0'} 2022-09-13T07:39:36.031 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2b5dcdb58994083-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.032 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.033 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.033 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.034 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.034 DEBUG:boto:Method: GET 2022-09-13T07:39:36.035 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.035 DEBUG:boto:Data: 2022-09-13T07:39:36.036 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.036 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.037 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.037 DEBUG:boto:Params: {} 2022-09-13T07:39:36.038 DEBUG:boto:Token: None 2022-09-13T07:39:36.039 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.039 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FO1aKNi9P2bkbWjwtnUDFHCsoIE= 2022-09-13T07:39:36.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FO1aKNi9P2bkbWjwtnUDFHCsoIE=', 'Content-Length': '0'} 2022-09-13T07:39:36.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aba66fb54948a693-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.087 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.088 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.089 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.089 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.090 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.090 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.091 DEBUG:boto:Data: 2022-09-13T07:39:36.091 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.092 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.092 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.093 DEBUG:boto:Params: {} 2022-09-13T07:39:36.093 DEBUG:boto:Token: None 2022-09-13T07:39:36.094 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.095 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:w1Khtm6wrexVfETv08tbRWsZzmM= 2022-09-13T07:39:36.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:w1Khtm6wrexVfETv08tbRWsZzmM=', 'Content-Length': '0'} 2022-09-13T07:39:36.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf4c5b1bdeb47b81-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.099 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.099 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.100 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.100 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.101 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.101 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.102 DEBUG:boto:Data: 2022-09-13T07:39:36.102 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.103 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.103 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.104 DEBUG:boto:Params: {} 2022-09-13T07:39:36.104 DEBUG:boto:Token: None 2022-09-13T07:39:36.105 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.105 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:w1Khtm6wrexVfETv08tbRWsZzmM= 2022-09-13T07:39:36.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:w1Khtm6wrexVfETv08tbRWsZzmM=', 'Content-Length': '0'} 2022-09-13T07:39:36.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ff90083893ec923-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:36.109 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.110 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.110 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.111 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.111 DEBUG:boto:Method: GET 2022-09-13T07:39:36.112 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.112 DEBUG:boto:Data: 2022-09-13T07:39:36.113 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.113 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.114 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.114 DEBUG:boto:Params: {} 2022-09-13T07:39:36.115 DEBUG:boto:Token: None 2022-09-13T07:39:36.116 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.116 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FO1aKNi9P2bkbWjwtnUDFHCsoIE= 2022-09-13T07:39:36.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FO1aKNi9P2bkbWjwtnUDFHCsoIE=', 'Content-Length': '0'} 2022-09-13T07:39:36.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009053ffa13419a71f-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.120 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.120 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.121 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.121 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.122 DEBUG:boto:Method: GET 2022-09-13T07:39:36.122 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.123 DEBUG:boto:Data: 2022-09-13T07:39:36.123 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.124 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.124 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.125 DEBUG:boto:Params: {} 2022-09-13T07:39:36.126 DEBUG:boto:Token: None 2022-09-13T07:39:36.126 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:36.127 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FO1aKNi9P2bkbWjwtnUDFHCsoIE= 2022-09-13T07:39:36.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FO1aKNi9P2bkbWjwtnUDFHCsoIE=', 'Content-Length': '0'} 2022-09-13T07:39:36.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089533864c7ec12f3-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.130 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.131 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.131 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.132 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.132 DEBUG:boto:Method: GET 2022-09-13T07:39:36.133 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.133 DEBUG:boto:Data: 2022-09-13T07:39:36.134 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.134 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.135 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.135 DEBUG:boto:Params: {} 2022-09-13T07:39:36.136 DEBUG:boto:Token: None 2022-09-13T07:39:36.137 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.137 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac62123e83b1ff93-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.142 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.142 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.143 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.144 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.144 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.145 DEBUG:boto:Method: GET 2022-09-13T07:39:36.145 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.146 DEBUG:boto:Data: 2022-09-13T07:39:36.146 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.147 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.147 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.148 DEBUG:boto:Params: {} 2022-09-13T07:39:36.148 DEBUG:boto:Token: None 2022-09-13T07:39:36.149 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.149 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b94a1290f6ec5c29-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.154 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.155 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.155 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.156 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.156 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:36.157 DEBUG:boto:Data: 2022-09-13T07:39:36.157 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.158 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.158 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.159 DEBUG:boto:Params: {} 2022-09-13T07:39:36.159 DEBUG:boto:Token: None 2022-09-13T07:39:36.160 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj 2022-09-13T07:39:36.160 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE= 2022-09-13T07:39:36.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE=', 'Content-Length': '0'} 2022-09-13T07:39:36.164 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068278bfed8136624-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.165 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.166 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.166 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.167 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:36.167 DEBUG:boto:Data: 2022-09-13T07:39:36.168 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.168 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.169 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.169 DEBUG:boto:Params: {} 2022-09-13T07:39:36.170 DEBUG:boto:Token: None 2022-09-13T07:39:36.170 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj 2022-09-13T07:39:36.171 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE= 2022-09-13T07:39:36.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE=', 'Content-Length': '0'} 2022-09-13T07:39:36.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec0c2fd678045bc3-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:36.176 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.177 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.177 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.178 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.178 DEBUG:boto:Method: GET 2022-09-13T07:39:36.179 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.179 DEBUG:boto:Data: 2022-09-13T07:39:36.180 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.180 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.181 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.181 DEBUG:boto:Params: {} 2022-09-13T07:39:36.182 DEBUG:boto:Token: None 2022-09-13T07:39:36.182 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.183 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk= 2022-09-13T07:39:36.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk=', 'Content-Length': '0'} 2022-09-13T07:39:36.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a12829c42319c32-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.186 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.187 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.187 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.188 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.188 DEBUG:boto:Method: GET 2022-09-13T07:39:36.189 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.189 DEBUG:boto:Data: 2022-09-13T07:39:36.190 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.191 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.191 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.192 DEBUG:boto:Params: {} 2022-09-13T07:39:36.192 DEBUG:boto:Token: None 2022-09-13T07:39:36.193 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.193 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk= 2022-09-13T07:39:36.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk=', 'Content-Length': '0'} 2022-09-13T07:39:36.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f88fe4937ba8656-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.197 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.197 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.198 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.199 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.199 DEBUG:boto:Method: GET 2022-09-13T07:39:36.200 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.200 DEBUG:boto:Data: 2022-09-13T07:39:36.201 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.202 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.202 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.203 DEBUG:boto:Params: {} 2022-09-13T07:39:36.203 DEBUG:boto:Token: None 2022-09-13T07:39:36.204 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.204 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001257ca90ba091561-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.208 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.209 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.210 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.211 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.211 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.212 DEBUG:boto:Method: GET 2022-09-13T07:39:36.212 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.213 DEBUG:boto:Data: 2022-09-13T07:39:36.213 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.214 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.215 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.215 DEBUG:boto:Params: {} 2022-09-13T07:39:36.216 DEBUG:boto:Token: None 2022-09-13T07:39:36.217 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.217 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000369157cf25f34bc0-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.222 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:36.223 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.224 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.224 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.225 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.225 DEBUG:boto:Method: GET 2022-09-13T07:39:36.226 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.226 DEBUG:boto:Data: 2022-09-13T07:39:36.227 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.227 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.228 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.228 DEBUG:boto:Params: {} 2022-09-13T07:39:36.229 DEBUG:boto:Token: None 2022-09-13T07:39:36.230 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.231 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CJrZPXh/Ax8wIxiKndkhGgJSUIQ= 2022-09-13T07:39:36.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CJrZPXh/Ax8wIxiKndkhGgJSUIQ=', 'Content-Length': '0'} 2022-09-13T07:39:36.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071829708230a7cfc-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.234 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.235 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.235 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.236 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.236 DEBUG:boto:Method: GET 2022-09-13T07:39:36.237 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.237 DEBUG:boto:Data: 2022-09-13T07:39:36.238 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.239 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.239 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.240 DEBUG:boto:Params: {} 2022-09-13T07:39:36.241 DEBUG:boto:Token: None 2022-09-13T07:39:36.241 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.242 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CJrZPXh/Ax8wIxiKndkhGgJSUIQ= 2022-09-13T07:39:36.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CJrZPXh/Ax8wIxiKndkhGgJSUIQ=', 'Content-Length': '0'} 2022-09-13T07:39:36.244 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fda7c9422e810b0b-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.245 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.246 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.247 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.247 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.248 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.248 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.249 DEBUG:boto:Data: 2022-09-13T07:39:36.249 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.250 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.251 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.251 DEBUG:boto:Params: {} 2022-09-13T07:39:36.252 DEBUG:boto:Token: None 2022-09-13T07:39:36.252 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.253 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6EFl1djTkoBiLAE+mznebXvyvLQ= 2022-09-13T07:39:36.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6EFl1djTkoBiLAE+mznebXvyvLQ=', 'Content-Length': '0'} 2022-09-13T07:39:36.256 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8aaa286b280daa2-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.257 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.258 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.259 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.260 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.260 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.261 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.262 DEBUG:boto:Data: 2022-09-13T07:39:36.263 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.264 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.264 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.265 DEBUG:boto:Params: {} 2022-09-13T07:39:36.265 DEBUG:boto:Token: None 2022-09-13T07:39:36.266 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.266 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6EFl1djTkoBiLAE+mznebXvyvLQ= 2022-09-13T07:39:36.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6EFl1djTkoBiLAE+mznebXvyvLQ=', 'Content-Length': '0'} 2022-09-13T07:39:36.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059ee370065028a7c-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:36.271 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.272 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.273 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.274 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.274 DEBUG:boto:Method: GET 2022-09-13T07:39:36.275 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.275 DEBUG:boto:Data: 2022-09-13T07:39:36.276 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.276 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.277 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.277 DEBUG:boto:Params: {} 2022-09-13T07:39:36.278 DEBUG:boto:Token: None 2022-09-13T07:39:36.279 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.279 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CJrZPXh/Ax8wIxiKndkhGgJSUIQ= 2022-09-13T07:39:36.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CJrZPXh/Ax8wIxiKndkhGgJSUIQ=', 'Content-Length': '0'} 2022-09-13T07:39:36.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000335e494898c20071-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.283 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.283 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.284 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.284 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.285 DEBUG:boto:Method: GET 2022-09-13T07:39:36.286 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.286 DEBUG:boto:Data: 2022-09-13T07:39:36.287 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.287 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.288 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.288 DEBUG:boto:Params: {} 2022-09-13T07:39:36.289 DEBUG:boto:Token: None 2022-09-13T07:39:36.289 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:36.290 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CJrZPXh/Ax8wIxiKndkhGgJSUIQ= 2022-09-13T07:39:36.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CJrZPXh/Ax8wIxiKndkhGgJSUIQ=', 'Content-Length': '0'} 2022-09-13T07:39:36.293 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003be215e5008f43ea-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.294 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.294 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.295 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.296 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.297 DEBUG:boto:Method: GET 2022-09-13T07:39:36.297 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.298 DEBUG:boto:Data: 2022-09-13T07:39:36.299 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.300 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.301 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.301 DEBUG:boto:Params: {} 2022-09-13T07:39:36.302 DEBUG:boto:Token: None 2022-09-13T07:39:36.303 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.304 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009051102bb2b74ea5-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.350 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.351 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.352 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.352 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.353 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.353 DEBUG:boto:Method: GET 2022-09-13T07:39:36.354 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.354 DEBUG:boto:Data: 2022-09-13T07:39:36.355 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.355 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.356 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.356 DEBUG:boto:Params: {} 2022-09-13T07:39:36.357 DEBUG:boto:Token: None 2022-09-13T07:39:36.358 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.358 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b7c1343cfba7008-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.363 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.363 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.364 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.364 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.365 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:36.365 DEBUG:boto:Data: 2022-09-13T07:39:36.366 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.366 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.367 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.367 DEBUG:boto:Params: {} 2022-09-13T07:39:36.368 DEBUG:boto:Token: None 2022-09-13T07:39:36.369 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj 2022-09-13T07:39:36.369 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE= 2022-09-13T07:39:36.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE=', 'Content-Length': '0'} 2022-09-13T07:39:36.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097dbb43652bc6290-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.374 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.374 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.375 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.375 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:36.376 DEBUG:boto:Data: 2022-09-13T07:39:36.376 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.377 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.377 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.378 DEBUG:boto:Params: {} 2022-09-13T07:39:36.379 DEBUG:boto:Token: None 2022-09-13T07:39:36.379 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj 2022-09-13T07:39:36.380 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE= 2022-09-13T07:39:36.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE=', 'Content-Length': '0'} 2022-09-13T07:39:36.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f26f0a3ae674613-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:36.385 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.385 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.386 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.386 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.387 DEBUG:boto:Method: GET 2022-09-13T07:39:36.388 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.388 DEBUG:boto:Data: 2022-09-13T07:39:36.389 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.389 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.390 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.390 DEBUG:boto:Params: {} 2022-09-13T07:39:36.391 DEBUG:boto:Token: None 2022-09-13T07:39:36.392 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.392 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk= 2022-09-13T07:39:36.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk=', 'Content-Length': '0'} 2022-09-13T07:39:36.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fda2b428133ca382-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.396 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.396 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.397 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.397 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.398 DEBUG:boto:Method: GET 2022-09-13T07:39:36.398 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.399 DEBUG:boto:Data: 2022-09-13T07:39:36.400 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.400 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.401 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.401 DEBUG:boto:Params: {} 2022-09-13T07:39:36.402 DEBUG:boto:Token: None 2022-09-13T07:39:36.403 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.403 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk= 2022-09-13T07:39:36.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk=', 'Content-Length': '0'} 2022-09-13T07:39:36.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a826bab56cbf92a2-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.454 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.455 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.455 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.456 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.456 DEBUG:boto:Method: GET 2022-09-13T07:39:36.457 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.458 DEBUG:boto:Data: 2022-09-13T07:39:36.458 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.459 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.459 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.460 DEBUG:boto:Params: {} 2022-09-13T07:39:36.460 DEBUG:boto:Token: None 2022-09-13T07:39:36.461 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.462 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dba32085f6cf6a83-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.466 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.467 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.467 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.468 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.468 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.469 DEBUG:boto:Method: GET 2022-09-13T07:39:36.469 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.470 DEBUG:boto:Data: 2022-09-13T07:39:36.470 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.471 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.471 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.472 DEBUG:boto:Params: {} 2022-09-13T07:39:36.473 DEBUG:boto:Token: None 2022-09-13T07:39:36.473 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.474 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4626111dc1fbff4-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.478 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:36.480 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.480 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.481 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.481 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.482 DEBUG:boto:Method: GET 2022-09-13T07:39:36.483 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.483 DEBUG:boto:Data: 2022-09-13T07:39:36.484 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.484 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.485 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.485 DEBUG:boto:Params: {} 2022-09-13T07:39:36.486 DEBUG:boto:Token: None 2022-09-13T07:39:36.486 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.487 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5RpTbkWnzj7xp5F16K+CTRpvxmI= 2022-09-13T07:39:36.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5RpTbkWnzj7xp5F16K+CTRpvxmI=', 'Content-Length': '0'} 2022-09-13T07:39:36.490 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f92cfe3f0d1f8cef-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.490 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.491 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.491 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.492 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.493 DEBUG:boto:Method: GET 2022-09-13T07:39:36.493 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.494 DEBUG:boto:Data: 2022-09-13T07:39:36.494 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.495 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.495 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.496 DEBUG:boto:Params: {} 2022-09-13T07:39:36.497 DEBUG:boto:Token: None 2022-09-13T07:39:36.497 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.498 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5RpTbkWnzj7xp5F16K+CTRpvxmI= 2022-09-13T07:39:36.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5RpTbkWnzj7xp5F16K+CTRpvxmI=', 'Content-Length': '0'} 2022-09-13T07:39:36.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014ca1131e935e56d-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.501 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.502 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.503 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.503 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.504 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.504 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.505 DEBUG:boto:Data: 2022-09-13T07:39:36.505 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.506 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.506 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.507 DEBUG:boto:Params: {} 2022-09-13T07:39:36.507 DEBUG:boto:Token: None 2022-09-13T07:39:36.508 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.508 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FEwLapQS1JYUa+a6qjfTxY0be3c= 2022-09-13T07:39:36.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FEwLapQS1JYUa+a6qjfTxY0be3c=', 'Content-Length': '0'} 2022-09-13T07:39:36.511 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b758d617f53ef431-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.512 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.512 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.513 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.513 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.514 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.514 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.515 DEBUG:boto:Data: 2022-09-13T07:39:36.515 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.516 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.516 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.517 DEBUG:boto:Params: {} 2022-09-13T07:39:36.517 DEBUG:boto:Token: None 2022-09-13T07:39:36.518 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.518 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FEwLapQS1JYUa+a6qjfTxY0be3c= 2022-09-13T07:39:36.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FEwLapQS1JYUa+a6qjfTxY0be3c=', 'Content-Length': '0'} 2022-09-13T07:39:36.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000541a876be2f8c615-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:36.522 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.523 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.523 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.524 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.524 DEBUG:boto:Method: GET 2022-09-13T07:39:36.525 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.525 DEBUG:boto:Data: 2022-09-13T07:39:36.526 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.526 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.527 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.527 DEBUG:boto:Params: {} 2022-09-13T07:39:36.528 DEBUG:boto:Token: None 2022-09-13T07:39:36.529 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.529 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5RpTbkWnzj7xp5F16K+CTRpvxmI= 2022-09-13T07:39:36.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5RpTbkWnzj7xp5F16K+CTRpvxmI=', 'Content-Length': '0'} 2022-09-13T07:39:36.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006924ddec42399e4f-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.533 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.534 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.535 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.536 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.536 DEBUG:boto:Method: GET 2022-09-13T07:39:36.537 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.537 DEBUG:boto:Data: 2022-09-13T07:39:36.538 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.539 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.539 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.540 DEBUG:boto:Params: {} 2022-09-13T07:39:36.541 DEBUG:boto:Token: None 2022-09-13T07:39:36.541 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:36.542 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5RpTbkWnzj7xp5F16K+CTRpvxmI= 2022-09-13T07:39:36.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5RpTbkWnzj7xp5F16K+CTRpvxmI=', 'Content-Length': '0'} 2022-09-13T07:39:36.545 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef8d51050f52c24c-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.546 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.547 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.548 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.548 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.549 DEBUG:boto:Method: GET 2022-09-13T07:39:36.549 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.550 DEBUG:boto:Data: 2022-09-13T07:39:36.550 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.551 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.551 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.552 DEBUG:boto:Params: {} 2022-09-13T07:39:36.552 DEBUG:boto:Token: None 2022-09-13T07:39:36.553 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.554 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084c8a7c28fdb5977-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.558 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.559 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.560 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.560 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.561 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.561 DEBUG:boto:Method: GET 2022-09-13T07:39:36.562 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.563 DEBUG:boto:Data: 2022-09-13T07:39:36.563 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.564 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.564 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.565 DEBUG:boto:Params: {} 2022-09-13T07:39:36.565 DEBUG:boto:Token: None 2022-09-13T07:39:36.566 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.567 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087bfd0a4e539157b-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.573 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.574 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.575 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.576 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.577 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:36.578 DEBUG:boto:Data: 2022-09-13T07:39:36.579 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.580 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.580 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.581 DEBUG:boto:Params: {} 2022-09-13T07:39:36.582 DEBUG:boto:Token: None 2022-09-13T07:39:36.593 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj 2022-09-13T07:39:36.594 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE= 2022-09-13T07:39:36.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE=', 'Content-Length': '0'} 2022-09-13T07:39:36.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b16cfff5107f1f00-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.599 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.600 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.600 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.601 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:36.601 DEBUG:boto:Data: 2022-09-13T07:39:36.602 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.602 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.603 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.604 DEBUG:boto:Params: {} 2022-09-13T07:39:36.604 DEBUG:boto:Token: None 2022-09-13T07:39:36.605 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj 2022-09-13T07:39:36.606 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE= 2022-09-13T07:39:36.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE=', 'Content-Length': '0'} 2022-09-13T07:39:36.610 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe9f2bc10d6a81bf-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:36.612 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.613 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.616 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.616 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.617 DEBUG:boto:Method: GET 2022-09-13T07:39:36.617 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.618 DEBUG:boto:Data: 2022-09-13T07:39:36.618 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.619 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.619 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.620 DEBUG:boto:Params: {} 2022-09-13T07:39:36.620 DEBUG:boto:Token: None 2022-09-13T07:39:36.621 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.622 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk= 2022-09-13T07:39:36.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk=', 'Content-Length': '0'} 2022-09-13T07:39:36.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000214bb32a5c90ed96-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.626 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.626 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.627 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.627 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.628 DEBUG:boto:Method: GET 2022-09-13T07:39:36.628 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.629 DEBUG:boto:Data: 2022-09-13T07:39:36.629 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.630 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.631 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.631 DEBUG:boto:Params: {} 2022-09-13T07:39:36.632 DEBUG:boto:Token: None 2022-09-13T07:39:36.632 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.633 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk= 2022-09-13T07:39:36.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk=', 'Content-Length': '0'} 2022-09-13T07:39:36.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e171a6d6533fc41-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.638 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.638 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.639 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.639 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.640 DEBUG:boto:Method: GET 2022-09-13T07:39:36.640 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.641 DEBUG:boto:Data: 2022-09-13T07:39:36.641 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.642 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.643 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.644 DEBUG:boto:Params: {} 2022-09-13T07:39:36.645 DEBUG:boto:Token: None 2022-09-13T07:39:36.647 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.648 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000883c2535286f379a-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.653 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.654 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.655 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.655 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.656 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.657 DEBUG:boto:Method: GET 2022-09-13T07:39:36.658 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.659 DEBUG:boto:Data: 2022-09-13T07:39:36.659 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.660 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.660 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.661 DEBUG:boto:Params: {} 2022-09-13T07:39:36.661 DEBUG:boto:Token: None 2022-09-13T07:39:36.662 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.663 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b46dbfd5d0e6bc1f-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.667 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:36.669 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.669 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.670 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.670 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.671 DEBUG:boto:Method: GET 2022-09-13T07:39:36.671 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.672 DEBUG:boto:Data: 2022-09-13T07:39:36.673 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.673 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.674 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.674 DEBUG:boto:Params: {} 2022-09-13T07:39:36.675 DEBUG:boto:Token: None 2022-09-13T07:39:36.675 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.676 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S+SQ+wQp0Tzup4SoVg91gMMkC1s= 2022-09-13T07:39:36.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S+SQ+wQp0Tzup4SoVg91gMMkC1s=', 'Content-Length': '0'} 2022-09-13T07:39:36.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000836941f0592d31b5-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.680 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.680 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.681 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.681 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.682 DEBUG:boto:Method: GET 2022-09-13T07:39:36.682 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.683 DEBUG:boto:Data: 2022-09-13T07:39:36.683 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.684 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.684 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.685 DEBUG:boto:Params: {} 2022-09-13T07:39:36.685 DEBUG:boto:Token: None 2022-09-13T07:39:36.686 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.687 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S+SQ+wQp0Tzup4SoVg91gMMkC1s= 2022-09-13T07:39:36.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S+SQ+wQp0Tzup4SoVg91gMMkC1s=', 'Content-Length': '0'} 2022-09-13T07:39:36.690 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ec37f000e806943-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.730 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.731 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.731 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.732 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.732 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.733 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.733 DEBUG:boto:Data: 2022-09-13T07:39:36.734 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.734 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.735 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.735 DEBUG:boto:Params: {} 2022-09-13T07:39:36.736 DEBUG:boto:Token: None 2022-09-13T07:39:36.736 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.737 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aw43JZqpFyGyAEpoz/oYRM4W7gw= 2022-09-13T07:39:36.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aw43JZqpFyGyAEpoz/oYRM4W7gw=', 'Content-Length': '0'} 2022-09-13T07:39:36.740 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003395e3401c9e28b5-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.742 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.743 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.743 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.743 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.744 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.744 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.745 DEBUG:boto:Data: 2022-09-13T07:39:36.745 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.746 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.747 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.747 DEBUG:boto:Params: {} 2022-09-13T07:39:36.748 DEBUG:boto:Token: None 2022-09-13T07:39:36.749 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.749 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aw43JZqpFyGyAEpoz/oYRM4W7gw= 2022-09-13T07:39:36.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aw43JZqpFyGyAEpoz/oYRM4W7gw=', 'Content-Length': '0'} 2022-09-13T07:39:36.752 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cc69eaf54a0092c-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:36.753 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.754 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.754 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.755 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.755 DEBUG:boto:Method: GET 2022-09-13T07:39:36.756 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.757 DEBUG:boto:Data: 2022-09-13T07:39:36.757 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.758 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.758 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.759 DEBUG:boto:Params: {} 2022-09-13T07:39:36.759 DEBUG:boto:Token: None 2022-09-13T07:39:36.760 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.761 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S+SQ+wQp0Tzup4SoVg91gMMkC1s= 2022-09-13T07:39:36.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S+SQ+wQp0Tzup4SoVg91gMMkC1s=', 'Content-Length': '0'} 2022-09-13T07:39:36.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7d4d2428b209f11-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.765 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.766 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.766 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.767 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.767 DEBUG:boto:Method: GET 2022-09-13T07:39:36.768 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.768 DEBUG:boto:Data: 2022-09-13T07:39:36.769 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.769 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.770 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.770 DEBUG:boto:Params: {} 2022-09-13T07:39:36.771 DEBUG:boto:Token: None 2022-09-13T07:39:36.772 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:36.772 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:S+SQ+wQp0Tzup4SoVg91gMMkC1s= 2022-09-13T07:39:36.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:S+SQ+wQp0Tzup4SoVg91gMMkC1s=', 'Content-Length': '0'} 2022-09-13T07:39:36.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b72888c32fb65c13-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.776 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.776 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.777 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.777 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.778 DEBUG:boto:Method: GET 2022-09-13T07:39:36.779 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.779 DEBUG:boto:Data: 2022-09-13T07:39:36.780 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.780 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.781 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.781 DEBUG:boto:Params: {} 2022-09-13T07:39:36.782 DEBUG:boto:Token: None 2022-09-13T07:39:36.782 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.783 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcb30cf1eb6506d2-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.787 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.788 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.788 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.789 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.789 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.790 DEBUG:boto:Method: GET 2022-09-13T07:39:36.791 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.791 DEBUG:boto:Data: 2022-09-13T07:39:36.792 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.792 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.793 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.793 DEBUG:boto:Params: {} 2022-09-13T07:39:36.794 DEBUG:boto:Token: None 2022-09-13T07:39:36.795 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.795 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000569ed02a4e6f4b3b-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.800 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.800 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.801 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.802 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.802 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:36.803 DEBUG:boto:Data: 2022-09-13T07:39:36.803 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.804 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.804 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.805 DEBUG:boto:Params: {} 2022-09-13T07:39:36.805 DEBUG:boto:Token: None 2022-09-13T07:39:36.806 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj 2022-09-13T07:39:36.806 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE= 2022-09-13T07:39:36.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE=', 'Content-Length': '0'} 2022-09-13T07:39:36.810 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000180d00ce1b2bffdd-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.811 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.812 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.812 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.813 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:36.813 DEBUG:boto:Data: 2022-09-13T07:39:36.814 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.814 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.815 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.815 DEBUG:boto:Params: {} 2022-09-13T07:39:36.816 DEBUG:boto:Token: None 2022-09-13T07:39:36.816 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj 2022-09-13T07:39:36.817 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE= 2022-09-13T07:39:36.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mkdIOWwsacYhAGWx35ywW0cZxaE=', 'Content-Length': '0'} 2022-09-13T07:39:36.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000111c12ac4fabfe90-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:36.822 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.823 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.823 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.824 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.824 DEBUG:boto:Method: GET 2022-09-13T07:39:36.825 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.825 DEBUG:boto:Data: 2022-09-13T07:39:36.826 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.826 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.827 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.827 DEBUG:boto:Params: {} 2022-09-13T07:39:36.828 DEBUG:boto:Token: None 2022-09-13T07:39:36.829 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.829 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk= 2022-09-13T07:39:36.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk=', 'Content-Length': '0'} 2022-09-13T07:39:36.832 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bef16d72bcda191-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.878 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.879 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.880 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.880 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.881 DEBUG:boto:Method: GET 2022-09-13T07:39:36.881 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.882 DEBUG:boto:Data: 2022-09-13T07:39:36.883 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.883 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.884 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.884 DEBUG:boto:Params: {} 2022-09-13T07:39:36.885 DEBUG:boto:Token: None 2022-09-13T07:39:36.886 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:36.887 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk= 2022-09-13T07:39:36.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hZ4nWDJutC9naI5Gu3fLhiMc2uk=', 'Content-Length': '0'} 2022-09-13T07:39:36.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad636b9c7ed0efc7-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.891 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.892 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.893 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.894 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.895 DEBUG:boto:Method: GET 2022-09-13T07:39:36.896 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.897 DEBUG:boto:Data: 2022-09-13T07:39:36.898 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.898 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.899 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.900 DEBUG:boto:Params: {} 2022-09-13T07:39:36.900 DEBUG:boto:Token: None 2022-09-13T07:39:36.901 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.902 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076f2ab790fe63e3a-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.907 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.907 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:36.908 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:36.909 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.909 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.910 DEBUG:boto:Method: GET 2022-09-13T07:39:36.910 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:36.911 DEBUG:boto:Data: 2022-09-13T07:39:36.911 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.912 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.913 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.914 DEBUG:boto:Params: {} 2022-09-13T07:39:36.915 DEBUG:boto:Token: None 2022-09-13T07:39:36.915 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/ 2022-09-13T07:39:36.916 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40= 2022-09-13T07:39:36.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RDu5joFRdeJ44uABblzxCWGq40=', 'Content-Length': '0'} 2022-09-13T07:39:36.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2343479b9f563d7-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.921 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:36.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:36.924 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.925 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.925 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.926 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.926 DEBUG:boto:Method: GET 2022-09-13T07:39:36.927 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.927 DEBUG:boto:Data: 2022-09-13T07:39:36.928 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.928 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.929 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.929 DEBUG:boto:Params: {} 2022-09-13T07:39:36.930 DEBUG:boto:Token: None 2022-09-13T07:39:36.931 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.931 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wjOtI2auwCt3MP0TswoJYimBze0= 2022-09-13T07:39:36.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wjOtI2auwCt3MP0TswoJYimBze0=', 'Content-Length': '0'} 2022-09-13T07:39:36.934 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040d9a919a07436fc-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.935 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.935 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.936 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.936 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.937 DEBUG:boto:Method: GET 2022-09-13T07:39:36.938 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.938 DEBUG:boto:Data: 2022-09-13T07:39:36.939 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.939 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.940 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.940 DEBUG:boto:Params: {} 2022-09-13T07:39:36.941 DEBUG:boto:Token: None 2022-09-13T07:39:36.941 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.942 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wjOtI2auwCt3MP0TswoJYimBze0= 2022-09-13T07:39:36.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wjOtI2auwCt3MP0TswoJYimBze0=', 'Content-Length': '0'} 2022-09-13T07:39:36.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007468e93cd5af6248-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.946 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.946 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.947 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.947 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.948 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.948 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.949 DEBUG:boto:Data: 2022-09-13T07:39:36.949 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.949 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.950 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.951 DEBUG:boto:Params: {} 2022-09-13T07:39:36.951 DEBUG:boto:Token: None 2022-09-13T07:39:36.952 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.952 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AVcgG2GJNlKg66LtQZWYE8pwIgI= 2022-09-13T07:39:36.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AVcgG2GJNlKg66LtQZWYE8pwIgI=', 'Content-Length': '0'} 2022-09-13T07:39:36.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074cb8def6c7a55c6-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.956 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.956 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.957 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.957 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.958 DEBUG:boto:Method: HEAD 2022-09-13T07:39:36.958 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.959 DEBUG:boto:Data: 2022-09-13T07:39:36.959 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.959 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:36.960 DEBUG:boto:Port: 8000 2022-09-13T07:39:36.960 DEBUG:boto:Params: {} 2022-09-13T07:39:36.961 DEBUG:boto:Token: None 2022-09-13T07:39:36.961 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.962 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AVcgG2GJNlKg66LtQZWYE8pwIgI= 2022-09-13T07:39:36.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AVcgG2GJNlKg66LtQZWYE8pwIgI=', 'Content-Length': '0'} 2022-09-13T07:39:36.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e31b85e03b7a0dd6-00632033b8-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:36.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:36.966 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:36.966 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:36.967 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.967 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.968 DEBUG:boto:Method: GET 2022-09-13T07:39:36.968 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.969 DEBUG:boto:Data: 2022-09-13T07:39:36.969 DEBUG:boto:Headers: {} 2022-09-13T07:39:36.970 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:36.970 DEBUG:boto:Port: 8001 2022-09-13T07:39:36.971 DEBUG:boto:Params: {} 2022-09-13T07:39:36.972 DEBUG:boto:Token: None 2022-09-13T07:39:36.972 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:36 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:36.973 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wjOtI2auwCt3MP0TswoJYimBze0= 2022-09-13T07:39:36.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wjOtI2auwCt3MP0TswoJYimBze0=', 'Content-Length': '0'} 2022-09-13T07:39:36.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a7f53a0d099ec5e-00632033b8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.018 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.019 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.020 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:37.021 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:37.022 DEBUG:boto:Method: GET 2022-09-13T07:39:37.023 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:37.024 DEBUG:boto:Data: 2022-09-13T07:39:37.025 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.025 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.026 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.027 DEBUG:boto:Params: {} 2022-09-13T07:39:37.028 DEBUG:boto:Token: None 2022-09-13T07:39:37.029 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:37.030 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zJ0M4ZCWv3KKhONtCWSoVMUzlCE= 2022-09-13T07:39:37.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zJ0M4ZCWv3KKhONtCWSoVMUzlCE=', 'Content-Length': '0'} 2022-09-13T07:39:37.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fe93c7913601731-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.035 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.037 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.038 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.038 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.039 DEBUG:boto:Method: GET 2022-09-13T07:39:37.039 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.040 DEBUG:boto:Data: 2022-09-13T07:39:37.040 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.041 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.041 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.042 DEBUG:boto:Params: {} 2022-09-13T07:39:37.042 DEBUG:boto:Token: None 2022-09-13T07:39:37.043 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.044 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c508726bc701702a-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.048 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.049 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.049 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.050 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.050 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.051 DEBUG:boto:Method: GET 2022-09-13T07:39:37.051 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.052 DEBUG:boto:Data: 2022-09-13T07:39:37.052 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.053 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.053 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.054 DEBUG:boto:Params: {} 2022-09-13T07:39:37.054 DEBUG:boto:Token: None 2022-09-13T07:39:37.055 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.055 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f874342a484ca4c5-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.060 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.060 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.061 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.062 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.062 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:37.062 DEBUG:boto:Data: 2022-09-13T07:39:37.063 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.063 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.064 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.064 DEBUG:boto:Params: {} 2022-09-13T07:39:37.065 DEBUG:boto:Token: None 2022-09-13T07:39:37.065 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj 2022-09-13T07:39:37.066 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI= 2022-09-13T07:39:37.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI=', 'Content-Length': '0'} 2022-09-13T07:39:37.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024a36775352d6fb2-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.071 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.071 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.072 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.072 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:37.073 DEBUG:boto:Data: 2022-09-13T07:39:37.073 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.074 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.074 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.075 DEBUG:boto:Params: {} 2022-09-13T07:39:37.075 DEBUG:boto:Token: None 2022-09-13T07:39:37.076 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj 2022-09-13T07:39:37.077 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI= 2022-09-13T07:39:37.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI=', 'Content-Length': '0'} 2022-09-13T07:39:37.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000206bab1377aad10a-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:37.082 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.082 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.083 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.083 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.084 DEBUG:boto:Method: GET 2022-09-13T07:39:37.085 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.085 DEBUG:boto:Data: 2022-09-13T07:39:37.086 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.086 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.087 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.087 DEBUG:boto:Params: {} 2022-09-13T07:39:37.088 DEBUG:boto:Token: None 2022-09-13T07:39:37.088 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.089 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM= 2022-09-13T07:39:37.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM=', 'Content-Length': '0'} 2022-09-13T07:39:37.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b73b1376320c0ce4-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.134 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.135 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.136 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.137 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.138 DEBUG:boto:Method: GET 2022-09-13T07:39:37.139 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.139 DEBUG:boto:Data: 2022-09-13T07:39:37.140 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.140 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.141 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.141 DEBUG:boto:Params: {} 2022-09-13T07:39:37.142 DEBUG:boto:Token: None 2022-09-13T07:39:37.142 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.143 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM= 2022-09-13T07:39:37.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM=', 'Content-Length': '0'} 2022-09-13T07:39:37.146 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e50d9e582dfbd2c-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.147 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.148 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.148 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.149 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.150 DEBUG:boto:Method: GET 2022-09-13T07:39:37.150 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.151 DEBUG:boto:Data: 2022-09-13T07:39:37.151 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.152 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.152 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.153 DEBUG:boto:Params: {} 2022-09-13T07:39:37.153 DEBUG:boto:Token: None 2022-09-13T07:39:37.156 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.157 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000385c3087f77c5c6b-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.162 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.163 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.163 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.164 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.164 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.165 DEBUG:boto:Method: GET 2022-09-13T07:39:37.165 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.166 DEBUG:boto:Data: 2022-09-13T07:39:37.167 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.168 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.169 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.170 DEBUG:boto:Params: {} 2022-09-13T07:39:37.170 DEBUG:boto:Token: None 2022-09-13T07:39:37.171 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.172 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011fc92daf228d867-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.177 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:37.179 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.180 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.181 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.182 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.183 DEBUG:boto:Method: GET 2022-09-13T07:39:37.183 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.184 DEBUG:boto:Data: 2022-09-13T07:39:37.185 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.185 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.186 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.187 DEBUG:boto:Params: {} 2022-09-13T07:39:37.187 DEBUG:boto:Token: None 2022-09-13T07:39:37.188 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.188 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bGpvVZW1NXNQ7pY+i/7yUAMPkrk= 2022-09-13T07:39:37.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bGpvVZW1NXNQ7pY+i/7yUAMPkrk=', 'Content-Length': '0'} 2022-09-13T07:39:37.191 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da60de1ab1ab754b-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.192 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.192 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.193 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.193 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.194 DEBUG:boto:Method: GET 2022-09-13T07:39:37.195 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.195 DEBUG:boto:Data: 2022-09-13T07:39:37.196 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.196 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.197 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.197 DEBUG:boto:Params: {} 2022-09-13T07:39:37.198 DEBUG:boto:Token: None 2022-09-13T07:39:37.199 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.199 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bGpvVZW1NXNQ7pY+i/7yUAMPkrk= 2022-09-13T07:39:37.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bGpvVZW1NXNQ7pY+i/7yUAMPkrk=', 'Content-Length': '0'} 2022-09-13T07:39:37.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000949b4aa06c1617de-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.203 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.203 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.204 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.204 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.205 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.205 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.206 DEBUG:boto:Data: 2022-09-13T07:39:37.206 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.207 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.208 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.208 DEBUG:boto:Params: {} 2022-09-13T07:39:37.209 DEBUG:boto:Token: None 2022-09-13T07:39:37.209 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.210 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uSr29tYiMWTU7biz/msWAWHgv+A= 2022-09-13T07:39:37.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uSr29tYiMWTU7biz/msWAWHgv+A=', 'Content-Length': '0'} 2022-09-13T07:39:37.212 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba975be063c73cfa-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.213 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.213 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.214 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.214 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.215 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.215 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.216 DEBUG:boto:Data: 2022-09-13T07:39:37.216 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.217 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.217 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.218 DEBUG:boto:Params: {} 2022-09-13T07:39:37.218 DEBUG:boto:Token: None 2022-09-13T07:39:37.219 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.220 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uSr29tYiMWTU7biz/msWAWHgv+A= 2022-09-13T07:39:37.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uSr29tYiMWTU7biz/msWAWHgv+A=', 'Content-Length': '0'} 2022-09-13T07:39:37.222 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029b061e84f16b31e-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:37.223 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.224 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.224 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.225 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.226 DEBUG:boto:Method: GET 2022-09-13T07:39:37.226 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.227 DEBUG:boto:Data: 2022-09-13T07:39:37.227 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.228 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.228 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.229 DEBUG:boto:Params: {} 2022-09-13T07:39:37.229 DEBUG:boto:Token: None 2022-09-13T07:39:37.230 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.231 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bGpvVZW1NXNQ7pY+i/7yUAMPkrk= 2022-09-13T07:39:37.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bGpvVZW1NXNQ7pY+i/7yUAMPkrk=', 'Content-Length': '0'} 2022-09-13T07:39:37.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c0203ef0098be0d-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.234 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.235 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.235 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.236 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.236 DEBUG:boto:Method: GET 2022-09-13T07:39:37.237 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.237 DEBUG:boto:Data: 2022-09-13T07:39:37.238 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.238 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.239 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.239 DEBUG:boto:Params: {} 2022-09-13T07:39:37.240 DEBUG:boto:Token: None 2022-09-13T07:39:37.241 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:37.241 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bGpvVZW1NXNQ7pY+i/7yUAMPkrk= 2022-09-13T07:39:37.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bGpvVZW1NXNQ7pY+i/7yUAMPkrk=', 'Content-Length': '0'} 2022-09-13T07:39:37.244 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b0f439461ff464d-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.286 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.287 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.288 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.289 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.290 DEBUG:boto:Method: GET 2022-09-13T07:39:37.291 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.292 DEBUG:boto:Data: 2022-09-13T07:39:37.293 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.294 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.295 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.296 DEBUG:boto:Params: {} 2022-09-13T07:39:37.297 DEBUG:boto:Token: None 2022-09-13T07:39:37.298 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.299 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001604fc826aa81d75-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.305 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.306 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.307 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.307 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.308 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.308 DEBUG:boto:Method: GET 2022-09-13T07:39:37.309 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.309 DEBUG:boto:Data: 2022-09-13T07:39:37.310 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.311 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.311 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.312 DEBUG:boto:Params: {} 2022-09-13T07:39:37.312 DEBUG:boto:Token: None 2022-09-13T07:39:37.313 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.314 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c209c1e7150e994b-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.318 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.320 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.320 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.321 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.321 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:37.322 DEBUG:boto:Data: 2022-09-13T07:39:37.322 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.323 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.323 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.324 DEBUG:boto:Params: {} 2022-09-13T07:39:37.324 DEBUG:boto:Token: None 2022-09-13T07:39:37.325 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj 2022-09-13T07:39:37.326 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI= 2022-09-13T07:39:37.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI=', 'Content-Length': '0'} 2022-09-13T07:39:37.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ecdd6303c1d29f4-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.331 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.331 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.332 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.333 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:37.333 DEBUG:boto:Data: 2022-09-13T07:39:37.334 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.334 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.335 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.335 DEBUG:boto:Params: {} 2022-09-13T07:39:37.336 DEBUG:boto:Token: None 2022-09-13T07:39:37.336 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj 2022-09-13T07:39:37.337 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI= 2022-09-13T07:39:37.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI=', 'Content-Length': '0'} 2022-09-13T07:39:37.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050fdb801be819040-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:37.342 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.343 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.343 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.344 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.344 DEBUG:boto:Method: GET 2022-09-13T07:39:37.345 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.345 DEBUG:boto:Data: 2022-09-13T07:39:37.346 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.346 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.347 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.347 DEBUG:boto:Params: {} 2022-09-13T07:39:37.348 DEBUG:boto:Token: None 2022-09-13T07:39:37.349 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.349 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM= 2022-09-13T07:39:37.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM=', 'Content-Length': '0'} 2022-09-13T07:39:37.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015758e0b5bab497c-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.398 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.399 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.399 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.400 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.400 DEBUG:boto:Method: GET 2022-09-13T07:39:37.401 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.401 DEBUG:boto:Data: 2022-09-13T07:39:37.402 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.402 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.403 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.403 DEBUG:boto:Params: {} 2022-09-13T07:39:37.404 DEBUG:boto:Token: None 2022-09-13T07:39:37.405 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.405 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM= 2022-09-13T07:39:37.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM=', 'Content-Length': '0'} 2022-09-13T07:39:37.408 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e00b11a46d3dbd59-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.409 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.410 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.410 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.411 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.411 DEBUG:boto:Method: GET 2022-09-13T07:39:37.412 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.413 DEBUG:boto:Data: 2022-09-13T07:39:37.413 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.414 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.415 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.415 DEBUG:boto:Params: {} 2022-09-13T07:39:37.416 DEBUG:boto:Token: None 2022-09-13T07:39:37.416 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.417 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb7b86ad87a317dc-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.421 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.422 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.423 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.423 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.424 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.424 DEBUG:boto:Method: GET 2022-09-13T07:39:37.425 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.425 DEBUG:boto:Data: 2022-09-13T07:39:37.426 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.426 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.427 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.427 DEBUG:boto:Params: {} 2022-09-13T07:39:37.428 DEBUG:boto:Token: None 2022-09-13T07:39:37.428 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.429 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000909b7eff5ef1d43a-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.433 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:37.435 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.435 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.436 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.436 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.437 DEBUG:boto:Method: GET 2022-09-13T07:39:37.438 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.438 DEBUG:boto:Data: 2022-09-13T07:39:37.439 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.439 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.440 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.440 DEBUG:boto:Params: {} 2022-09-13T07:39:37.441 DEBUG:boto:Token: None 2022-09-13T07:39:37.441 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.442 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QIEK0FgktC+1zZXYwATQYLEhX8Q= 2022-09-13T07:39:37.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QIEK0FgktC+1zZXYwATQYLEhX8Q=', 'Content-Length': '0'} 2022-09-13T07:39:37.445 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e837f52dc2c1101-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.445 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.446 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.447 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.447 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.448 DEBUG:boto:Method: GET 2022-09-13T07:39:37.448 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.449 DEBUG:boto:Data: 2022-09-13T07:39:37.449 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.450 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.450 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.451 DEBUG:boto:Params: {} 2022-09-13T07:39:37.451 DEBUG:boto:Token: None 2022-09-13T07:39:37.452 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.452 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QIEK0FgktC+1zZXYwATQYLEhX8Q= 2022-09-13T07:39:37.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QIEK0FgktC+1zZXYwATQYLEhX8Q=', 'Content-Length': '0'} 2022-09-13T07:39:37.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000982641450e77948d-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.456 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.457 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.457 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.458 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.458 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.459 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.459 DEBUG:boto:Data: 2022-09-13T07:39:37.460 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.460 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.461 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.461 DEBUG:boto:Params: {} 2022-09-13T07:39:37.462 DEBUG:boto:Token: None 2022-09-13T07:39:37.462 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.463 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zDFQ2iUFBAzpZrW69cvoZoffihI= 2022-09-13T07:39:37.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zDFQ2iUFBAzpZrW69cvoZoffihI=', 'Content-Length': '0'} 2022-09-13T07:39:37.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077e8bbf041742842-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.466 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.467 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.467 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.468 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.468 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.469 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.469 DEBUG:boto:Data: 2022-09-13T07:39:37.470 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.470 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.471 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.471 DEBUG:boto:Params: {} 2022-09-13T07:39:37.472 DEBUG:boto:Token: None 2022-09-13T07:39:37.473 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.474 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zDFQ2iUFBAzpZrW69cvoZoffihI= 2022-09-13T07:39:37.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zDFQ2iUFBAzpZrW69cvoZoffihI=', 'Content-Length': '0'} 2022-09-13T07:39:37.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4a0f4181372a3ff-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:37.479 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.479 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.480 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.480 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.481 DEBUG:boto:Method: GET 2022-09-13T07:39:37.481 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.482 DEBUG:boto:Data: 2022-09-13T07:39:37.482 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.483 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.483 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.484 DEBUG:boto:Params: {} 2022-09-13T07:39:37.485 DEBUG:boto:Token: None 2022-09-13T07:39:37.485 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.486 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QIEK0FgktC+1zZXYwATQYLEhX8Q= 2022-09-13T07:39:37.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QIEK0FgktC+1zZXYwATQYLEhX8Q=', 'Content-Length': '0'} 2022-09-13T07:39:37.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ade5684328ad2bed-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.489 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.490 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.490 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.491 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.492 DEBUG:boto:Method: GET 2022-09-13T07:39:37.492 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.493 DEBUG:boto:Data: 2022-09-13T07:39:37.493 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.494 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.494 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.495 DEBUG:boto:Params: {} 2022-09-13T07:39:37.495 DEBUG:boto:Token: None 2022-09-13T07:39:37.496 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:37.496 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QIEK0FgktC+1zZXYwATQYLEhX8Q= 2022-09-13T07:39:37.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QIEK0FgktC+1zZXYwATQYLEhX8Q=', 'Content-Length': '0'} 2022-09-13T07:39:37.499 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084015ff05c7fde51-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.500 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.500 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.501 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.501 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.502 DEBUG:boto:Method: GET 2022-09-13T07:39:37.502 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.503 DEBUG:boto:Data: 2022-09-13T07:39:37.503 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.504 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.504 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.505 DEBUG:boto:Params: {} 2022-09-13T07:39:37.506 DEBUG:boto:Token: None 2022-09-13T07:39:37.506 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.507 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee56f0e9e26d2840-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.511 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.512 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.513 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.513 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.514 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.514 DEBUG:boto:Method: GET 2022-09-13T07:39:37.515 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.515 DEBUG:boto:Data: 2022-09-13T07:39:37.516 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.516 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.517 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.518 DEBUG:boto:Params: {} 2022-09-13T07:39:37.519 DEBUG:boto:Token: None 2022-09-13T07:39:37.520 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.521 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf653a91a03e75b7-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.527 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.527 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.528 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.528 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.529 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:37.529 DEBUG:boto:Data: 2022-09-13T07:39:37.530 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.530 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.531 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.531 DEBUG:boto:Params: {} 2022-09-13T07:39:37.532 DEBUG:boto:Token: None 2022-09-13T07:39:37.532 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj 2022-09-13T07:39:37.533 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI= 2022-09-13T07:39:37.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI=', 'Content-Length': '0'} 2022-09-13T07:39:37.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005112415bc6ba5259-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.538 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.538 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.539 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.540 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:37.540 DEBUG:boto:Data: 2022-09-13T07:39:37.540 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.541 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.542 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.542 DEBUG:boto:Params: {} 2022-09-13T07:39:37.543 DEBUG:boto:Token: None 2022-09-13T07:39:37.544 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj 2022-09-13T07:39:37.545 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI= 2022-09-13T07:39:37.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI=', 'Content-Length': '0'} 2022-09-13T07:39:37.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003041a01387e2f4ee-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:37.550 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.551 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.551 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.552 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.552 DEBUG:boto:Method: GET 2022-09-13T07:39:37.553 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.553 DEBUG:boto:Data: 2022-09-13T07:39:37.554 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.555 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.555 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.556 DEBUG:boto:Params: {} 2022-09-13T07:39:37.556 DEBUG:boto:Token: None 2022-09-13T07:39:37.557 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.557 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM= 2022-09-13T07:39:37.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM=', 'Content-Length': '0'} 2022-09-13T07:39:37.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054c5ef20ca966017-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.561 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.562 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.562 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.563 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.563 DEBUG:boto:Method: GET 2022-09-13T07:39:37.564 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.564 DEBUG:boto:Data: 2022-09-13T07:39:37.565 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.566 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.566 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.567 DEBUG:boto:Params: {} 2022-09-13T07:39:37.567 DEBUG:boto:Token: None 2022-09-13T07:39:37.568 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.568 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM= 2022-09-13T07:39:37.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM=', 'Content-Length': '0'} 2022-09-13T07:39:37.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b21f77d39772042-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.618 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.619 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.620 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.623 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.624 DEBUG:boto:Method: GET 2022-09-13T07:39:37.625 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.626 DEBUG:boto:Data: 2022-09-13T07:39:37.626 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.627 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.629 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.630 DEBUG:boto:Params: {} 2022-09-13T07:39:37.631 DEBUG:boto:Token: None 2022-09-13T07:39:37.632 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.633 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000437f97bd4d88df76-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.639 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.640 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.640 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.641 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.643 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.644 DEBUG:boto:Method: GET 2022-09-13T07:39:37.645 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.646 DEBUG:boto:Data: 2022-09-13T07:39:37.647 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.648 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.649 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.650 DEBUG:boto:Params: {} 2022-09-13T07:39:37.651 DEBUG:boto:Token: None 2022-09-13T07:39:37.652 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.653 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048d211438b8fd94d-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.659 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:37.660 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.661 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.661 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.662 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.662 DEBUG:boto:Method: GET 2022-09-13T07:39:37.663 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.663 DEBUG:boto:Data: 2022-09-13T07:39:37.664 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.664 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.665 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.665 DEBUG:boto:Params: {} 2022-09-13T07:39:37.666 DEBUG:boto:Token: None 2022-09-13T07:39:37.667 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.667 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VgwOA9+Hr/BJm7vCyvOm3HLNqyc= 2022-09-13T07:39:37.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VgwOA9+Hr/BJm7vCyvOm3HLNqyc=', 'Content-Length': '0'} 2022-09-13T07:39:37.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba9eb841aa2dd5b1-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.710 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.711 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.712 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.713 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.714 DEBUG:boto:Method: GET 2022-09-13T07:39:37.715 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.716 DEBUG:boto:Data: 2022-09-13T07:39:37.717 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.718 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.719 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.719 DEBUG:boto:Params: {} 2022-09-13T07:39:37.720 DEBUG:boto:Token: None 2022-09-13T07:39:37.721 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.722 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VgwOA9+Hr/BJm7vCyvOm3HLNqyc= 2022-09-13T07:39:37.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VgwOA9+Hr/BJm7vCyvOm3HLNqyc=', 'Content-Length': '0'} 2022-09-13T07:39:37.726 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067023a646a880074-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.727 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.728 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.728 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.729 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.730 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.730 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.731 DEBUG:boto:Data: 2022-09-13T07:39:37.731 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.732 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.732 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.733 DEBUG:boto:Params: {} 2022-09-13T07:39:37.733 DEBUG:boto:Token: None 2022-09-13T07:39:37.734 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.735 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:L6wfOyBb3laz1UG/Dizh0Zg49ns= 2022-09-13T07:39:37.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:L6wfOyBb3laz1UG/Dizh0Zg49ns=', 'Content-Length': '0'} 2022-09-13T07:39:37.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0f5fe85885dac63-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.738 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.738 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.739 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.739 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.740 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.740 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.741 DEBUG:boto:Data: 2022-09-13T07:39:37.741 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.742 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.742 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.743 DEBUG:boto:Params: {} 2022-09-13T07:39:37.743 DEBUG:boto:Token: None 2022-09-13T07:39:37.744 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.744 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:L6wfOyBb3laz1UG/Dizh0Zg49ns= 2022-09-13T07:39:37.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:L6wfOyBb3laz1UG/Dizh0Zg49ns=', 'Content-Length': '0'} 2022-09-13T07:39:37.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b35772cd97129a0-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:37.748 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.749 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.749 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.750 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.750 DEBUG:boto:Method: GET 2022-09-13T07:39:37.751 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.751 DEBUG:boto:Data: 2022-09-13T07:39:37.752 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.753 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.753 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.754 DEBUG:boto:Params: {} 2022-09-13T07:39:37.754 DEBUG:boto:Token: None 2022-09-13T07:39:37.755 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.755 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VgwOA9+Hr/BJm7vCyvOm3HLNqyc= 2022-09-13T07:39:37.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VgwOA9+Hr/BJm7vCyvOm3HLNqyc=', 'Content-Length': '0'} 2022-09-13T07:39:37.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c394de61b6de724-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.760 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.761 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.761 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.762 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.763 DEBUG:boto:Method: GET 2022-09-13T07:39:37.764 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.764 DEBUG:boto:Data: 2022-09-13T07:39:37.765 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.766 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.767 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.768 DEBUG:boto:Params: {} 2022-09-13T07:39:37.768 DEBUG:boto:Token: None 2022-09-13T07:39:37.769 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:37.770 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VgwOA9+Hr/BJm7vCyvOm3HLNqyc= 2022-09-13T07:39:37.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VgwOA9+Hr/BJm7vCyvOm3HLNqyc=', 'Content-Length': '0'} 2022-09-13T07:39:37.773 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b21e3c5a75a2858d-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.774 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.775 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.775 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.776 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.776 DEBUG:boto:Method: GET 2022-09-13T07:39:37.777 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.777 DEBUG:boto:Data: 2022-09-13T07:39:37.778 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.779 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.779 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.780 DEBUG:boto:Params: {} 2022-09-13T07:39:37.780 DEBUG:boto:Token: None 2022-09-13T07:39:37.781 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.781 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003414ee3643f377f9-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.787 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.788 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.789 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.790 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.791 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.792 DEBUG:boto:Method: GET 2022-09-13T07:39:37.793 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.794 DEBUG:boto:Data: 2022-09-13T07:39:37.795 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.796 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.797 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.799 DEBUG:boto:Params: {} 2022-09-13T07:39:37.800 DEBUG:boto:Token: None 2022-09-13T07:39:37.801 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.802 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2cee65f0fb3f90f-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.808 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.810 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.811 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.812 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.812 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:37.813 DEBUG:boto:Data: 2022-09-13T07:39:37.814 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.815 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.816 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.817 DEBUG:boto:Params: {} 2022-09-13T07:39:37.818 DEBUG:boto:Token: None 2022-09-13T07:39:37.819 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj 2022-09-13T07:39:37.819 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI= 2022-09-13T07:39:37.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI=', 'Content-Length': '0'} 2022-09-13T07:39:37.823 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005de60e515dd53ef8-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.824 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.825 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.826 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.826 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:37.827 DEBUG:boto:Data: 2022-09-13T07:39:37.827 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.828 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.828 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.829 DEBUG:boto:Params: {} 2022-09-13T07:39:37.830 DEBUG:boto:Token: None 2022-09-13T07:39:37.830 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj 2022-09-13T07:39:37.831 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI= 2022-09-13T07:39:37.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wx6osNjD7zFGqrKhwQB6KohmItI=', 'Content-Length': '0'} 2022-09-13T07:39:37.835 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018f3f5780d21b7ee-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:37.837 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.837 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.838 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.839 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.839 DEBUG:boto:Method: GET 2022-09-13T07:39:37.840 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.840 DEBUG:boto:Data: 2022-09-13T07:39:37.841 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.841 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.842 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.842 DEBUG:boto:Params: {} 2022-09-13T07:39:37.843 DEBUG:boto:Token: None 2022-09-13T07:39:37.844 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.844 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM= 2022-09-13T07:39:37.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM=', 'Content-Length': '0'} 2022-09-13T07:39:37.847 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f2f1a0afdf17967-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.890 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.891 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.891 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.892 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.892 DEBUG:boto:Method: GET 2022-09-13T07:39:37.893 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.893 DEBUG:boto:Data: 2022-09-13T07:39:37.894 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.894 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.895 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.895 DEBUG:boto:Params: {} 2022-09-13T07:39:37.896 DEBUG:boto:Token: None 2022-09-13T07:39:37.897 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:37.898 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM= 2022-09-13T07:39:37.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7oDubMTRWjJT/+4p7/V73E46HoM=', 'Content-Length': '0'} 2022-09-13T07:39:37.900 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097ba6b8040e83bd2-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.901 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.902 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.902 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.903 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.903 DEBUG:boto:Method: GET 2022-09-13T07:39:37.904 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.904 DEBUG:boto:Data: 2022-09-13T07:39:37.905 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.905 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.906 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.906 DEBUG:boto:Params: {} 2022-09-13T07:39:37.907 DEBUG:boto:Token: None 2022-09-13T07:39:37.908 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.908 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4c20a800b7a5631-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.912 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.913 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:37.914 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:37.914 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.915 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.915 DEBUG:boto:Method: GET 2022-09-13T07:39:37.916 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:37.917 DEBUG:boto:Data: 2022-09-13T07:39:37.917 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.918 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.918 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.919 DEBUG:boto:Params: {} 2022-09-13T07:39:37.919 DEBUG:boto:Token: None 2022-09-13T07:39:37.920 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/ 2022-09-13T07:39:37.920 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ= 2022-09-13T07:39:37.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:T1NyWjcLjxsLyYSmMpE3ID+jdBQ=', 'Content-Length': '0'} 2022-09-13T07:39:37.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068a0e77194f3aedf-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.925 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:37.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:37.927 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.927 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.928 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.928 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.929 DEBUG:boto:Method: GET 2022-09-13T07:39:37.929 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.930 DEBUG:boto:Data: 2022-09-13T07:39:37.931 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.931 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.932 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.932 DEBUG:boto:Params: {} 2022-09-13T07:39:37.933 DEBUG:boto:Token: None 2022-09-13T07:39:37.933 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.934 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uNAcmwdrSVvJ4EwulxSkuht6GBM= 2022-09-13T07:39:37.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uNAcmwdrSVvJ4EwulxSkuht6GBM=', 'Content-Length': '0'} 2022-09-13T07:39:37.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3210dd1b1a3fe94-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.938 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.938 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.939 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.939 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.940 DEBUG:boto:Method: GET 2022-09-13T07:39:37.940 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.941 DEBUG:boto:Data: 2022-09-13T07:39:37.941 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.942 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.942 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.943 DEBUG:boto:Params: {} 2022-09-13T07:39:37.943 DEBUG:boto:Token: None 2022-09-13T07:39:37.944 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.945 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uNAcmwdrSVvJ4EwulxSkuht6GBM= 2022-09-13T07:39:37.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uNAcmwdrSVvJ4EwulxSkuht6GBM=', 'Content-Length': '0'} 2022-09-13T07:39:37.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002365192af71aef08-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.948 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.949 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.949 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.950 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.950 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.951 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.951 DEBUG:boto:Data: 2022-09-13T07:39:37.952 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.952 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.953 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.953 DEBUG:boto:Params: {} 2022-09-13T07:39:37.954 DEBUG:boto:Token: None 2022-09-13T07:39:37.954 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.955 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yFeAs6+RQaWXN0r3hn0znXvUJaA= 2022-09-13T07:39:37.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yFeAs6+RQaWXN0r3hn0znXvUJaA=', 'Content-Length': '0'} 2022-09-13T07:39:37.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba9a04a8c0eaa072-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.958 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.958 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.959 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.959 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.960 DEBUG:boto:Method: HEAD 2022-09-13T07:39:37.960 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.961 DEBUG:boto:Data: 2022-09-13T07:39:37.961 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.962 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.962 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.963 DEBUG:boto:Params: {} 2022-09-13T07:39:37.963 DEBUG:boto:Token: None 2022-09-13T07:39:37.964 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.965 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yFeAs6+RQaWXN0r3hn0znXvUJaA= 2022-09-13T07:39:37.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yFeAs6+RQaWXN0r3hn0znXvUJaA=', 'Content-Length': '0'} 2022-09-13T07:39:37.967 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d60d10448f381a6-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:37.969 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.969 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.970 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.971 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.972 DEBUG:boto:Method: GET 2022-09-13T07:39:37.973 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.974 DEBUG:boto:Data: 2022-09-13T07:39:37.974 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.975 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:37.976 DEBUG:boto:Port: 8001 2022-09-13T07:39:37.977 DEBUG:boto:Params: {} 2022-09-13T07:39:37.978 DEBUG:boto:Token: None 2022-09-13T07:39:37.978 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.979 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uNAcmwdrSVvJ4EwulxSkuht6GBM= 2022-09-13T07:39:37.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uNAcmwdrSVvJ4EwulxSkuht6GBM=', 'Content-Length': '0'} 2022-09-13T07:39:37.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075e975ae0cb5cf7c-00632033b9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:37.983 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:37.983 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:37.984 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.984 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.985 DEBUG:boto:Method: GET 2022-09-13T07:39:37.985 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.986 DEBUG:boto:Data: 2022-09-13T07:39:37.986 DEBUG:boto:Headers: {} 2022-09-13T07:39:37.987 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:37.987 DEBUG:boto:Port: 8000 2022-09-13T07:39:37.988 DEBUG:boto:Params: {} 2022-09-13T07:39:37.988 DEBUG:boto:Token: None 2022-09-13T07:39:37.989 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:37 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:37.989 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uNAcmwdrSVvJ4EwulxSkuht6GBM= 2022-09-13T07:39:37.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:37 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uNAcmwdrSVvJ4EwulxSkuht6GBM=', 'Content-Length': '0'} 2022-09-13T07:39:37.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000809a968b47baf4a2-00632033b9-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.013 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.014 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.014 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.015 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.015 DEBUG:boto:Method: GET 2022-09-13T07:39:38.016 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.017 DEBUG:boto:Data: 2022-09-13T07:39:38.017 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.018 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.018 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.019 DEBUG:boto:Params: {} 2022-09-13T07:39:38.020 DEBUG:boto:Token: None 2022-09-13T07:39:38.020 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.021 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009912634828213503-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.026 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.027 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.028 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.029 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.030 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.031 DEBUG:boto:Method: GET 2022-09-13T07:39:38.032 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.032 DEBUG:boto:Data: 2022-09-13T07:39:38.034 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.035 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.036 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.036 DEBUG:boto:Params: {} 2022-09-13T07:39:38.037 DEBUG:boto:Token: None 2022-09-13T07:39:38.038 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.039 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009acf2ebc44d06253-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.044 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.045 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.046 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.046 DEBUG:boto:Method: HEAD 2022-09-13T07:39:38.047 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:38.047 DEBUG:boto:Data: 2022-09-13T07:39:38.048 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.049 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.049 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.050 DEBUG:boto:Params: {} 2022-09-13T07:39:38.050 DEBUG:boto:Token: None 2022-09-13T07:39:38.051 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj 2022-09-13T07:39:38.052 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs= 2022-09-13T07:39:38.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs=', 'Content-Length': '0'} 2022-09-13T07:39:38.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017ffd7505b22c909-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.056 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.057 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.057 DEBUG:boto:Method: HEAD 2022-09-13T07:39:38.058 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:38.059 DEBUG:boto:Data: 2022-09-13T07:39:38.059 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.060 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.060 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.061 DEBUG:boto:Params: {} 2022-09-13T07:39:38.061 DEBUG:boto:Token: None 2022-09-13T07:39:38.062 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj 2022-09-13T07:39:38.062 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs= 2022-09-13T07:39:38.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs=', 'Content-Length': '0'} 2022-09-13T07:39:38.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e799ca06a4db864-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:38.067 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.068 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.069 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.069 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.070 DEBUG:boto:Method: GET 2022-09-13T07:39:38.070 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.071 DEBUG:boto:Data: 2022-09-13T07:39:38.072 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.072 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.073 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.073 DEBUG:boto:Params: {} 2022-09-13T07:39:38.074 DEBUG:boto:Token: None 2022-09-13T07:39:38.075 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.076 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE= 2022-09-13T07:39:38.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE=', 'Content-Length': '0'} 2022-09-13T07:39:38.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6e2d0a86e5952fd-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.081 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.082 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.083 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.084 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.085 DEBUG:boto:Method: GET 2022-09-13T07:39:38.085 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.086 DEBUG:boto:Data: 2022-09-13T07:39:38.087 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.088 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.089 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.089 DEBUG:boto:Params: {} 2022-09-13T07:39:38.090 DEBUG:boto:Token: None 2022-09-13T07:39:38.091 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.091 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE= 2022-09-13T07:39:38.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE=', 'Content-Length': '0'} 2022-09-13T07:39:38.094 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007deacf1a0ddc3690-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.142 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.143 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.144 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.145 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.146 DEBUG:boto:Method: GET 2022-09-13T07:39:38.146 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.147 DEBUG:boto:Data: 2022-09-13T07:39:38.148 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.149 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.150 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.151 DEBUG:boto:Params: {} 2022-09-13T07:39:38.152 DEBUG:boto:Token: None 2022-09-13T07:39:38.153 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.154 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb9910d32dde3751-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.158 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.159 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.160 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.161 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.161 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.162 DEBUG:boto:Method: GET 2022-09-13T07:39:38.162 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.163 DEBUG:boto:Data: 2022-09-13T07:39:38.163 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.164 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.165 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.165 DEBUG:boto:Params: {} 2022-09-13T07:39:38.166 DEBUG:boto:Token: None 2022-09-13T07:39:38.167 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.168 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cab57c7af72c4cf8-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.174 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:38.175 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.176 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.176 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.177 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.177 DEBUG:boto:Method: GET 2022-09-13T07:39:38.178 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.178 DEBUG:boto:Data: 2022-09-13T07:39:38.179 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.179 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.180 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.180 DEBUG:boto:Params: {} 2022-09-13T07:39:38.181 DEBUG:boto:Token: None 2022-09-13T07:39:38.182 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.182 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HNSlVqixvZf3B4c85FLKlUiTZhE= 2022-09-13T07:39:38.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HNSlVqixvZf3B4c85FLKlUiTZhE=', 'Content-Length': '0'} 2022-09-13T07:39:38.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d7fcbdd0c43b6d9-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.186 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.187 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.187 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.188 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.188 DEBUG:boto:Method: GET 2022-09-13T07:39:38.189 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.189 DEBUG:boto:Data: 2022-09-13T07:39:38.190 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.190 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.191 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.192 DEBUG:boto:Params: {} 2022-09-13T07:39:38.192 DEBUG:boto:Token: None 2022-09-13T07:39:38.193 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.193 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HNSlVqixvZf3B4c85FLKlUiTZhE= 2022-09-13T07:39:38.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HNSlVqixvZf3B4c85FLKlUiTZhE=', 'Content-Length': '0'} 2022-09-13T07:39:38.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a975a6a8304cdf92-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.197 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.198 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.199 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.200 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.201 DEBUG:boto:Method: HEAD 2022-09-13T07:39:38.201 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.202 DEBUG:boto:Data: 2022-09-13T07:39:38.203 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.204 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.204 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.205 DEBUG:boto:Params: {} 2022-09-13T07:39:38.206 DEBUG:boto:Token: None 2022-09-13T07:39:38.207 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.208 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PJEUs5Ajq5Zcvbvaz/WE9WP01Uk= 2022-09-13T07:39:38.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PJEUs5Ajq5Zcvbvaz/WE9WP01Uk=', 'Content-Length': '0'} 2022-09-13T07:39:38.212 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e17a3a0cb90ce687-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.213 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.214 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.215 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.215 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.215 DEBUG:boto:Method: HEAD 2022-09-13T07:39:38.217 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.217 DEBUG:boto:Data: 2022-09-13T07:39:38.218 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.219 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.220 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.221 DEBUG:boto:Params: {} 2022-09-13T07:39:38.222 DEBUG:boto:Token: None 2022-09-13T07:39:38.222 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.223 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PJEUs5Ajq5Zcvbvaz/WE9WP01Uk= 2022-09-13T07:39:38.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PJEUs5Ajq5Zcvbvaz/WE9WP01Uk=', 'Content-Length': '0'} 2022-09-13T07:39:38.226 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9a0da3bb82f3910-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:38.228 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.228 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.229 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.229 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.230 DEBUG:boto:Method: GET 2022-09-13T07:39:38.230 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.231 DEBUG:boto:Data: 2022-09-13T07:39:38.231 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.232 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.232 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.233 DEBUG:boto:Params: {} 2022-09-13T07:39:38.234 DEBUG:boto:Token: None 2022-09-13T07:39:38.234 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.235 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HNSlVqixvZf3B4c85FLKlUiTZhE= 2022-09-13T07:39:38.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HNSlVqixvZf3B4c85FLKlUiTZhE=', 'Content-Length': '0'} 2022-09-13T07:39:38.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052dc1f2d9796a73d-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.238 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.239 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.239 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.240 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.241 DEBUG:boto:Method: GET 2022-09-13T07:39:38.241 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.242 DEBUG:boto:Data: 2022-09-13T07:39:38.243 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.243 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.244 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.244 DEBUG:boto:Params: {} 2022-09-13T07:39:38.245 DEBUG:boto:Token: None 2022-09-13T07:39:38.246 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:38.246 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HNSlVqixvZf3B4c85FLKlUiTZhE= 2022-09-13T07:39:38.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HNSlVqixvZf3B4c85FLKlUiTZhE=', 'Content-Length': '0'} 2022-09-13T07:39:38.251 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000beea9a573cce34d0-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.252 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.252 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.253 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.253 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.254 DEBUG:boto:Method: GET 2022-09-13T07:39:38.255 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.255 DEBUG:boto:Data: 2022-09-13T07:39:38.256 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.256 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.257 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.257 DEBUG:boto:Params: {} 2022-09-13T07:39:38.258 DEBUG:boto:Token: None 2022-09-13T07:39:38.259 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.259 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000826f9f75d2df89cc-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.264 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.265 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.265 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.266 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.266 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.267 DEBUG:boto:Method: GET 2022-09-13T07:39:38.267 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.268 DEBUG:boto:Data: 2022-09-13T07:39:38.268 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.269 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.269 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.270 DEBUG:boto:Params: {} 2022-09-13T07:39:38.271 DEBUG:boto:Token: None 2022-09-13T07:39:38.271 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.272 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c63f5115b4d2a54-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.276 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.277 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.278 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.279 DEBUG:boto:Method: HEAD 2022-09-13T07:39:38.279 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:38.280 DEBUG:boto:Data: 2022-09-13T07:39:38.280 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.281 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.281 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.282 DEBUG:boto:Params: {} 2022-09-13T07:39:38.282 DEBUG:boto:Token: None 2022-09-13T07:39:38.283 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj 2022-09-13T07:39:38.283 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs= 2022-09-13T07:39:38.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs=', 'Content-Length': '0'} 2022-09-13T07:39:38.287 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028da1712e8da11b1-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.288 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.289 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.289 DEBUG:boto:Method: HEAD 2022-09-13T07:39:38.290 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:38.290 DEBUG:boto:Data: 2022-09-13T07:39:38.291 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.291 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.292 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.292 DEBUG:boto:Params: {} 2022-09-13T07:39:38.293 DEBUG:boto:Token: None 2022-09-13T07:39:38.293 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj 2022-09-13T07:39:38.294 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs= 2022-09-13T07:39:38.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs=', 'Content-Length': '0'} 2022-09-13T07:39:38.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067d18e983e8cb2ab-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:38.299 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.300 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.300 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.301 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.301 DEBUG:boto:Method: GET 2022-09-13T07:39:38.302 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.302 DEBUG:boto:Data: 2022-09-13T07:39:38.303 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.303 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.304 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.304 DEBUG:boto:Params: {} 2022-09-13T07:39:38.305 DEBUG:boto:Token: None 2022-09-13T07:39:38.305 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.306 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE= 2022-09-13T07:39:38.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE=', 'Content-Length': '0'} 2022-09-13T07:39:38.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001164d3a79f743868-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.350 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.351 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.351 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.352 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.353 DEBUG:boto:Method: GET 2022-09-13T07:39:38.353 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.354 DEBUG:boto:Data: 2022-09-13T07:39:38.354 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.355 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.355 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.356 DEBUG:boto:Params: {} 2022-09-13T07:39:38.357 DEBUG:boto:Token: None 2022-09-13T07:39:38.357 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.358 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE= 2022-09-13T07:39:38.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE=', 'Content-Length': '0'} 2022-09-13T07:39:38.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005503165e2757eef7-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.362 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.363 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.363 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.364 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.364 DEBUG:boto:Method: GET 2022-09-13T07:39:38.365 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.365 DEBUG:boto:Data: 2022-09-13T07:39:38.366 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.366 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.367 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.367 DEBUG:boto:Params: {} 2022-09-13T07:39:38.368 DEBUG:boto:Token: None 2022-09-13T07:39:38.368 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.369 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000059f1e898c9f77a9-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.374 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.375 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.375 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.376 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.377 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.378 DEBUG:boto:Method: GET 2022-09-13T07:39:38.379 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.379 DEBUG:boto:Data: 2022-09-13T07:39:38.380 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.381 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.382 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.383 DEBUG:boto:Params: {} 2022-09-13T07:39:38.384 DEBUG:boto:Token: None 2022-09-13T07:39:38.385 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.386 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c26af5b66eaa8084-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.391 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:38.393 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.394 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.394 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.395 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.395 DEBUG:boto:Method: GET 2022-09-13T07:39:38.396 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.396 DEBUG:boto:Data: 2022-09-13T07:39:38.397 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.397 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.398 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.399 DEBUG:boto:Params: {} 2022-09-13T07:39:38.399 DEBUG:boto:Token: None 2022-09-13T07:39:38.400 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.401 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:W7oDeuLhv7S6PlhdRrxjKGmE/cs= 2022-09-13T07:39:38.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:W7oDeuLhv7S6PlhdRrxjKGmE/cs=', 'Content-Length': '0'} 2022-09-13T07:39:38.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000415ddf556e6c5736-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.405 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.405 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.406 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.406 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.407 DEBUG:boto:Method: GET 2022-09-13T07:39:38.407 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.408 DEBUG:boto:Data: 2022-09-13T07:39:38.408 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.409 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.409 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.410 DEBUG:boto:Params: {} 2022-09-13T07:39:38.410 DEBUG:boto:Token: None 2022-09-13T07:39:38.411 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.412 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:W7oDeuLhv7S6PlhdRrxjKGmE/cs= 2022-09-13T07:39:38.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:W7oDeuLhv7S6PlhdRrxjKGmE/cs=', 'Content-Length': '0'} 2022-09-13T07:39:38.415 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d539515a77363edc-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.458 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.459 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.460 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.461 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.462 DEBUG:boto:Method: HEAD 2022-09-13T07:39:38.462 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.463 DEBUG:boto:Data: 2022-09-13T07:39:38.464 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.465 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.466 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.466 DEBUG:boto:Params: {} 2022-09-13T07:39:38.468 DEBUG:boto:Token: None 2022-09-13T07:39:38.469 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.469 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YYGkqEWDyXj0jF5lkuwjohWQMME= 2022-09-13T07:39:38.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YYGkqEWDyXj0jF5lkuwjohWQMME=', 'Content-Length': '0'} 2022-09-13T07:39:38.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c95f90d31cbf3509-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.473 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.474 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.474 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.475 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.475 DEBUG:boto:Method: HEAD 2022-09-13T07:39:38.476 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.477 DEBUG:boto:Data: 2022-09-13T07:39:38.478 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.479 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.480 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.481 DEBUG:boto:Params: {} 2022-09-13T07:39:38.482 DEBUG:boto:Token: None 2022-09-13T07:39:38.483 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.484 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YYGkqEWDyXj0jF5lkuwjohWQMME= 2022-09-13T07:39:38.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YYGkqEWDyXj0jF5lkuwjohWQMME=', 'Content-Length': '0'} 2022-09-13T07:39:38.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007699b10d6e08d84c-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:38.490 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.491 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.492 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.493 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.494 DEBUG:boto:Method: GET 2022-09-13T07:39:38.495 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.496 DEBUG:boto:Data: 2022-09-13T07:39:38.496 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.497 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.498 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.499 DEBUG:boto:Params: {} 2022-09-13T07:39:38.500 DEBUG:boto:Token: None 2022-09-13T07:39:38.501 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.502 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:W7oDeuLhv7S6PlhdRrxjKGmE/cs= 2022-09-13T07:39:38.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:W7oDeuLhv7S6PlhdRrxjKGmE/cs=', 'Content-Length': '0'} 2022-09-13T07:39:38.506 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5fcd5c9dc7fa6c9-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.508 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.508 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.509 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.510 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.511 DEBUG:boto:Method: GET 2022-09-13T07:39:38.512 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.513 DEBUG:boto:Data: 2022-09-13T07:39:38.514 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.515 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.516 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.517 DEBUG:boto:Params: {} 2022-09-13T07:39:38.518 DEBUG:boto:Token: None 2022-09-13T07:39:38.519 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:38.520 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:W7oDeuLhv7S6PlhdRrxjKGmE/cs= 2022-09-13T07:39:38.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:W7oDeuLhv7S6PlhdRrxjKGmE/cs=', 'Content-Length': '0'} 2022-09-13T07:39:38.524 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014f8194084a3a54d-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.525 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.526 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.527 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.528 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.529 DEBUG:boto:Method: GET 2022-09-13T07:39:38.529 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.530 DEBUG:boto:Data: 2022-09-13T07:39:38.531 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.532 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.532 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.533 DEBUG:boto:Params: {} 2022-09-13T07:39:38.533 DEBUG:boto:Token: None 2022-09-13T07:39:38.534 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.534 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000276f924c7e1e1b5c-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.539 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.541 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.541 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.542 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.543 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.543 DEBUG:boto:Method: GET 2022-09-13T07:39:38.544 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.552 DEBUG:boto:Data: 2022-09-13T07:39:38.553 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.553 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.554 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.554 DEBUG:boto:Params: {} 2022-09-13T07:39:38.555 DEBUG:boto:Token: None 2022-09-13T07:39:38.556 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.557 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000602b1fa9ddac8e61-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.564 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.565 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.566 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.568 DEBUG:boto:Method: HEAD 2022-09-13T07:39:38.568 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:38.569 DEBUG:boto:Data: 2022-09-13T07:39:38.570 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.571 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.572 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.573 DEBUG:boto:Params: {} 2022-09-13T07:39:38.574 DEBUG:boto:Token: None 2022-09-13T07:39:38.575 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj 2022-09-13T07:39:38.576 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs= 2022-09-13T07:39:38.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs=', 'Content-Length': '0'} 2022-09-13T07:39:38.581 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f6c2d551079541a-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.582 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.583 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.584 DEBUG:boto:Method: HEAD 2022-09-13T07:39:38.585 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:38.586 DEBUG:boto:Data: 2022-09-13T07:39:38.587 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.587 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.588 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.589 DEBUG:boto:Params: {} 2022-09-13T07:39:38.590 DEBUG:boto:Token: None 2022-09-13T07:39:38.591 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj 2022-09-13T07:39:38.592 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs= 2022-09-13T07:39:38.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs=', 'Content-Length': '0'} 2022-09-13T07:39:38.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ffacfff76f136fd-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:38.599 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.602 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.603 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.604 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.605 DEBUG:boto:Method: GET 2022-09-13T07:39:38.605 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.606 DEBUG:boto:Data: 2022-09-13T07:39:38.607 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.608 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.609 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.610 DEBUG:boto:Params: {} 2022-09-13T07:39:38.611 DEBUG:boto:Token: None 2022-09-13T07:39:38.612 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.613 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE= 2022-09-13T07:39:38.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE=', 'Content-Length': '0'} 2022-09-13T07:39:38.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073a88eaa2fbbfa04-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.619 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.620 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.620 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.621 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.622 DEBUG:boto:Method: GET 2022-09-13T07:39:38.623 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.624 DEBUG:boto:Data: 2022-09-13T07:39:38.625 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.626 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.627 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.628 DEBUG:boto:Params: {} 2022-09-13T07:39:38.628 DEBUG:boto:Token: None 2022-09-13T07:39:38.629 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.629 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE= 2022-09-13T07:39:38.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE=', 'Content-Length': '0'} 2022-09-13T07:39:38.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004eb697fe309040c1-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.634 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.635 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.635 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.636 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.637 DEBUG:boto:Method: GET 2022-09-13T07:39:38.637 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.638 DEBUG:boto:Data: 2022-09-13T07:39:38.639 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.640 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.641 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.642 DEBUG:boto:Params: {} 2022-09-13T07:39:38.642 DEBUG:boto:Token: None 2022-09-13T07:39:38.643 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.643 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d45fd9da703374a-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.648 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.649 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.650 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.650 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.651 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.652 DEBUG:boto:Method: GET 2022-09-13T07:39:38.652 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.653 DEBUG:boto:Data: 2022-09-13T07:39:38.654 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.655 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.656 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.657 DEBUG:boto:Params: {} 2022-09-13T07:39:38.658 DEBUG:boto:Token: None 2022-09-13T07:39:38.659 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.660 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af75c0897992c569-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.664 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:38.666 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.667 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.667 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.668 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.669 DEBUG:boto:Method: GET 2022-09-13T07:39:38.670 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.671 DEBUG:boto:Data: 2022-09-13T07:39:38.672 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.673 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.673 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.674 DEBUG:boto:Params: {} 2022-09-13T07:39:38.675 DEBUG:boto:Token: None 2022-09-13T07:39:38.676 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.677 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8LjI+gMI/9g4i8r+XDYqFeMX96I= 2022-09-13T07:39:38.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8LjI+gMI/9g4i8r+XDYqFeMX96I=', 'Content-Length': '0'} 2022-09-13T07:39:38.681 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057cda2031b59be86-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.726 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.727 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.727 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.728 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.728 DEBUG:boto:Method: GET 2022-09-13T07:39:38.729 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.729 DEBUG:boto:Data: 2022-09-13T07:39:38.730 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.731 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.731 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.732 DEBUG:boto:Params: {} 2022-09-13T07:39:38.732 DEBUG:boto:Token: None 2022-09-13T07:39:38.733 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.733 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8LjI+gMI/9g4i8r+XDYqFeMX96I= 2022-09-13T07:39:38.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8LjI+gMI/9g4i8r+XDYqFeMX96I=', 'Content-Length': '0'} 2022-09-13T07:39:38.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003994ff2363a1a91f-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.737 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.738 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.739 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.740 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.741 DEBUG:boto:Method: HEAD 2022-09-13T07:39:38.742 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.743 DEBUG:boto:Data: 2022-09-13T07:39:38.743 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.744 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.745 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.746 DEBUG:boto:Params: {} 2022-09-13T07:39:38.747 DEBUG:boto:Token: None 2022-09-13T07:39:38.748 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.748 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4cB47j7QinNi34lVdMwNiNPwHzo= 2022-09-13T07:39:38.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4cB47j7QinNi34lVdMwNiNPwHzo=', 'Content-Length': '0'} 2022-09-13T07:39:38.751 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d5098175721b5aa-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.752 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.752 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.753 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.753 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.754 DEBUG:boto:Method: HEAD 2022-09-13T07:39:38.754 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.755 DEBUG:boto:Data: 2022-09-13T07:39:38.755 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.756 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.756 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.757 DEBUG:boto:Params: {} 2022-09-13T07:39:38.758 DEBUG:boto:Token: None 2022-09-13T07:39:38.759 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.760 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4cB47j7QinNi34lVdMwNiNPwHzo= 2022-09-13T07:39:38.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4cB47j7QinNi34lVdMwNiNPwHzo=', 'Content-Length': '0'} 2022-09-13T07:39:38.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abd7956afdc9af62-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:38.765 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.766 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.766 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.767 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.768 DEBUG:boto:Method: GET 2022-09-13T07:39:38.769 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.769 DEBUG:boto:Data: 2022-09-13T07:39:38.770 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.771 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.772 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.772 DEBUG:boto:Params: {} 2022-09-13T07:39:38.773 DEBUG:boto:Token: None 2022-09-13T07:39:38.773 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.774 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8LjI+gMI/9g4i8r+XDYqFeMX96I= 2022-09-13T07:39:38.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8LjI+gMI/9g4i8r+XDYqFeMX96I=', 'Content-Length': '0'} 2022-09-13T07:39:38.778 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005c614c407e2943a-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.779 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.780 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.781 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.782 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.783 DEBUG:boto:Method: GET 2022-09-13T07:39:38.784 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.784 DEBUG:boto:Data: 2022-09-13T07:39:38.785 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.786 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.787 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.787 DEBUG:boto:Params: {} 2022-09-13T07:39:38.788 DEBUG:boto:Token: None 2022-09-13T07:39:38.789 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:38.789 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8LjI+gMI/9g4i8r+XDYqFeMX96I= 2022-09-13T07:39:38.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8LjI+gMI/9g4i8r+XDYqFeMX96I=', 'Content-Length': '0'} 2022-09-13T07:39:38.792 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ad15b9182609bc6-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.794 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.794 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.795 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.795 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.796 DEBUG:boto:Method: GET 2022-09-13T07:39:38.797 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.797 DEBUG:boto:Data: 2022-09-13T07:39:38.798 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.798 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.799 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.799 DEBUG:boto:Params: {} 2022-09-13T07:39:38.800 DEBUG:boto:Token: None 2022-09-13T07:39:38.800 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.801 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a90f1bb1198c4ec3-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.846 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.848 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.849 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.849 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.851 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.852 DEBUG:boto:Method: GET 2022-09-13T07:39:38.852 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.853 DEBUG:boto:Data: 2022-09-13T07:39:38.854 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.855 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.856 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.857 DEBUG:boto:Params: {} 2022-09-13T07:39:38.858 DEBUG:boto:Token: None 2022-09-13T07:39:38.859 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.860 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076a9961914d5f02f-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.867 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.868 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.869 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.870 DEBUG:boto:Method: HEAD 2022-09-13T07:39:38.871 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:38.872 DEBUG:boto:Data: 2022-09-13T07:39:38.873 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.874 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.875 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.876 DEBUG:boto:Params: {} 2022-09-13T07:39:38.877 DEBUG:boto:Token: None 2022-09-13T07:39:38.878 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj 2022-09-13T07:39:38.879 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs= 2022-09-13T07:39:38.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs=', 'Content-Length': '0'} 2022-09-13T07:39:38.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068dbeed85acd463c-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.886 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.887 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.888 DEBUG:boto:Method: HEAD 2022-09-13T07:39:38.889 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:38.889 DEBUG:boto:Data: 2022-09-13T07:39:38.890 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.891 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.892 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.893 DEBUG:boto:Params: {} 2022-09-13T07:39:38.894 DEBUG:boto:Token: None 2022-09-13T07:39:38.896 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj 2022-09-13T07:39:38.897 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs= 2022-09-13T07:39:38.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:swdf7gVIIRet4DmCY60u/QxNyBs=', 'Content-Length': '0'} 2022-09-13T07:39:38.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9618eb5e23d1d8a-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:38.903 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.903 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.904 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.904 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.905 DEBUG:boto:Method: GET 2022-09-13T07:39:38.905 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.906 DEBUG:boto:Data: 2022-09-13T07:39:38.906 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.907 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.907 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.908 DEBUG:boto:Params: {} 2022-09-13T07:39:38.908 DEBUG:boto:Token: None 2022-09-13T07:39:38.909 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.910 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE= 2022-09-13T07:39:38.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE=', 'Content-Length': '0'} 2022-09-13T07:39:38.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074f3a3217ae90c63-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.913 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.914 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.914 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.915 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.915 DEBUG:boto:Method: GET 2022-09-13T07:39:38.916 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.916 DEBUG:boto:Data: 2022-09-13T07:39:38.917 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.917 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.918 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.919 DEBUG:boto:Params: {} 2022-09-13T07:39:38.919 DEBUG:boto:Token: None 2022-09-13T07:39:38.920 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:38.920 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE= 2022-09-13T07:39:38.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3ScfVHWUB7XwdmXALopGdfKPJeE=', 'Content-Length': '0'} 2022-09-13T07:39:38.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000caaf11162c5c92bf-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.926 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.927 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.927 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.928 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.929 DEBUG:boto:Method: GET 2022-09-13T07:39:38.929 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.930 DEBUG:boto:Data: 2022-09-13T07:39:38.930 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.931 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.931 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.932 DEBUG:boto:Params: {} 2022-09-13T07:39:38.933 DEBUG:boto:Token: None 2022-09-13T07:39:38.934 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.934 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9020a982606369b-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.939 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.939 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:38.940 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:38.940 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.941 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.941 DEBUG:boto:Method: GET 2022-09-13T07:39:38.942 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:38.943 DEBUG:boto:Data: 2022-09-13T07:39:38.943 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.944 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:38.944 DEBUG:boto:Port: 8000 2022-09-13T07:39:38.945 DEBUG:boto:Params: {} 2022-09-13T07:39:38.945 DEBUG:boto:Token: None 2022-09-13T07:39:38.946 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/ 2022-09-13T07:39:38.947 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg= 2022-09-13T07:39:38.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VvMlmPls6trlydNIfj7bqTV2Elg=', 'Content-Length': '0'} 2022-09-13T07:39:38.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f38008b0e778e211-00632033ba-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:38.952 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:38.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:38.953 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:38.954 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:38.955 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:38.955 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:38.956 DEBUG:boto:Method: GET 2022-09-13T07:39:38.956 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:38.957 DEBUG:boto:Data: 2022-09-13T07:39:38.957 DEBUG:boto:Headers: {} 2022-09-13T07:39:38.958 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:38.958 DEBUG:boto:Port: 8001 2022-09-13T07:39:38.959 DEBUG:boto:Params: {} 2022-09-13T07:39:38.959 DEBUG:boto:Token: None 2022-09-13T07:39:38.960 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:38 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:38.960 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tei22h9veRpd2Mj7ByqEJvoukSg= 2022-09-13T07:39:38.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:38 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tei22h9veRpd2Mj7ByqEJvoukSg=', 'Content-Length': '0'} 2022-09-13T07:39:38.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edc350f2ace5439d-00632033ba-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.010 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.011 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.011 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.012 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.013 DEBUG:boto:Method: GET 2022-09-13T07:39:39.013 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.014 DEBUG:boto:Data: 2022-09-13T07:39:39.014 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.015 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.015 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.016 DEBUG:boto:Params: {} 2022-09-13T07:39:39.017 DEBUG:boto:Token: None 2022-09-13T07:39:39.017 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.018 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DhcE+KrWWSM35q8Z+x6lVIIjoAk= 2022-09-13T07:39:39.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DhcE+KrWWSM35q8Z+x6lVIIjoAk=', 'Content-Length': '0'} 2022-09-13T07:39:39.021 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027409378e4223789-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.022 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.023 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.023 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.024 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.024 DEBUG:boto:Method: HEAD 2022-09-13T07:39:39.025 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.025 DEBUG:boto:Data: 2022-09-13T07:39:39.026 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.027 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.027 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.028 DEBUG:boto:Params: {} 2022-09-13T07:39:39.028 DEBUG:boto:Token: None 2022-09-13T07:39:39.029 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.030 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3jkQihoflelCZJdEf9wwN7QCWY8= 2022-09-13T07:39:39.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3jkQihoflelCZJdEf9wwN7QCWY8=', 'Content-Length': '0'} 2022-09-13T07:39:39.033 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b81819dc8c361942-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.033 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.034 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.034 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.035 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.036 DEBUG:boto:Method: HEAD 2022-09-13T07:39:39.036 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.037 DEBUG:boto:Data: 2022-09-13T07:39:39.038 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.038 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.039 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.040 DEBUG:boto:Params: {} 2022-09-13T07:39:39.040 DEBUG:boto:Token: None 2022-09-13T07:39:39.041 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.042 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3jkQihoflelCZJdEf9wwN7QCWY8= 2022-09-13T07:39:39.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3jkQihoflelCZJdEf9wwN7QCWY8=', 'Content-Length': '0'} 2022-09-13T07:39:39.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a49a36ccfea17449-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:39.046 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.046 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.047 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.047 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.048 DEBUG:boto:Method: GET 2022-09-13T07:39:39.048 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.049 DEBUG:boto:Data: 2022-09-13T07:39:39.049 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.050 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.051 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.051 DEBUG:boto:Params: {} 2022-09-13T07:39:39.052 DEBUG:boto:Token: None 2022-09-13T07:39:39.052 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.053 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DhcE+KrWWSM35q8Z+x6lVIIjoAk= 2022-09-13T07:39:39.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DhcE+KrWWSM35q8Z+x6lVIIjoAk=', 'Content-Length': '0'} 2022-09-13T07:39:39.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6fe9f0bb100ecf5-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.057 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.058 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.059 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.059 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.060 DEBUG:boto:Method: GET 2022-09-13T07:39:39.061 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.061 DEBUG:boto:Data: 2022-09-13T07:39:39.062 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.062 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.063 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.063 DEBUG:boto:Params: {} 2022-09-13T07:39:39.064 DEBUG:boto:Token: None 2022-09-13T07:39:39.065 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:39.066 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DhcE+KrWWSM35q8Z+x6lVIIjoAk= 2022-09-13T07:39:39.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DhcE+KrWWSM35q8Z+x6lVIIjoAk=', 'Content-Length': '0'} 2022-09-13T07:39:39.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b1af7c8029744da-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.074 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:39.075 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:39.075 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.076 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.077 DEBUG:boto:Method: GET 2022-09-13T07:39:39.078 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.079 DEBUG:boto:Data: 2022-09-13T07:39:39.079 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.080 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.081 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.082 DEBUG:boto:Params: {} 2022-09-13T07:39:39.083 DEBUG:boto:Token: None 2022-09-13T07:39:39.084 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/ 2022-09-13T07:39:39.085 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM= 2022-09-13T07:39:39.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM=', 'Content-Length': '0'} 2022-09-13T07:39:39.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc0b2ac4921ffa05-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.090 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:39.091 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:39.091 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:39.092 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.092 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.093 DEBUG:boto:Method: GET 2022-09-13T07:39:39.093 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.094 DEBUG:boto:Data: 2022-09-13T07:39:39.095 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.096 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.097 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.098 DEBUG:boto:Params: {} 2022-09-13T07:39:39.099 DEBUG:boto:Token: None 2022-09-13T07:39:39.100 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/ 2022-09-13T07:39:39.101 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM= 2022-09-13T07:39:39.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM=', 'Content-Length': '0'} 2022-09-13T07:39:39.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000581e3348ecfa8f15-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.154 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:39.155 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.156 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.156 DEBUG:boto:Method: HEAD 2022-09-13T07:39:39.157 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:39.157 DEBUG:boto:Data: 2022-09-13T07:39:39.158 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.158 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.159 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.159 DEBUG:boto:Params: {} 2022-09-13T07:39:39.160 DEBUG:boto:Token: None 2022-09-13T07:39:39.160 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj 2022-09-13T07:39:39.161 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yNRgBgvM/Dazb60sLtPQzs9zxAc= 2022-09-13T07:39:39.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yNRgBgvM/Dazb60sLtPQzs9zxAc=', 'Content-Length': '0'} 2022-09-13T07:39:39.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f8fc3747093e1ab-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.166 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.167 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.167 DEBUG:boto:Method: HEAD 2022-09-13T07:39:39.168 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:39.168 DEBUG:boto:Data: 2022-09-13T07:39:39.169 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.169 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.170 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.170 DEBUG:boto:Params: {} 2022-09-13T07:39:39.171 DEBUG:boto:Token: None 2022-09-13T07:39:39.171 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj 2022-09-13T07:39:39.172 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yNRgBgvM/Dazb60sLtPQzs9zxAc= 2022-09-13T07:39:39.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yNRgBgvM/Dazb60sLtPQzs9zxAc=', 'Content-Length': '0'} 2022-09-13T07:39:39.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001eaff16367621263-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:39.177 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.178 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.178 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.179 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.179 DEBUG:boto:Method: GET 2022-09-13T07:39:39.180 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.180 DEBUG:boto:Data: 2022-09-13T07:39:39.181 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.181 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.182 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.183 DEBUG:boto:Params: {} 2022-09-13T07:39:39.183 DEBUG:boto:Token: None 2022-09-13T07:39:39.184 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.184 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DYcRX25eJU+B3TLJlUARQXNsQNM= 2022-09-13T07:39:39.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DYcRX25eJU+B3TLJlUARQXNsQNM=', 'Content-Length': '0'} 2022-09-13T07:39:39.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027fb37d09c2e8013-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.188 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.188 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.189 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.189 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.190 DEBUG:boto:Method: GET 2022-09-13T07:39:39.190 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.191 DEBUG:boto:Data: 2022-09-13T07:39:39.191 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.192 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.192 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.193 DEBUG:boto:Params: {} 2022-09-13T07:39:39.193 DEBUG:boto:Token: None 2022-09-13T07:39:39.194 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.195 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DYcRX25eJU+B3TLJlUARQXNsQNM= 2022-09-13T07:39:39.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DYcRX25eJU+B3TLJlUARQXNsQNM=', 'Content-Length': '0'} 2022-09-13T07:39:39.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b82996918414c4a6-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.198 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:39.199 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:39.199 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.200 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.200 DEBUG:boto:Method: GET 2022-09-13T07:39:39.201 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.201 DEBUG:boto:Data: 2022-09-13T07:39:39.202 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.202 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.203 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.203 DEBUG:boto:Params: {} 2022-09-13T07:39:39.204 DEBUG:boto:Token: None 2022-09-13T07:39:39.204 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/ 2022-09-13T07:39:39.205 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM= 2022-09-13T07:39:39.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM=', 'Content-Length': '0'} 2022-09-13T07:39:39.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa391fda6fd2c074-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.209 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:39.210 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:39.210 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:39.211 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.211 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.212 DEBUG:boto:Method: GET 2022-09-13T07:39:39.212 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.213 DEBUG:boto:Data: 2022-09-13T07:39:39.213 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.214 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.214 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.215 DEBUG:boto:Params: {} 2022-09-13T07:39:39.215 DEBUG:boto:Token: None 2022-09-13T07:39:39.216 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/ 2022-09-13T07:39:39.217 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM= 2022-09-13T07:39:39.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM=', 'Content-Length': '0'} 2022-09-13T07:39:39.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1cebb2b8f8d16d4-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.221 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:39.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:39.224 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.225 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.226 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.228 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.229 DEBUG:boto:Method: GET 2022-09-13T07:39:39.230 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.231 DEBUG:boto:Data: 2022-09-13T07:39:39.233 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.234 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.235 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.236 DEBUG:boto:Params: {} 2022-09-13T07:39:39.237 DEBUG:boto:Token: None 2022-09-13T07:39:39.238 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.239 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NoY8wwqzaDst7nXlUyDDshRgorI= 2022-09-13T07:39:39.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NoY8wwqzaDst7nXlUyDDshRgorI=', 'Content-Length': '0'} 2022-09-13T07:39:39.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc6bb56094f5df80-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.244 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.245 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.245 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.246 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.246 DEBUG:boto:Method: GET 2022-09-13T07:39:39.247 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.248 DEBUG:boto:Data: 2022-09-13T07:39:39.249 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.250 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.251 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.252 DEBUG:boto:Params: {} 2022-09-13T07:39:39.253 DEBUG:boto:Token: None 2022-09-13T07:39:39.254 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.255 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NoY8wwqzaDst7nXlUyDDshRgorI= 2022-09-13T07:39:39.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NoY8wwqzaDst7nXlUyDDshRgorI=', 'Content-Length': '0'} 2022-09-13T07:39:39.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b7fb07fea3b34ea-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.261 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.261 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.263 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.264 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.266 DEBUG:boto:Method: HEAD 2022-09-13T07:39:39.268 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.269 DEBUG:boto:Data: 2022-09-13T07:39:39.270 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.271 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.272 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.274 DEBUG:boto:Params: {} 2022-09-13T07:39:39.275 DEBUG:boto:Token: None 2022-09-13T07:39:39.276 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.277 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Nc3r6/C19ABPN005Zt8zyF5nBM8= 2022-09-13T07:39:39.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Nc3r6/C19ABPN005Zt8zyF5nBM8=', 'Content-Length': '0'} 2022-09-13T07:39:39.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f76b99e894e046a1-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.282 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.283 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.283 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.284 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.285 DEBUG:boto:Method: HEAD 2022-09-13T07:39:39.286 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.287 DEBUG:boto:Data: 2022-09-13T07:39:39.288 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.289 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.289 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.290 DEBUG:boto:Params: {} 2022-09-13T07:39:39.291 DEBUG:boto:Token: None 2022-09-13T07:39:39.292 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.292 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Nc3r6/C19ABPN005Zt8zyF5nBM8= 2022-09-13T07:39:39.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Nc3r6/C19ABPN005Zt8zyF5nBM8=', 'Content-Length': '0'} 2022-09-13T07:39:39.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005217235891c7dbe1-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:39.297 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.298 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.299 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.300 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.301 DEBUG:boto:Method: GET 2022-09-13T07:39:39.302 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.303 DEBUG:boto:Data: 2022-09-13T07:39:39.304 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.305 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.306 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.307 DEBUG:boto:Params: {} 2022-09-13T07:39:39.308 DEBUG:boto:Token: None 2022-09-13T07:39:39.309 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.310 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NoY8wwqzaDst7nXlUyDDshRgorI= 2022-09-13T07:39:39.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NoY8wwqzaDst7nXlUyDDshRgorI=', 'Content-Length': '0'} 2022-09-13T07:39:39.314 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000828129c1d1b667d6-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.315 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.316 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.317 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.318 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.319 DEBUG:boto:Method: GET 2022-09-13T07:39:39.320 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.321 DEBUG:boto:Data: 2022-09-13T07:39:39.321 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.322 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.323 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.324 DEBUG:boto:Params: {} 2022-09-13T07:39:39.325 DEBUG:boto:Token: None 2022-09-13T07:39:39.326 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:39.328 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NoY8wwqzaDst7nXlUyDDshRgorI= 2022-09-13T07:39:39.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NoY8wwqzaDst7nXlUyDDshRgorI=', 'Content-Length': '0'} 2022-09-13T07:39:39.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000247258a90c6fd85-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.374 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:39.376 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:39.377 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.378 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.380 DEBUG:boto:Method: GET 2022-09-13T07:39:39.381 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.382 DEBUG:boto:Data: 2022-09-13T07:39:39.382 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.383 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.383 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.384 DEBUG:boto:Params: {} 2022-09-13T07:39:39.385 DEBUG:boto:Token: None 2022-09-13T07:39:39.385 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/ 2022-09-13T07:39:39.386 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM= 2022-09-13T07:39:39.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM=', 'Content-Length': '0'} 2022-09-13T07:39:39.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c4b721689313f25-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.391 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:39.392 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:39.392 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:39.393 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.393 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.394 DEBUG:boto:Method: GET 2022-09-13T07:39:39.395 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.395 DEBUG:boto:Data: 2022-09-13T07:39:39.396 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.397 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.398 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.399 DEBUG:boto:Params: {} 2022-09-13T07:39:39.400 DEBUG:boto:Token: None 2022-09-13T07:39:39.400 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/ 2022-09-13T07:39:39.401 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM= 2022-09-13T07:39:39.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM=', 'Content-Length': '0'} 2022-09-13T07:39:39.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007441fcc85f6f6af3-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.408 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:39.409 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.410 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.411 DEBUG:boto:Method: HEAD 2022-09-13T07:39:39.412 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:39.412 DEBUG:boto:Data: 2022-09-13T07:39:39.413 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.413 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.414 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.414 DEBUG:boto:Params: {} 2022-09-13T07:39:39.416 DEBUG:boto:Token: None 2022-09-13T07:39:39.416 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj 2022-09-13T07:39:39.417 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yNRgBgvM/Dazb60sLtPQzs9zxAc= 2022-09-13T07:39:39.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yNRgBgvM/Dazb60sLtPQzs9zxAc=', 'Content-Length': '0'} 2022-09-13T07:39:39.421 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033d89746d205c800-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.423 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.424 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.424 DEBUG:boto:Method: HEAD 2022-09-13T07:39:39.425 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:39.426 DEBUG:boto:Data: 2022-09-13T07:39:39.426 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.427 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.427 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.428 DEBUG:boto:Params: {} 2022-09-13T07:39:39.429 DEBUG:boto:Token: None 2022-09-13T07:39:39.429 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj 2022-09-13T07:39:39.430 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yNRgBgvM/Dazb60sLtPQzs9zxAc= 2022-09-13T07:39:39.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yNRgBgvM/Dazb60sLtPQzs9zxAc=', 'Content-Length': '0'} 2022-09-13T07:39:39.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003847bc337a0b7e69-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:39.436 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.437 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.437 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.438 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.438 DEBUG:boto:Method: GET 2022-09-13T07:39:39.439 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.439 DEBUG:boto:Data: 2022-09-13T07:39:39.440 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.440 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.441 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.441 DEBUG:boto:Params: {} 2022-09-13T07:39:39.442 DEBUG:boto:Token: None 2022-09-13T07:39:39.443 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.444 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DYcRX25eJU+B3TLJlUARQXNsQNM= 2022-09-13T07:39:39.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DYcRX25eJU+B3TLJlUARQXNsQNM=', 'Content-Length': '0'} 2022-09-13T07:39:39.447 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007676c81b86bdac34-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.494 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.495 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.495 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.496 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.497 DEBUG:boto:Method: GET 2022-09-13T07:39:39.497 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.498 DEBUG:boto:Data: 2022-09-13T07:39:39.498 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.499 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.499 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.500 DEBUG:boto:Params: {} 2022-09-13T07:39:39.500 DEBUG:boto:Token: None 2022-09-13T07:39:39.501 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.503 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DYcRX25eJU+B3TLJlUARQXNsQNM= 2022-09-13T07:39:39.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DYcRX25eJU+B3TLJlUARQXNsQNM=', 'Content-Length': '0'} 2022-09-13T07:39:39.507 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9fb29b5a8f4aa82-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.508 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:39.509 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:39.510 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.511 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.512 DEBUG:boto:Method: GET 2022-09-13T07:39:39.512 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.513 DEBUG:boto:Data: 2022-09-13T07:39:39.514 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.515 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.515 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.516 DEBUG:boto:Params: {} 2022-09-13T07:39:39.517 DEBUG:boto:Token: None 2022-09-13T07:39:39.517 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/ 2022-09-13T07:39:39.518 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM= 2022-09-13T07:39:39.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM=', 'Content-Length': '0'} 2022-09-13T07:39:39.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077c9edd6f5158843-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.524 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:39.525 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:39.526 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:39.527 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.528 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.528 DEBUG:boto:Method: GET 2022-09-13T07:39:39.529 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.529 DEBUG:boto:Data: 2022-09-13T07:39:39.530 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.530 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.531 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.532 DEBUG:boto:Params: {} 2022-09-13T07:39:39.532 DEBUG:boto:Token: None 2022-09-13T07:39:39.533 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/ 2022-09-13T07:39:39.534 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM= 2022-09-13T07:39:39.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM=', 'Content-Length': '0'} 2022-09-13T07:39:39.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e021940ef87befb9-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.578 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:39.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:39.580 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.581 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.581 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.582 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.583 DEBUG:boto:Method: GET 2022-09-13T07:39:39.583 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.584 DEBUG:boto:Data: 2022-09-13T07:39:39.584 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.585 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.586 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.586 DEBUG:boto:Params: {} 2022-09-13T07:39:39.587 DEBUG:boto:Token: None 2022-09-13T07:39:39.587 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.588 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WR3NIQGVlcsfZpMOJA9i9e1rW/8= 2022-09-13T07:39:39.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WR3NIQGVlcsfZpMOJA9i9e1rW/8=', 'Content-Length': '0'} 2022-09-13T07:39:39.591 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db661071a5ee887d-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.592 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.592 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.593 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.593 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.594 DEBUG:boto:Method: GET 2022-09-13T07:39:39.595 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.595 DEBUG:boto:Data: 2022-09-13T07:39:39.596 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.596 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.597 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.597 DEBUG:boto:Params: {} 2022-09-13T07:39:39.598 DEBUG:boto:Token: None 2022-09-13T07:39:39.599 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.600 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WR3NIQGVlcsfZpMOJA9i9e1rW/8= 2022-09-13T07:39:39.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WR3NIQGVlcsfZpMOJA9i9e1rW/8=', 'Content-Length': '0'} 2022-09-13T07:39:39.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072794ef62a7a72c5-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.605 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.606 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.607 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.608 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.608 DEBUG:boto:Method: HEAD 2022-09-13T07:39:39.609 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.609 DEBUG:boto:Data: 2022-09-13T07:39:39.610 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.610 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.611 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.611 DEBUG:boto:Params: {} 2022-09-13T07:39:39.612 DEBUG:boto:Token: None 2022-09-13T07:39:39.613 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.613 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KgGgelawVrFzaJhE4oXKsWetkYs= 2022-09-13T07:39:39.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KgGgelawVrFzaJhE4oXKsWetkYs=', 'Content-Length': '0'} 2022-09-13T07:39:39.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045599a0b65bb75e8-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.617 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.617 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.618 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.618 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.619 DEBUG:boto:Method: HEAD 2022-09-13T07:39:39.619 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.620 DEBUG:boto:Data: 2022-09-13T07:39:39.621 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.621 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.622 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.622 DEBUG:boto:Params: {} 2022-09-13T07:39:39.623 DEBUG:boto:Token: None 2022-09-13T07:39:39.623 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.624 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KgGgelawVrFzaJhE4oXKsWetkYs= 2022-09-13T07:39:39.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KgGgelawVrFzaJhE4oXKsWetkYs=', 'Content-Length': '0'} 2022-09-13T07:39:39.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003fb322d4149b9390-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:39.628 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.629 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.629 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.630 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.630 DEBUG:boto:Method: GET 2022-09-13T07:39:39.631 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.631 DEBUG:boto:Data: 2022-09-13T07:39:39.632 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.632 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.633 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.633 DEBUG:boto:Params: {} 2022-09-13T07:39:39.634 DEBUG:boto:Token: None 2022-09-13T07:39:39.635 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.636 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WR3NIQGVlcsfZpMOJA9i9e1rW/8= 2022-09-13T07:39:39.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WR3NIQGVlcsfZpMOJA9i9e1rW/8=', 'Content-Length': '0'} 2022-09-13T07:39:39.639 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f5e2baa0191616f-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.639 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.640 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.641 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.641 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.642 DEBUG:boto:Method: GET 2022-09-13T07:39:39.642 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.643 DEBUG:boto:Data: 2022-09-13T07:39:39.643 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.644 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.644 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.645 DEBUG:boto:Params: {} 2022-09-13T07:39:39.646 DEBUG:boto:Token: None 2022-09-13T07:39:39.647 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:39.647 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WR3NIQGVlcsfZpMOJA9i9e1rW/8= 2022-09-13T07:39:39.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WR3NIQGVlcsfZpMOJA9i9e1rW/8=', 'Content-Length': '0'} 2022-09-13T07:39:39.650 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007db3b4904b59ec73-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.651 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:39.652 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:39.652 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.653 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.653 DEBUG:boto:Method: GET 2022-09-13T07:39:39.654 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.654 DEBUG:boto:Data: 2022-09-13T07:39:39.655 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.656 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.657 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.658 DEBUG:boto:Params: {} 2022-09-13T07:39:39.658 DEBUG:boto:Token: None 2022-09-13T07:39:39.659 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/ 2022-09-13T07:39:39.660 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM= 2022-09-13T07:39:39.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM=', 'Content-Length': '0'} 2022-09-13T07:39:39.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000661fa39db5e09505-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.706 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:39.707 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:39.708 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:39.709 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.710 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.711 DEBUG:boto:Method: GET 2022-09-13T07:39:39.712 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.712 DEBUG:boto:Data: 2022-09-13T07:39:39.713 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.714 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.715 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.716 DEBUG:boto:Params: {} 2022-09-13T07:39:39.717 DEBUG:boto:Token: None 2022-09-13T07:39:39.719 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/ 2022-09-13T07:39:39.720 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM= 2022-09-13T07:39:39.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM=', 'Content-Length': '0'} 2022-09-13T07:39:39.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8cd4108539e9ea5-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.725 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:39.727 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.727 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.728 DEBUG:boto:Method: HEAD 2022-09-13T07:39:39.729 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:39.730 DEBUG:boto:Data: 2022-09-13T07:39:39.731 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.731 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.732 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.733 DEBUG:boto:Params: {} 2022-09-13T07:39:39.734 DEBUG:boto:Token: None 2022-09-13T07:39:39.734 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj 2022-09-13T07:39:39.735 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yNRgBgvM/Dazb60sLtPQzs9zxAc= 2022-09-13T07:39:39.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yNRgBgvM/Dazb60sLtPQzs9zxAc=', 'Content-Length': '0'} 2022-09-13T07:39:39.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000aad47a0c1be227e-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.741 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.742 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.742 DEBUG:boto:Method: HEAD 2022-09-13T07:39:39.743 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:39.744 DEBUG:boto:Data: 2022-09-13T07:39:39.744 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.745 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.745 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.746 DEBUG:boto:Params: {} 2022-09-13T07:39:39.747 DEBUG:boto:Token: None 2022-09-13T07:39:39.747 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj 2022-09-13T07:39:39.748 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yNRgBgvM/Dazb60sLtPQzs9zxAc= 2022-09-13T07:39:39.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yNRgBgvM/Dazb60sLtPQzs9zxAc=', 'Content-Length': '0'} 2022-09-13T07:39:39.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007dcaa5ae0f5df937-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:39.763 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.765 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.766 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.767 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.768 DEBUG:boto:Method: GET 2022-09-13T07:39:39.769 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.770 DEBUG:boto:Data: 2022-09-13T07:39:39.771 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.772 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.773 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.775 DEBUG:boto:Params: {} 2022-09-13T07:39:39.776 DEBUG:boto:Token: None 2022-09-13T07:39:39.777 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.778 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DYcRX25eJU+B3TLJlUARQXNsQNM= 2022-09-13T07:39:39.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DYcRX25eJU+B3TLJlUARQXNsQNM=', 'Content-Length': '0'} 2022-09-13T07:39:39.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f50ce9797138771d-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.783 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.784 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.786 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.787 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.788 DEBUG:boto:Method: GET 2022-09-13T07:39:39.789 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.790 DEBUG:boto:Data: 2022-09-13T07:39:39.791 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.792 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.793 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.794 DEBUG:boto:Params: {} 2022-09-13T07:39:39.795 DEBUG:boto:Token: None 2022-09-13T07:39:39.796 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:39.797 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:DYcRX25eJU+B3TLJlUARQXNsQNM= 2022-09-13T07:39:39.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:DYcRX25eJU+B3TLJlUARQXNsQNM=', 'Content-Length': '0'} 2022-09-13T07:39:39.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e71d76cee253c263-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.846 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:39.847 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:39.848 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.849 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.850 DEBUG:boto:Method: GET 2022-09-13T07:39:39.851 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.852 DEBUG:boto:Data: 2022-09-13T07:39:39.853 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.854 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.855 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.856 DEBUG:boto:Params: {} 2022-09-13T07:39:39.858 DEBUG:boto:Token: None 2022-09-13T07:39:39.859 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/ 2022-09-13T07:39:39.860 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM= 2022-09-13T07:39:39.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM=', 'Content-Length': '0'} 2022-09-13T07:39:39.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cabf23086a30da69-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.866 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:39.868 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:39.869 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:39.870 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.871 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.872 DEBUG:boto:Method: GET 2022-09-13T07:39:39.873 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.874 DEBUG:boto:Data: 2022-09-13T07:39:39.875 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.876 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.877 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.878 DEBUG:boto:Params: {} 2022-09-13T07:39:39.879 DEBUG:boto:Token: None 2022-09-13T07:39:39.880 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/ 2022-09-13T07:39:39.880 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM= 2022-09-13T07:39:39.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OqG/SmYmtaqqkeFFagtXPrv5KEM=', 'Content-Length': '0'} 2022-09-13T07:39:39.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003413a91d9d39b3bd-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.886 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:39.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:39.888 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.889 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.891 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.892 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.893 DEBUG:boto:Method: GET 2022-09-13T07:39:39.893 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.894 DEBUG:boto:Data: 2022-09-13T07:39:39.895 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.896 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.897 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.898 DEBUG:boto:Params: {} 2022-09-13T07:39:39.899 DEBUG:boto:Token: None 2022-09-13T07:39:39.900 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.901 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FQSfk0/de5FIbEQBEL/bEbITaWM= 2022-09-13T07:39:39.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FQSfk0/de5FIbEQBEL/bEbITaWM=', 'Content-Length': '0'} 2022-09-13T07:39:39.905 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e876d7fa02f8e8f-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.907 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.909 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.911 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.912 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.914 DEBUG:boto:Method: GET 2022-09-13T07:39:39.915 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.917 DEBUG:boto:Data: 2022-09-13T07:39:39.917 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.918 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.918 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.919 DEBUG:boto:Params: {} 2022-09-13T07:39:39.920 DEBUG:boto:Token: None 2022-09-13T07:39:39.920 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.921 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FQSfk0/de5FIbEQBEL/bEbITaWM= 2022-09-13T07:39:39.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FQSfk0/de5FIbEQBEL/bEbITaWM=', 'Content-Length': '0'} 2022-09-13T07:39:39.924 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d338734a7153dc2e-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.926 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.927 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.928 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.929 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.930 DEBUG:boto:Method: HEAD 2022-09-13T07:39:39.931 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.932 DEBUG:boto:Data: 2022-09-13T07:39:39.933 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.934 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.935 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.936 DEBUG:boto:Params: {} 2022-09-13T07:39:39.937 DEBUG:boto:Token: None 2022-09-13T07:39:39.938 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.939 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ga9HI5fgTpXP6lVjlPTnZMrJ/U4= 2022-09-13T07:39:39.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ga9HI5fgTpXP6lVjlPTnZMrJ/U4=', 'Content-Length': '0'} 2022-09-13T07:39:39.943 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c819f3f11f5e5bf-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.944 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.945 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.946 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.947 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.948 DEBUG:boto:Method: HEAD 2022-09-13T07:39:39.949 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.950 DEBUG:boto:Data: 2022-09-13T07:39:39.951 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.952 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.953 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.953 DEBUG:boto:Params: {} 2022-09-13T07:39:39.955 DEBUG:boto:Token: None 2022-09-13T07:39:39.956 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.957 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ga9HI5fgTpXP6lVjlPTnZMrJ/U4= 2022-09-13T07:39:39.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ga9HI5fgTpXP6lVjlPTnZMrJ/U4=', 'Content-Length': '0'} 2022-09-13T07:39:39.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020bf9d1204c44217-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:39.963 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.963 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.965 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.966 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.967 DEBUG:boto:Method: GET 2022-09-13T07:39:39.968 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.969 DEBUG:boto:Data: 2022-09-13T07:39:39.969 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.970 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.971 DEBUG:boto:Port: 8001 2022-09-13T07:39:39.972 DEBUG:boto:Params: {} 2022-09-13T07:39:39.973 DEBUG:boto:Token: None 2022-09-13T07:39:39.974 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.975 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FQSfk0/de5FIbEQBEL/bEbITaWM= 2022-09-13T07:39:39.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FQSfk0/de5FIbEQBEL/bEbITaWM=', 'Content-Length': '0'} 2022-09-13T07:39:39.979 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066be5b6d1bd0e623-00632033bb-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.980 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:39.981 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:39.982 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.983 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.984 DEBUG:boto:Method: GET 2022-09-13T07:39:39.985 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.986 DEBUG:boto:Data: 2022-09-13T07:39:39.987 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.987 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:39.988 DEBUG:boto:Port: 8000 2022-09-13T07:39:39.988 DEBUG:boto:Params: {} 2022-09-13T07:39:39.989 DEBUG:boto:Token: None 2022-09-13T07:39:39.989 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:39 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:39.990 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FQSfk0/de5FIbEQBEL/bEbITaWM= 2022-09-13T07:39:39.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:39 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FQSfk0/de5FIbEQBEL/bEbITaWM=', 'Content-Length': '0'} 2022-09-13T07:39:39.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000294fc7554410cb3d-00632033bb-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:39.994 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:39.995 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:39.995 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.996 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.997 DEBUG:boto:Method: GET 2022-09-13T07:39:39.997 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:39.998 DEBUG:boto:Data: 2022-09-13T07:39:39.998 DEBUG:boto:Headers: {} 2022-09-13T07:39:39.999 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:39.999 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.000 DEBUG:boto:Params: {} 2022-09-13T07:39:40.000 DEBUG:boto:Token: None 2022-09-13T07:39:40.001 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.002 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005917e07bf8c7b9d3-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.006 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.007 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.007 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.008 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.008 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.009 DEBUG:boto:Method: GET 2022-09-13T07:39:40.010 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.011 DEBUG:boto:Data: 2022-09-13T07:39:40.011 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.012 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.012 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.013 DEBUG:boto:Params: {} 2022-09-13T07:39:40.013 DEBUG:boto:Token: None 2022-09-13T07:39:40.014 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.015 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc58eb54b223f814-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.019 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.020 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.021 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.021 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.022 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:40.022 DEBUG:boto:Data: 2022-09-13T07:39:40.023 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.023 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.024 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.024 DEBUG:boto:Params: {} 2022-09-13T07:39:40.025 DEBUG:boto:Token: None 2022-09-13T07:39:40.025 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj 2022-09-13T07:39:40.026 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ= 2022-09-13T07:39:40.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.038 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000104f5b7f52c303bc-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.040 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.041 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.042 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.042 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:40.043 DEBUG:boto:Data: 2022-09-13T07:39:40.044 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.045 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.045 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.046 DEBUG:boto:Params: {} 2022-09-13T07:39:40.047 DEBUG:boto:Token: None 2022-09-13T07:39:40.048 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj 2022-09-13T07:39:40.048 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ= 2022-09-13T07:39:40.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000af82e092b4c3680-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:40.055 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.055 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.056 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.057 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.057 DEBUG:boto:Method: GET 2022-09-13T07:39:40.058 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.058 DEBUG:boto:Data: 2022-09-13T07:39:40.059 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.059 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.060 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.060 DEBUG:boto:Params: {} 2022-09-13T07:39:40.061 DEBUG:boto:Token: None 2022-09-13T07:39:40.062 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.062 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI= 2022-09-13T07:39:40.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI=', 'Content-Length': '0'} 2022-09-13T07:39:40.065 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5cfecef145c257a-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.111 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.112 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.113 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.114 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.115 DEBUG:boto:Method: GET 2022-09-13T07:39:40.115 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.116 DEBUG:boto:Data: 2022-09-13T07:39:40.116 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.117 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.117 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.118 DEBUG:boto:Params: {} 2022-09-13T07:39:40.118 DEBUG:boto:Token: None 2022-09-13T07:39:40.119 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.120 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI= 2022-09-13T07:39:40.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI=', 'Content-Length': '0'} 2022-09-13T07:39:40.122 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fec597b0c1e2db9-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.123 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.124 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.124 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.125 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.125 DEBUG:boto:Method: GET 2022-09-13T07:39:40.126 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.126 DEBUG:boto:Data: 2022-09-13T07:39:40.127 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.127 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.128 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.128 DEBUG:boto:Params: {} 2022-09-13T07:39:40.129 DEBUG:boto:Token: None 2022-09-13T07:39:40.130 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.130 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac8a0cdc847f4b68-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.135 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.135 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.136 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.136 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.137 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.137 DEBUG:boto:Method: GET 2022-09-13T07:39:40.138 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.139 DEBUG:boto:Data: 2022-09-13T07:39:40.139 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.140 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.140 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.141 DEBUG:boto:Params: {} 2022-09-13T07:39:40.141 DEBUG:boto:Token: None 2022-09-13T07:39:40.142 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.142 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ed473062046fb3f-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.190 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:40.192 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.192 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.193 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.193 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.194 DEBUG:boto:Method: GET 2022-09-13T07:39:40.195 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.196 DEBUG:boto:Data: 2022-09-13T07:39:40.197 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.197 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.198 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.199 DEBUG:boto:Params: {} 2022-09-13T07:39:40.200 DEBUG:boto:Token: None 2022-09-13T07:39:40.201 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.202 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cqTGUlX9zMlvbMshJxcBA6BRv5A= 2022-09-13T07:39:40.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cqTGUlX9zMlvbMshJxcBA6BRv5A=', 'Content-Length': '0'} 2022-09-13T07:39:40.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072b606e6ce27f1ae-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.207 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.208 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.209 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.210 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.211 DEBUG:boto:Method: GET 2022-09-13T07:39:40.212 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.213 DEBUG:boto:Data: 2022-09-13T07:39:40.214 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.215 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.216 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.217 DEBUG:boto:Params: {} 2022-09-13T07:39:40.218 DEBUG:boto:Token: None 2022-09-13T07:39:40.219 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.220 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cqTGUlX9zMlvbMshJxcBA6BRv5A= 2022-09-13T07:39:40.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cqTGUlX9zMlvbMshJxcBA6BRv5A=', 'Content-Length': '0'} 2022-09-13T07:39:40.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000188a6e69ee848376-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.226 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.227 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.228 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.229 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.230 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.231 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.231 DEBUG:boto:Data: 2022-09-13T07:39:40.232 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.233 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.234 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.235 DEBUG:boto:Params: {} 2022-09-13T07:39:40.236 DEBUG:boto:Token: None 2022-09-13T07:39:40.237 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.238 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EH9Hg0P3JNU/bO+c1SQzUgmvj6k= 2022-09-13T07:39:40.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EH9Hg0P3JNU/bO+c1SQzUgmvj6k=', 'Content-Length': '0'} 2022-09-13T07:39:40.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e49dad99456d35dc-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.243 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.245 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.245 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.246 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.247 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.248 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.249 DEBUG:boto:Data: 2022-09-13T07:39:40.250 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.251 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.252 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.253 DEBUG:boto:Params: {} 2022-09-13T07:39:40.254 DEBUG:boto:Token: None 2022-09-13T07:39:40.255 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.255 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EH9Hg0P3JNU/bO+c1SQzUgmvj6k= 2022-09-13T07:39:40.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EH9Hg0P3JNU/bO+c1SQzUgmvj6k=', 'Content-Length': '0'} 2022-09-13T07:39:40.258 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c78e89029f533f67-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:40.260 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.260 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.261 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.261 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.262 DEBUG:boto:Method: GET 2022-09-13T07:39:40.263 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.263 DEBUG:boto:Data: 2022-09-13T07:39:40.264 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.264 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.265 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.265 DEBUG:boto:Params: {} 2022-09-13T07:39:40.266 DEBUG:boto:Token: None 2022-09-13T07:39:40.266 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.267 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cqTGUlX9zMlvbMshJxcBA6BRv5A= 2022-09-13T07:39:40.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cqTGUlX9zMlvbMshJxcBA6BRv5A=', 'Content-Length': '0'} 2022-09-13T07:39:40.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d425c9b0a92c786-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.271 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.272 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.273 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.274 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.275 DEBUG:boto:Method: GET 2022-09-13T07:39:40.276 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.277 DEBUG:boto:Data: 2022-09-13T07:39:40.278 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.279 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.280 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.281 DEBUG:boto:Params: {} 2022-09-13T07:39:40.282 DEBUG:boto:Token: None 2022-09-13T07:39:40.283 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:40.284 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cqTGUlX9zMlvbMshJxcBA6BRv5A= 2022-09-13T07:39:40.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cqTGUlX9zMlvbMshJxcBA6BRv5A=', 'Content-Length': '0'} 2022-09-13T07:39:40.288 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055d406fdcedf5e0e-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.289 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.290 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.291 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.291 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.292 DEBUG:boto:Method: GET 2022-09-13T07:39:40.292 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.293 DEBUG:boto:Data: 2022-09-13T07:39:40.293 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.294 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.294 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.295 DEBUG:boto:Params: {} 2022-09-13T07:39:40.297 DEBUG:boto:Token: None 2022-09-13T07:39:40.298 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.299 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5ea3346adf3936f-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.346 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.348 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.349 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.350 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.350 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.351 DEBUG:boto:Method: GET 2022-09-13T07:39:40.352 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.352 DEBUG:boto:Data: 2022-09-13T07:39:40.353 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.353 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.354 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.354 DEBUG:boto:Params: {} 2022-09-13T07:39:40.355 DEBUG:boto:Token: None 2022-09-13T07:39:40.356 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.357 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000072c786d547adf83-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.363 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.364 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.365 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.365 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.366 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:40.366 DEBUG:boto:Data: 2022-09-13T07:39:40.367 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.367 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.368 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.368 DEBUG:boto:Params: {} 2022-09-13T07:39:40.369 DEBUG:boto:Token: None 2022-09-13T07:39:40.370 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj 2022-09-13T07:39:40.371 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ= 2022-09-13T07:39:40.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.375 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000169c58220a4b38ab-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.376 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.377 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.377 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.378 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:40.378 DEBUG:boto:Data: 2022-09-13T07:39:40.379 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.379 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.379 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.380 DEBUG:boto:Params: {} 2022-09-13T07:39:40.380 DEBUG:boto:Token: None 2022-09-13T07:39:40.381 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj 2022-09-13T07:39:40.382 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ= 2022-09-13T07:39:40.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.385 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099093c90f6ac1d6e-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:40.387 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.387 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.388 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.388 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.389 DEBUG:boto:Method: GET 2022-09-13T07:39:40.389 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.390 DEBUG:boto:Data: 2022-09-13T07:39:40.390 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.391 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.391 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.392 DEBUG:boto:Params: {} 2022-09-13T07:39:40.393 DEBUG:boto:Token: None 2022-09-13T07:39:40.393 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.394 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI= 2022-09-13T07:39:40.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI=', 'Content-Length': '0'} 2022-09-13T07:39:40.397 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c25b051fc1f1f173-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.398 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.398 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.399 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.399 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.400 DEBUG:boto:Method: GET 2022-09-13T07:39:40.400 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.401 DEBUG:boto:Data: 2022-09-13T07:39:40.402 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.402 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.403 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.403 DEBUG:boto:Params: {} 2022-09-13T07:39:40.404 DEBUG:boto:Token: None 2022-09-13T07:39:40.404 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.405 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI= 2022-09-13T07:39:40.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI=', 'Content-Length': '0'} 2022-09-13T07:39:40.408 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000af31e7592de41f-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.409 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.409 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.410 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.410 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.411 DEBUG:boto:Method: GET 2022-09-13T07:39:40.411 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.412 DEBUG:boto:Data: 2022-09-13T07:39:40.412 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.413 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.413 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.414 DEBUG:boto:Params: {} 2022-09-13T07:39:40.414 DEBUG:boto:Token: None 2022-09-13T07:39:40.415 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.415 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000692ebe7ad2fd730a-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.419 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.420 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.421 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.421 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.422 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.422 DEBUG:boto:Method: GET 2022-09-13T07:39:40.423 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.423 DEBUG:boto:Data: 2022-09-13T07:39:40.424 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.424 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.425 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.425 DEBUG:boto:Params: {} 2022-09-13T07:39:40.426 DEBUG:boto:Token: None 2022-09-13T07:39:40.427 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.427 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000161019b91dd172c1-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.432 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:40.433 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.434 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.434 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.435 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.435 DEBUG:boto:Method: GET 2022-09-13T07:39:40.436 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.436 DEBUG:boto:Data: 2022-09-13T07:39:40.437 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.437 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.438 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.438 DEBUG:boto:Params: {} 2022-09-13T07:39:40.439 DEBUG:boto:Token: None 2022-09-13T07:39:40.440 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.440 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EsitYk0n7wNyvKU5zY9eEJ8RB5o= 2022-09-13T07:39:40.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EsitYk0n7wNyvKU5zY9eEJ8RB5o=', 'Content-Length': '0'} 2022-09-13T07:39:40.444 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd0d7e2acd79eca9-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.486 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.487 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.487 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.488 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.488 DEBUG:boto:Method: GET 2022-09-13T07:39:40.489 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.490 DEBUG:boto:Data: 2022-09-13T07:39:40.490 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.491 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.491 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.492 DEBUG:boto:Params: {} 2022-09-13T07:39:40.492 DEBUG:boto:Token: None 2022-09-13T07:39:40.493 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.493 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EsitYk0n7wNyvKU5zY9eEJ8RB5o= 2022-09-13T07:39:40.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EsitYk0n7wNyvKU5zY9eEJ8RB5o=', 'Content-Length': '0'} 2022-09-13T07:39:40.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000011cd280120c9bca-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.497 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.498 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.498 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.499 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.499 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.500 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.500 DEBUG:boto:Data: 2022-09-13T07:39:40.501 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.501 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.502 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.502 DEBUG:boto:Params: {} 2022-09-13T07:39:40.503 DEBUG:boto:Token: None 2022-09-13T07:39:40.503 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.504 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ThFwsbFkrhIC2FD2M8GRQHMjS/w= 2022-09-13T07:39:40.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ThFwsbFkrhIC2FD2M8GRQHMjS/w=', 'Content-Length': '0'} 2022-09-13T07:39:40.507 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c51abaa3a73ef86d-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.508 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.508 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.509 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.509 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.510 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.510 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.511 DEBUG:boto:Data: 2022-09-13T07:39:40.511 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.512 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.512 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.513 DEBUG:boto:Params: {} 2022-09-13T07:39:40.513 DEBUG:boto:Token: None 2022-09-13T07:39:40.514 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.514 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ThFwsbFkrhIC2FD2M8GRQHMjS/w= 2022-09-13T07:39:40.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ThFwsbFkrhIC2FD2M8GRQHMjS/w=', 'Content-Length': '0'} 2022-09-13T07:39:40.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068d10e6e46d2298d-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:40.518 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.519 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.519 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.520 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.520 DEBUG:boto:Method: GET 2022-09-13T07:39:40.521 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.521 DEBUG:boto:Data: 2022-09-13T07:39:40.522 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.522 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.523 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.523 DEBUG:boto:Params: {} 2022-09-13T07:39:40.524 DEBUG:boto:Token: None 2022-09-13T07:39:40.525 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.525 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EsitYk0n7wNyvKU5zY9eEJ8RB5o= 2022-09-13T07:39:40.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EsitYk0n7wNyvKU5zY9eEJ8RB5o=', 'Content-Length': '0'} 2022-09-13T07:39:40.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d33d4b756d6b954-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.529 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.530 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.530 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.531 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.531 DEBUG:boto:Method: GET 2022-09-13T07:39:40.532 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.532 DEBUG:boto:Data: 2022-09-13T07:39:40.533 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.533 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.534 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.534 DEBUG:boto:Params: {} 2022-09-13T07:39:40.535 DEBUG:boto:Token: None 2022-09-13T07:39:40.536 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:40.536 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EsitYk0n7wNyvKU5zY9eEJ8RB5o= 2022-09-13T07:39:40.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EsitYk0n7wNyvKU5zY9eEJ8RB5o=', 'Content-Length': '0'} 2022-09-13T07:39:40.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000599f219e5634f43-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.540 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.541 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.542 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.542 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.543 DEBUG:boto:Method: GET 2022-09-13T07:39:40.543 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.544 DEBUG:boto:Data: 2022-09-13T07:39:40.544 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.545 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.545 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.546 DEBUG:boto:Params: {} 2022-09-13T07:39:40.547 DEBUG:boto:Token: None 2022-09-13T07:39:40.547 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.548 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffce5f4a0c6e8390-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.552 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.553 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.554 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.554 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.555 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.555 DEBUG:boto:Method: GET 2022-09-13T07:39:40.556 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.556 DEBUG:boto:Data: 2022-09-13T07:39:40.557 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.557 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.558 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.558 DEBUG:boto:Params: {} 2022-09-13T07:39:40.559 DEBUG:boto:Token: None 2022-09-13T07:39:40.560 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.560 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005787a6b5b786b2fd-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.565 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.565 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.566 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.567 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.567 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:40.568 DEBUG:boto:Data: 2022-09-13T07:39:40.568 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.568 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.569 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.569 DEBUG:boto:Params: {} 2022-09-13T07:39:40.570 DEBUG:boto:Token: None 2022-09-13T07:39:40.571 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj 2022-09-13T07:39:40.571 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ= 2022-09-13T07:39:40.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.575 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4282d4de2cf6ca7-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.576 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.576 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.577 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.578 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:40.578 DEBUG:boto:Data: 2022-09-13T07:39:40.579 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.579 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.580 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.580 DEBUG:boto:Params: {} 2022-09-13T07:39:40.581 DEBUG:boto:Token: None 2022-09-13T07:39:40.581 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj 2022-09-13T07:39:40.582 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ= 2022-09-13T07:39:40.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.586 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000016d2eeb489462e-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:40.587 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.588 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.588 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.589 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.590 DEBUG:boto:Method: GET 2022-09-13T07:39:40.590 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.591 DEBUG:boto:Data: 2022-09-13T07:39:40.592 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.592 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.593 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.593 DEBUG:boto:Params: {} 2022-09-13T07:39:40.594 DEBUG:boto:Token: None 2022-09-13T07:39:40.595 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.595 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI= 2022-09-13T07:39:40.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI=', 'Content-Length': '0'} 2022-09-13T07:39:40.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005511363d65bc8421-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.644 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.645 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.646 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.647 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.648 DEBUG:boto:Method: GET 2022-09-13T07:39:40.649 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.650 DEBUG:boto:Data: 2022-09-13T07:39:40.651 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.652 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.653 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.654 DEBUG:boto:Params: {} 2022-09-13T07:39:40.655 DEBUG:boto:Token: None 2022-09-13T07:39:40.656 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.657 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI= 2022-09-13T07:39:40.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI=', 'Content-Length': '0'} 2022-09-13T07:39:40.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002cfc014df583fbfc-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.663 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.663 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.664 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.665 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.665 DEBUG:boto:Method: GET 2022-09-13T07:39:40.666 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.667 DEBUG:boto:Data: 2022-09-13T07:39:40.668 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.668 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.669 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.669 DEBUG:boto:Params: {} 2022-09-13T07:39:40.670 DEBUG:boto:Token: None 2022-09-13T07:39:40.671 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.671 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7fbed19192a24ba-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.676 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.677 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.677 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.678 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.679 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.679 DEBUG:boto:Method: GET 2022-09-13T07:39:40.680 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.680 DEBUG:boto:Data: 2022-09-13T07:39:40.681 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.681 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.682 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.682 DEBUG:boto:Params: {} 2022-09-13T07:39:40.683 DEBUG:boto:Token: None 2022-09-13T07:39:40.684 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.684 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012b7a2a661508c7a-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.689 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:40.691 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.691 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.692 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.692 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.693 DEBUG:boto:Method: GET 2022-09-13T07:39:40.693 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.694 DEBUG:boto:Data: 2022-09-13T07:39:40.694 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.695 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.696 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.696 DEBUG:boto:Params: {} 2022-09-13T07:39:40.697 DEBUG:boto:Token: None 2022-09-13T07:39:40.699 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.700 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7Sr/VDs4ke+RoNNkZLc27l7Bq5k= 2022-09-13T07:39:40.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7Sr/VDs4ke+RoNNkZLc27l7Bq5k=', 'Content-Length': '0'} 2022-09-13T07:39:40.704 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b79d5b54193b2bcf-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.705 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.706 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.707 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.708 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.709 DEBUG:boto:Method: GET 2022-09-13T07:39:40.709 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.710 DEBUG:boto:Data: 2022-09-13T07:39:40.711 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.711 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.712 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.712 DEBUG:boto:Params: {} 2022-09-13T07:39:40.713 DEBUG:boto:Token: None 2022-09-13T07:39:40.713 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.714 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7Sr/VDs4ke+RoNNkZLc27l7Bq5k= 2022-09-13T07:39:40.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7Sr/VDs4ke+RoNNkZLc27l7Bq5k=', 'Content-Length': '0'} 2022-09-13T07:39:40.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000594e2bae50f44e24-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.718 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.718 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.719 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.719 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.720 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.720 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.721 DEBUG:boto:Data: 2022-09-13T07:39:40.721 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.722 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.722 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.723 DEBUG:boto:Params: {} 2022-09-13T07:39:40.724 DEBUG:boto:Token: None 2022-09-13T07:39:40.724 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.725 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:28omMDm8Aj+gwCKmxWjoo2WN1Ms= 2022-09-13T07:39:40.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:28omMDm8Aj+gwCKmxWjoo2WN1Ms=', 'Content-Length': '0'} 2022-09-13T07:39:40.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa037cdd6dea2c1b-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.728 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.729 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.729 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.730 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.730 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.731 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.731 DEBUG:boto:Data: 2022-09-13T07:39:40.732 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.732 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.733 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.733 DEBUG:boto:Params: {} 2022-09-13T07:39:40.734 DEBUG:boto:Token: None 2022-09-13T07:39:40.734 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.735 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:28omMDm8Aj+gwCKmxWjoo2WN1Ms= 2022-09-13T07:39:40.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:28omMDm8Aj+gwCKmxWjoo2WN1Ms=', 'Content-Length': '0'} 2022-09-13T07:39:40.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f31a7a1ad24258a-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:40.739 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.739 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.740 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.740 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.741 DEBUG:boto:Method: GET 2022-09-13T07:39:40.741 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.742 DEBUG:boto:Data: 2022-09-13T07:39:40.742 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.743 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.743 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.744 DEBUG:boto:Params: {} 2022-09-13T07:39:40.745 DEBUG:boto:Token: None 2022-09-13T07:39:40.745 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.746 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7Sr/VDs4ke+RoNNkZLc27l7Bq5k= 2022-09-13T07:39:40.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7Sr/VDs4ke+RoNNkZLc27l7Bq5k=', 'Content-Length': '0'} 2022-09-13T07:39:40.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a28d0515265feafe-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.750 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.750 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.751 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.751 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.752 DEBUG:boto:Method: GET 2022-09-13T07:39:40.752 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.753 DEBUG:boto:Data: 2022-09-13T07:39:40.753 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.754 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.755 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.755 DEBUG:boto:Params: {} 2022-09-13T07:39:40.756 DEBUG:boto:Token: None 2022-09-13T07:39:40.756 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:40.757 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7Sr/VDs4ke+RoNNkZLc27l7Bq5k= 2022-09-13T07:39:40.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7Sr/VDs4ke+RoNNkZLc27l7Bq5k=', 'Content-Length': '0'} 2022-09-13T07:39:40.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc65a687799d4eb2-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.761 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.761 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.762 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.762 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.763 DEBUG:boto:Method: GET 2022-09-13T07:39:40.763 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.764 DEBUG:boto:Data: 2022-09-13T07:39:40.764 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.765 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.765 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.766 DEBUG:boto:Params: {} 2022-09-13T07:39:40.766 DEBUG:boto:Token: None 2022-09-13T07:39:40.767 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.768 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e92e8d5039c88a71-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.772 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.773 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.774 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.774 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.775 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.776 DEBUG:boto:Method: GET 2022-09-13T07:39:40.776 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.777 DEBUG:boto:Data: 2022-09-13T07:39:40.777 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.778 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.778 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.779 DEBUG:boto:Params: {} 2022-09-13T07:39:40.779 DEBUG:boto:Token: None 2022-09-13T07:39:40.780 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.781 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4d825afc1c77019-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.786 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.787 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.788 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.788 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.789 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:40.789 DEBUG:boto:Data: 2022-09-13T07:39:40.790 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.790 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.791 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.791 DEBUG:boto:Params: {} 2022-09-13T07:39:40.792 DEBUG:boto:Token: None 2022-09-13T07:39:40.793 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj 2022-09-13T07:39:40.793 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ= 2022-09-13T07:39:40.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.798 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d9790a23cd366e3-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.799 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.800 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.800 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.801 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:40.801 DEBUG:boto:Data: 2022-09-13T07:39:40.802 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.802 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.803 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.803 DEBUG:boto:Params: {} 2022-09-13T07:39:40.804 DEBUG:boto:Token: None 2022-09-13T07:39:40.804 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj 2022-09-13T07:39:40.805 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ= 2022-09-13T07:39:40.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HUdhoFPdPbNHUZ2i4TFOYSKcrxQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.810 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fd4f1178aee0fc6-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:40.812 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.813 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.814 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.814 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.815 DEBUG:boto:Method: GET 2022-09-13T07:39:40.816 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.817 DEBUG:boto:Data: 2022-09-13T07:39:40.817 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.818 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.819 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.819 DEBUG:boto:Params: {} 2022-09-13T07:39:40.820 DEBUG:boto:Token: None 2022-09-13T07:39:40.821 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.821 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI= 2022-09-13T07:39:40.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI=', 'Content-Length': '0'} 2022-09-13T07:39:40.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc8fb3a8aa7c8a85-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.866 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.867 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.868 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.868 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.869 DEBUG:boto:Method: GET 2022-09-13T07:39:40.869 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.870 DEBUG:boto:Data: 2022-09-13T07:39:40.870 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.871 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.871 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.872 DEBUG:boto:Params: {} 2022-09-13T07:39:40.873 DEBUG:boto:Token: None 2022-09-13T07:39:40.873 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:40.874 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI= 2022-09-13T07:39:40.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ISBepZc8ZXgFP1sIQv8v9HtqweI=', 'Content-Length': '0'} 2022-09-13T07:39:40.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b326bee29d0a92d-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.878 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.879 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.880 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.881 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.882 DEBUG:boto:Method: GET 2022-09-13T07:39:40.883 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.884 DEBUG:boto:Data: 2022-09-13T07:39:40.885 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.885 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.886 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.887 DEBUG:boto:Params: {} 2022-09-13T07:39:40.887 DEBUG:boto:Token: None 2022-09-13T07:39:40.888 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.888 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000895f2ff56b10d5ea-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.893 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.894 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.894 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.895 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.895 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.896 DEBUG:boto:Method: GET 2022-09-13T07:39:40.896 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.897 DEBUG:boto:Data: 2022-09-13T07:39:40.897 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.898 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.899 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.899 DEBUG:boto:Params: {} 2022-09-13T07:39:40.901 DEBUG:boto:Token: None 2022-09-13T07:39:40.901 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.903 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a75e3b8099b2ad82-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.909 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:40.911 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.912 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.913 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.913 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.914 DEBUG:boto:Method: GET 2022-09-13T07:39:40.914 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.915 DEBUG:boto:Data: 2022-09-13T07:39:40.915 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.916 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.916 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.917 DEBUG:boto:Params: {} 2022-09-13T07:39:40.917 DEBUG:boto:Token: None 2022-09-13T07:39:40.918 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.919 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:E8CaltbYQ+LjpI7j7fhX0my94UY= 2022-09-13T07:39:40.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:E8CaltbYQ+LjpI7j7fhX0my94UY=', 'Content-Length': '0'} 2022-09-13T07:39:40.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b818fd9c0102ed1e-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.922 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.923 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.923 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.924 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.924 DEBUG:boto:Method: GET 2022-09-13T07:39:40.925 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.926 DEBUG:boto:Data: 2022-09-13T07:39:40.926 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.927 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.927 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.928 DEBUG:boto:Params: {} 2022-09-13T07:39:40.928 DEBUG:boto:Token: None 2022-09-13T07:39:40.929 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.929 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:E8CaltbYQ+LjpI7j7fhX0my94UY= 2022-09-13T07:39:40.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:E8CaltbYQ+LjpI7j7fhX0my94UY=', 'Content-Length': '0'} 2022-09-13T07:39:40.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000258dce626b9ab9ca-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.933 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.933 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.934 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.934 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.935 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.935 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.936 DEBUG:boto:Data: 2022-09-13T07:39:40.936 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.937 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.937 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.938 DEBUG:boto:Params: {} 2022-09-13T07:39:40.938 DEBUG:boto:Token: None 2022-09-13T07:39:40.939 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.940 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:V0vmeZyn6nxvIwklXCeDpR5UXMo= 2022-09-13T07:39:40.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:V0vmeZyn6nxvIwklXCeDpR5UXMo=', 'Content-Length': '0'} 2022-09-13T07:39:40.942 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b99a4d1b9c18913-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.943 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.943 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.944 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.944 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.945 DEBUG:boto:Method: HEAD 2022-09-13T07:39:40.946 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.946 DEBUG:boto:Data: 2022-09-13T07:39:40.947 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.947 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.948 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.948 DEBUG:boto:Params: {} 2022-09-13T07:39:40.949 DEBUG:boto:Token: None 2022-09-13T07:39:40.949 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.950 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:V0vmeZyn6nxvIwklXCeDpR5UXMo= 2022-09-13T07:39:40.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:V0vmeZyn6nxvIwklXCeDpR5UXMo=', 'Content-Length': '0'} 2022-09-13T07:39:40.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f382c9c45115c69-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:40.953 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.954 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.955 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.955 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.956 DEBUG:boto:Method: GET 2022-09-13T07:39:40.956 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.957 DEBUG:boto:Data: 2022-09-13T07:39:40.957 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.958 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.958 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.959 DEBUG:boto:Params: {} 2022-09-13T07:39:40.959 DEBUG:boto:Token: None 2022-09-13T07:39:40.960 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.960 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:E8CaltbYQ+LjpI7j7fhX0my94UY= 2022-09-13T07:39:40.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:E8CaltbYQ+LjpI7j7fhX0my94UY=', 'Content-Length': '0'} 2022-09-13T07:39:40.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002052331567e72784-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.964 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.964 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:40.965 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.965 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.966 DEBUG:boto:Method: GET 2022-09-13T07:39:40.967 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.967 DEBUG:boto:Data: 2022-09-13T07:39:40.968 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.968 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.969 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.969 DEBUG:boto:Params: {} 2022-09-13T07:39:40.970 DEBUG:boto:Token: None 2022-09-13T07:39:40.970 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:40.971 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:E8CaltbYQ+LjpI7j7fhX0my94UY= 2022-09-13T07:39:40.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:E8CaltbYQ+LjpI7j7fhX0my94UY=', 'Content-Length': '0'} 2022-09-13T07:39:40.974 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b5acd345f418c62-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.975 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.975 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.976 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.976 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.977 DEBUG:boto:Method: GET 2022-09-13T07:39:40.977 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.978 DEBUG:boto:Data: 2022-09-13T07:39:40.978 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.979 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:40.979 DEBUG:boto:Port: 8001 2022-09-13T07:39:40.980 DEBUG:boto:Params: {} 2022-09-13T07:39:40.980 DEBUG:boto:Token: None 2022-09-13T07:39:40.981 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.981 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9a86d6e9ab2ded6-00632033bc-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.986 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.986 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:40.987 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:40.987 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.988 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.989 DEBUG:boto:Method: GET 2022-09-13T07:39:40.989 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:40.990 DEBUG:boto:Data: 2022-09-13T07:39:40.990 DEBUG:boto:Headers: {} 2022-09-13T07:39:40.991 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:40.991 DEBUG:boto:Port: 8000 2022-09-13T07:39:40.992 DEBUG:boto:Params: {} 2022-09-13T07:39:40.992 DEBUG:boto:Token: None 2022-09-13T07:39:40.993 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:40 GMT /yxvvuh-19/ 2022-09-13T07:39:40.993 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ= 2022-09-13T07:39:40.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Uc8FKx0tHr8BCTet08sePT4y4VQ=', 'Content-Length': '0'} 2022-09-13T07:39:40.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029e6a66e23d71412-00632033bc-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:40.998 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:40.999 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:40.999 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.000 DEBUG:boto:Method: HEAD 2022-09-13T07:39:41.000 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:41.001 DEBUG:boto:Data: 2022-09-13T07:39:41.001 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.002 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.002 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.003 DEBUG:boto:Params: {} 2022-09-13T07:39:41.003 DEBUG:boto:Token: None 2022-09-13T07:39:41.004 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj 2022-09-13T07:39:41.004 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss= 2022-09-13T07:39:41.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss=', 'Content-Length': '0'} 2022-09-13T07:39:41.009 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c4deeaecfab4cbd-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.009 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.010 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.010 DEBUG:boto:Method: HEAD 2022-09-13T07:39:41.011 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:41.011 DEBUG:boto:Data: 2022-09-13T07:39:41.012 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.012 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.013 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.013 DEBUG:boto:Params: {} 2022-09-13T07:39:41.014 DEBUG:boto:Token: None 2022-09-13T07:39:41.015 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj 2022-09-13T07:39:41.015 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss= 2022-09-13T07:39:41.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss=', 'Content-Length': '0'} 2022-09-13T07:39:41.019 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e20ee4857741d774-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:41.020 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.021 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.022 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.022 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.023 DEBUG:boto:Method: GET 2022-09-13T07:39:41.023 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.024 DEBUG:boto:Data: 2022-09-13T07:39:41.024 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.025 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.026 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.027 DEBUG:boto:Params: {} 2022-09-13T07:39:41.028 DEBUG:boto:Token: None 2022-09-13T07:39:41.029 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.030 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc= 2022-09-13T07:39:41.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc=', 'Content-Length': '0'} 2022-09-13T07:39:41.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064821d0822792ba6-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.035 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.036 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.037 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.038 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.039 DEBUG:boto:Method: GET 2022-09-13T07:39:41.040 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.040 DEBUG:boto:Data: 2022-09-13T07:39:41.041 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.042 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.043 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.044 DEBUG:boto:Params: {} 2022-09-13T07:39:41.045 DEBUG:boto:Token: None 2022-09-13T07:39:41.046 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.047 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc= 2022-09-13T07:39:41.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc=', 'Content-Length': '0'} 2022-09-13T07:39:41.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000985fecaaa1d85245-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.052 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:41.053 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:41.053 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.054 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.055 DEBUG:boto:Method: GET 2022-09-13T07:39:41.056 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.057 DEBUG:boto:Data: 2022-09-13T07:39:41.058 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.059 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.060 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.060 DEBUG:boto:Params: {} 2022-09-13T07:39:41.061 DEBUG:boto:Token: None 2022-09-13T07:39:41.062 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/ 2022-09-13T07:39:41.063 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k= 2022-09-13T07:39:41.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k=', 'Content-Length': '0'} 2022-09-13T07:39:41.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056f41f010e6fd913-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.083 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:41.084 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:41.085 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:41.086 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.086 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.087 DEBUG:boto:Method: GET 2022-09-13T07:39:41.087 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.088 DEBUG:boto:Data: 2022-09-13T07:39:41.088 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.089 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.090 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.090 DEBUG:boto:Params: {} 2022-09-13T07:39:41.091 DEBUG:boto:Token: None 2022-09-13T07:39:41.091 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/ 2022-09-13T07:39:41.092 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k= 2022-09-13T07:39:41.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k=', 'Content-Length': '0'} 2022-09-13T07:39:41.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008264cfd6c78128cb-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.144 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:41.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:41.148 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.149 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.150 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.151 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.152 DEBUG:boto:Method: GET 2022-09-13T07:39:41.153 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.153 DEBUG:boto:Data: 2022-09-13T07:39:41.154 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.154 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.155 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.156 DEBUG:boto:Params: {} 2022-09-13T07:39:41.157 DEBUG:boto:Token: None 2022-09-13T07:39:41.158 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.159 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7MjowUhuYngIdyyAVEERI0IYZZk= 2022-09-13T07:39:41.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7MjowUhuYngIdyyAVEERI0IYZZk=', 'Content-Length': '0'} 2022-09-13T07:39:41.163 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba09898a28266d8d-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.170 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.171 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.172 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.173 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.174 DEBUG:boto:Method: GET 2022-09-13T07:39:41.175 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.176 DEBUG:boto:Data: 2022-09-13T07:39:41.177 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.177 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.178 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.178 DEBUG:boto:Params: {} 2022-09-13T07:39:41.180 DEBUG:boto:Token: None 2022-09-13T07:39:41.181 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.182 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7MjowUhuYngIdyyAVEERI0IYZZk= 2022-09-13T07:39:41.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7MjowUhuYngIdyyAVEERI0IYZZk=', 'Content-Length': '0'} 2022-09-13T07:39:41.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057b79ebfd1518412-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.186 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.187 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.187 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.188 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.189 DEBUG:boto:Method: HEAD 2022-09-13T07:39:41.190 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.191 DEBUG:boto:Data: 2022-09-13T07:39:41.192 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.193 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.194 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.195 DEBUG:boto:Params: {} 2022-09-13T07:39:41.196 DEBUG:boto:Token: None 2022-09-13T07:39:41.197 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.198 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/8ofp5Q7Pjriir+izlv22NmSXw4= 2022-09-13T07:39:41.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/8ofp5Q7Pjriir+izlv22NmSXw4=', 'Content-Length': '0'} 2022-09-13T07:39:41.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6e8b98b363b71c1-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.206 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.207 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.208 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.208 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.209 DEBUG:boto:Method: HEAD 2022-09-13T07:39:41.209 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.210 DEBUG:boto:Data: 2022-09-13T07:39:41.210 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.211 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.211 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.212 DEBUG:boto:Params: {} 2022-09-13T07:39:41.213 DEBUG:boto:Token: None 2022-09-13T07:39:41.214 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.214 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/8ofp5Q7Pjriir+izlv22NmSXw4= 2022-09-13T07:39:41.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/8ofp5Q7Pjriir+izlv22NmSXw4=', 'Content-Length': '0'} 2022-09-13T07:39:41.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000afdbacf4bf03ddbf-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:41.223 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.225 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.226 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.228 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.230 DEBUG:boto:Method: GET 2022-09-13T07:39:41.231 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.234 DEBUG:boto:Data: 2022-09-13T07:39:41.235 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.236 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.237 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.238 DEBUG:boto:Params: {} 2022-09-13T07:39:41.239 DEBUG:boto:Token: None 2022-09-13T07:39:41.240 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.241 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7MjowUhuYngIdyyAVEERI0IYZZk= 2022-09-13T07:39:41.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7MjowUhuYngIdyyAVEERI0IYZZk=', 'Content-Length': '0'} 2022-09-13T07:39:41.249 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002988f9e4197895ce-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.290 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.291 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.292 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.292 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.293 DEBUG:boto:Method: GET 2022-09-13T07:39:41.293 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.294 DEBUG:boto:Data: 2022-09-13T07:39:41.294 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.295 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.296 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.296 DEBUG:boto:Params: {} 2022-09-13T07:39:41.297 DEBUG:boto:Token: None 2022-09-13T07:39:41.298 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:41.299 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7MjowUhuYngIdyyAVEERI0IYZZk= 2022-09-13T07:39:41.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7MjowUhuYngIdyyAVEERI0IYZZk=', 'Content-Length': '0'} 2022-09-13T07:39:41.304 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005eed08273bb1f9e2-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.306 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:41.307 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:41.308 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.309 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.310 DEBUG:boto:Method: GET 2022-09-13T07:39:41.311 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.312 DEBUG:boto:Data: 2022-09-13T07:39:41.313 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.314 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.315 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.316 DEBUG:boto:Params: {} 2022-09-13T07:39:41.317 DEBUG:boto:Token: None 2022-09-13T07:39:41.319 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/ 2022-09-13T07:39:41.320 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k= 2022-09-13T07:39:41.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k=', 'Content-Length': '0'} 2022-09-13T07:39:41.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009377df6577f9e6a5-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.331 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:41.332 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:41.333 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:41.334 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.336 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.337 DEBUG:boto:Method: GET 2022-09-13T07:39:41.338 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.339 DEBUG:boto:Data: 2022-09-13T07:39:41.340 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.341 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.342 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.343 DEBUG:boto:Params: {} 2022-09-13T07:39:41.344 DEBUG:boto:Token: None 2022-09-13T07:39:41.345 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/ 2022-09-13T07:39:41.346 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k= 2022-09-13T07:39:41.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k=', 'Content-Length': '0'} 2022-09-13T07:39:41.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040485b799bbae627-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.356 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:41.358 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.359 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.361 DEBUG:boto:Method: HEAD 2022-09-13T07:39:41.361 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:41.362 DEBUG:boto:Data: 2022-09-13T07:39:41.363 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.363 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.364 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.364 DEBUG:boto:Params: {} 2022-09-13T07:39:41.365 DEBUG:boto:Token: None 2022-09-13T07:39:41.366 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj 2022-09-13T07:39:41.366 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss= 2022-09-13T07:39:41.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss=', 'Content-Length': '0'} 2022-09-13T07:39:41.375 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1d70db8b432c88f-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.376 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.377 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.378 DEBUG:boto:Method: HEAD 2022-09-13T07:39:41.379 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:41.380 DEBUG:boto:Data: 2022-09-13T07:39:41.381 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.382 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.383 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.384 DEBUG:boto:Params: {} 2022-09-13T07:39:41.385 DEBUG:boto:Token: None 2022-09-13T07:39:41.386 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj 2022-09-13T07:39:41.387 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss= 2022-09-13T07:39:41.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss=', 'Content-Length': '0'} 2022-09-13T07:39:41.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095c7dfa2933931a6-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:41.396 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.396 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.397 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.397 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.398 DEBUG:boto:Method: GET 2022-09-13T07:39:41.398 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.399 DEBUG:boto:Data: 2022-09-13T07:39:41.399 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.400 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.400 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.401 DEBUG:boto:Params: {} 2022-09-13T07:39:41.402 DEBUG:boto:Token: None 2022-09-13T07:39:41.403 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.404 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc= 2022-09-13T07:39:41.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc=', 'Content-Length': '0'} 2022-09-13T07:39:41.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf015adae7e5256a-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.410 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.411 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.412 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.413 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.413 DEBUG:boto:Method: GET 2022-09-13T07:39:41.414 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.414 DEBUG:boto:Data: 2022-09-13T07:39:41.415 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.416 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.416 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.417 DEBUG:boto:Params: {} 2022-09-13T07:39:41.418 DEBUG:boto:Token: None 2022-09-13T07:39:41.418 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.419 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc= 2022-09-13T07:39:41.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc=', 'Content-Length': '0'} 2022-09-13T07:39:41.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044c9ce40a6e1ed4c-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.423 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:41.424 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:41.425 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.426 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.427 DEBUG:boto:Method: GET 2022-09-13T07:39:41.428 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.429 DEBUG:boto:Data: 2022-09-13T07:39:41.430 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.431 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.432 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.433 DEBUG:boto:Params: {} 2022-09-13T07:39:41.434 DEBUG:boto:Token: None 2022-09-13T07:39:41.435 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/ 2022-09-13T07:39:41.436 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k= 2022-09-13T07:39:41.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k=', 'Content-Length': '0'} 2022-09-13T07:39:41.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045a8e41a8f8ccd35-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.443 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:41.444 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:41.445 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:41.447 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.449 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.450 DEBUG:boto:Method: GET 2022-09-13T07:39:41.451 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.451 DEBUG:boto:Data: 2022-09-13T07:39:41.452 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.453 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.454 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.455 DEBUG:boto:Params: {} 2022-09-13T07:39:41.456 DEBUG:boto:Token: None 2022-09-13T07:39:41.457 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/ 2022-09-13T07:39:41.458 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k= 2022-09-13T07:39:41.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k=', 'Content-Length': '0'} 2022-09-13T07:39:41.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c0c8ff5a94df489-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.464 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:41.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:41.466 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.467 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.468 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.468 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.470 DEBUG:boto:Method: GET 2022-09-13T07:39:41.470 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.471 DEBUG:boto:Data: 2022-09-13T07:39:41.472 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.472 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.473 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.473 DEBUG:boto:Params: {} 2022-09-13T07:39:41.474 DEBUG:boto:Token: None 2022-09-13T07:39:41.475 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.476 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UBVR+x7Dj5YeNYzF+jr6m7yZK6M= 2022-09-13T07:39:41.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UBVR+x7Dj5YeNYzF+jr6m7yZK6M=', 'Content-Length': '0'} 2022-09-13T07:39:41.480 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056e44dba71c64722-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.522 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.525 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.530 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.531 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.532 DEBUG:boto:Method: GET 2022-09-13T07:39:41.533 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.533 DEBUG:boto:Data: 2022-09-13T07:39:41.534 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.534 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.535 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.535 DEBUG:boto:Params: {} 2022-09-13T07:39:41.536 DEBUG:boto:Token: None 2022-09-13T07:39:41.537 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.538 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UBVR+x7Dj5YeNYzF+jr6m7yZK6M= 2022-09-13T07:39:41.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UBVR+x7Dj5YeNYzF+jr6m7yZK6M=', 'Content-Length': '0'} 2022-09-13T07:39:41.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa5e4a55ff4e3acd-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.542 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.542 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.543 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.543 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.544 DEBUG:boto:Method: HEAD 2022-09-13T07:39:41.544 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.545 DEBUG:boto:Data: 2022-09-13T07:39:41.545 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.546 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.546 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.547 DEBUG:boto:Params: {} 2022-09-13T07:39:41.547 DEBUG:boto:Token: None 2022-09-13T07:39:41.548 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.548 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lVKh5B1ONPW0DVFL08aTVv/J9Uc= 2022-09-13T07:39:41.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lVKh5B1ONPW0DVFL08aTVv/J9Uc=', 'Content-Length': '0'} 2022-09-13T07:39:41.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f37369236e2d5a9d-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.557 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.558 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.559 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.561 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.562 DEBUG:boto:Method: HEAD 2022-09-13T07:39:41.563 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.564 DEBUG:boto:Data: 2022-09-13T07:39:41.565 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.566 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.567 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.568 DEBUG:boto:Params: {} 2022-09-13T07:39:41.570 DEBUG:boto:Token: None 2022-09-13T07:39:41.571 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.572 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lVKh5B1ONPW0DVFL08aTVv/J9Uc= 2022-09-13T07:39:41.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lVKh5B1ONPW0DVFL08aTVv/J9Uc=', 'Content-Length': '0'} 2022-09-13T07:39:41.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a5cc5edb81a6de6-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:41.578 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.579 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.579 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.580 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.581 DEBUG:boto:Method: GET 2022-09-13T07:39:41.582 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.582 DEBUG:boto:Data: 2022-09-13T07:39:41.583 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.583 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.584 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.584 DEBUG:boto:Params: {} 2022-09-13T07:39:41.585 DEBUG:boto:Token: None 2022-09-13T07:39:41.585 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.586 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UBVR+x7Dj5YeNYzF+jr6m7yZK6M= 2022-09-13T07:39:41.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UBVR+x7Dj5YeNYzF+jr6m7yZK6M=', 'Content-Length': '0'} 2022-09-13T07:39:41.589 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d281e769d2fb8b0e-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.590 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.591 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.592 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.593 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.595 DEBUG:boto:Method: GET 2022-09-13T07:39:41.596 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.597 DEBUG:boto:Data: 2022-09-13T07:39:41.598 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.599 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.600 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.601 DEBUG:boto:Params: {} 2022-09-13T07:39:41.603 DEBUG:boto:Token: None 2022-09-13T07:39:41.604 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:41.605 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UBVR+x7Dj5YeNYzF+jr6m7yZK6M= 2022-09-13T07:39:41.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UBVR+x7Dj5YeNYzF+jr6m7yZK6M=', 'Content-Length': '0'} 2022-09-13T07:39:41.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1158bb176b99cc3-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.610 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:41.610 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:41.611 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.611 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.612 DEBUG:boto:Method: GET 2022-09-13T07:39:41.612 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.613 DEBUG:boto:Data: 2022-09-13T07:39:41.613 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.614 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.614 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.615 DEBUG:boto:Params: {} 2022-09-13T07:39:41.616 DEBUG:boto:Token: None 2022-09-13T07:39:41.616 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/ 2022-09-13T07:39:41.617 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k= 2022-09-13T07:39:41.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k=', 'Content-Length': '0'} 2022-09-13T07:39:41.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5ebdcf8a843c0b6-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.621 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:41.622 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:41.623 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:41.624 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.624 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.625 DEBUG:boto:Method: GET 2022-09-13T07:39:41.625 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.626 DEBUG:boto:Data: 2022-09-13T07:39:41.626 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.627 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.627 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.628 DEBUG:boto:Params: {} 2022-09-13T07:39:41.628 DEBUG:boto:Token: None 2022-09-13T07:39:41.629 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/ 2022-09-13T07:39:41.630 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k= 2022-09-13T07:39:41.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k=', 'Content-Length': '0'} 2022-09-13T07:39:41.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044d40521e3eaed60-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.634 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:41.635 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.636 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.636 DEBUG:boto:Method: HEAD 2022-09-13T07:39:41.637 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:41.638 DEBUG:boto:Data: 2022-09-13T07:39:41.638 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.639 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.639 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.640 DEBUG:boto:Params: {} 2022-09-13T07:39:41.640 DEBUG:boto:Token: None 2022-09-13T07:39:41.641 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj 2022-09-13T07:39:41.642 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss= 2022-09-13T07:39:41.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss=', 'Content-Length': '0'} 2022-09-13T07:39:41.646 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000152a2b58325f2871-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.647 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.648 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.648 DEBUG:boto:Method: HEAD 2022-09-13T07:39:41.649 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:41.649 DEBUG:boto:Data: 2022-09-13T07:39:41.650 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.650 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.651 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.652 DEBUG:boto:Params: {} 2022-09-13T07:39:41.653 DEBUG:boto:Token: None 2022-09-13T07:39:41.654 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj 2022-09-13T07:39:41.654 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss= 2022-09-13T07:39:41.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss=', 'Content-Length': '0'} 2022-09-13T07:39:41.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b392e00a1133375-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:41.661 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.662 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.663 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.664 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.664 DEBUG:boto:Method: GET 2022-09-13T07:39:41.665 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.666 DEBUG:boto:Data: 2022-09-13T07:39:41.666 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.667 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.668 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.668 DEBUG:boto:Params: {} 2022-09-13T07:39:41.669 DEBUG:boto:Token: None 2022-09-13T07:39:41.670 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.670 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc= 2022-09-13T07:39:41.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc=', 'Content-Length': '0'} 2022-09-13T07:39:41.673 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028ea423eb39dced4-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.714 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.718 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.718 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.719 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.719 DEBUG:boto:Method: GET 2022-09-13T07:39:41.720 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.720 DEBUG:boto:Data: 2022-09-13T07:39:41.721 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.721 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.722 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.724 DEBUG:boto:Params: {} 2022-09-13T07:39:41.724 DEBUG:boto:Token: None 2022-09-13T07:39:41.725 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.726 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc= 2022-09-13T07:39:41.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc=', 'Content-Length': '0'} 2022-09-13T07:39:41.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000000bc2f2082871a6-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.730 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:41.730 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:41.731 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.732 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.732 DEBUG:boto:Method: GET 2022-09-13T07:39:41.733 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.733 DEBUG:boto:Data: 2022-09-13T07:39:41.734 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.735 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.735 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.736 DEBUG:boto:Params: {} 2022-09-13T07:39:41.737 DEBUG:boto:Token: None 2022-09-13T07:39:41.737 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/ 2022-09-13T07:39:41.738 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k= 2022-09-13T07:39:41.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k=', 'Content-Length': '0'} 2022-09-13T07:39:41.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed384bf66128de37-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.742 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:41.743 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:41.744 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:41.744 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.745 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.745 DEBUG:boto:Method: GET 2022-09-13T07:39:41.746 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.747 DEBUG:boto:Data: 2022-09-13T07:39:41.747 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.748 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.748 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.749 DEBUG:boto:Params: {} 2022-09-13T07:39:41.749 DEBUG:boto:Token: None 2022-09-13T07:39:41.750 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/ 2022-09-13T07:39:41.751 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k= 2022-09-13T07:39:41.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k=', 'Content-Length': '0'} 2022-09-13T07:39:41.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ada563a653cfc0ef-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.756 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:41.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:41.757 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.758 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.758 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.759 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.759 DEBUG:boto:Method: GET 2022-09-13T07:39:41.761 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.761 DEBUG:boto:Data: 2022-09-13T07:39:41.762 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.762 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.763 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.763 DEBUG:boto:Params: {} 2022-09-13T07:39:41.764 DEBUG:boto:Token: None 2022-09-13T07:39:41.764 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.765 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KOq5wOpyPH/nwfYdFH9C1vNTeR4= 2022-09-13T07:39:41.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KOq5wOpyPH/nwfYdFH9C1vNTeR4=', 'Content-Length': '0'} 2022-09-13T07:39:41.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008aabe9780e2c3e78-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.769 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.770 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.770 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.771 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.771 DEBUG:boto:Method: GET 2022-09-13T07:39:41.772 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.773 DEBUG:boto:Data: 2022-09-13T07:39:41.773 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.774 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.774 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.775 DEBUG:boto:Params: {} 2022-09-13T07:39:41.776 DEBUG:boto:Token: None 2022-09-13T07:39:41.776 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.777 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KOq5wOpyPH/nwfYdFH9C1vNTeR4= 2022-09-13T07:39:41.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KOq5wOpyPH/nwfYdFH9C1vNTeR4=', 'Content-Length': '0'} 2022-09-13T07:39:41.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e68487c8f3ced9b0-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.822 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.823 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.825 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.826 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.827 DEBUG:boto:Method: HEAD 2022-09-13T07:39:41.828 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.829 DEBUG:boto:Data: 2022-09-13T07:39:41.830 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.831 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.832 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.833 DEBUG:boto:Params: {} 2022-09-13T07:39:41.834 DEBUG:boto:Token: None 2022-09-13T07:39:41.834 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.835 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:P+7JUmrRIhTNub3tCs5taIJwwBY= 2022-09-13T07:39:41.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:P+7JUmrRIhTNub3tCs5taIJwwBY=', 'Content-Length': '0'} 2022-09-13T07:39:41.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d75b39a9ef056cd-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.838 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.839 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.839 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.840 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.841 DEBUG:boto:Method: HEAD 2022-09-13T07:39:41.841 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.842 DEBUG:boto:Data: 2022-09-13T07:39:41.842 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.843 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.843 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.844 DEBUG:boto:Params: {} 2022-09-13T07:39:41.845 DEBUG:boto:Token: None 2022-09-13T07:39:41.845 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.846 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:P+7JUmrRIhTNub3tCs5taIJwwBY= 2022-09-13T07:39:41.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:P+7JUmrRIhTNub3tCs5taIJwwBY=', 'Content-Length': '0'} 2022-09-13T07:39:41.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000959304b177e894c6-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:41.850 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.851 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.852 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.853 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.854 DEBUG:boto:Method: GET 2022-09-13T07:39:41.855 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.856 DEBUG:boto:Data: 2022-09-13T07:39:41.856 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.857 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.858 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.858 DEBUG:boto:Params: {} 2022-09-13T07:39:41.859 DEBUG:boto:Token: None 2022-09-13T07:39:41.859 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.860 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KOq5wOpyPH/nwfYdFH9C1vNTeR4= 2022-09-13T07:39:41.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KOq5wOpyPH/nwfYdFH9C1vNTeR4=', 'Content-Length': '0'} 2022-09-13T07:39:41.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d635355d25d85d7-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.864 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.865 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.865 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.866 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.866 DEBUG:boto:Method: GET 2022-09-13T07:39:41.867 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.868 DEBUG:boto:Data: 2022-09-13T07:39:41.868 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.869 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.869 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.870 DEBUG:boto:Params: {} 2022-09-13T07:39:41.870 DEBUG:boto:Token: None 2022-09-13T07:39:41.871 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:41.871 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KOq5wOpyPH/nwfYdFH9C1vNTeR4= 2022-09-13T07:39:41.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KOq5wOpyPH/nwfYdFH9C1vNTeR4=', 'Content-Length': '0'} 2022-09-13T07:39:41.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000377922310583b15c-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.875 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:41.876 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:41.876 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.877 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.877 DEBUG:boto:Method: GET 2022-09-13T07:39:41.878 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.878 DEBUG:boto:Data: 2022-09-13T07:39:41.879 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.879 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.880 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.880 DEBUG:boto:Params: {} 2022-09-13T07:39:41.881 DEBUG:boto:Token: None 2022-09-13T07:39:41.881 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/ 2022-09-13T07:39:41.882 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k= 2022-09-13T07:39:41.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k=', 'Content-Length': '0'} 2022-09-13T07:39:41.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6fb94a681cce419-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.887 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:41.887 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:41.888 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:41.889 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.889 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.890 DEBUG:boto:Method: GET 2022-09-13T07:39:41.890 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:41.891 DEBUG:boto:Data: 2022-09-13T07:39:41.891 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.892 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.892 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.893 DEBUG:boto:Params: {} 2022-09-13T07:39:41.893 DEBUG:boto:Token: None 2022-09-13T07:39:41.894 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/ 2022-09-13T07:39:41.895 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k= 2022-09-13T07:39:41.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gIkj9I1BaVjrLOkXm/5mz6efY9k=', 'Content-Length': '0'} 2022-09-13T07:39:41.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ce36b38999c3d4b-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.900 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:41.901 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.902 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.903 DEBUG:boto:Method: HEAD 2022-09-13T07:39:41.903 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:41.904 DEBUG:boto:Data: 2022-09-13T07:39:41.904 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.905 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.905 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.906 DEBUG:boto:Params: {} 2022-09-13T07:39:41.907 DEBUG:boto:Token: None 2022-09-13T07:39:41.907 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj 2022-09-13T07:39:41.908 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss= 2022-09-13T07:39:41.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss=', 'Content-Length': '0'} 2022-09-13T07:39:41.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003743962e0f63af47-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.915 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.916 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.917 DEBUG:boto:Method: HEAD 2022-09-13T07:39:41.917 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:41.918 DEBUG:boto:Data: 2022-09-13T07:39:41.918 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.919 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.920 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.920 DEBUG:boto:Params: {} 2022-09-13T07:39:41.921 DEBUG:boto:Token: None 2022-09-13T07:39:41.922 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj 2022-09-13T07:39:41.922 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss= 2022-09-13T07:39:41.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tMDZTvlQLLKxAjudVYKMZvlZXss=', 'Content-Length': '0'} 2022-09-13T07:39:41.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b19503949600129-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:41.929 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.930 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.930 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.931 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.932 DEBUG:boto:Method: GET 2022-09-13T07:39:41.933 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.934 DEBUG:boto:Data: 2022-09-13T07:39:41.934 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.935 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:41.936 DEBUG:boto:Port: 8001 2022-09-13T07:39:41.938 DEBUG:boto:Params: {} 2022-09-13T07:39:41.939 DEBUG:boto:Token: None 2022-09-13T07:39:41.940 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.941 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc= 2022-09-13T07:39:41.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc=', 'Content-Length': '0'} 2022-09-13T07:39:41.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1ae46d28aa1fa62-00632033bd-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:41.947 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:41.948 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:41.948 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.949 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.950 DEBUG:boto:Method: GET 2022-09-13T07:39:41.950 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.951 DEBUG:boto:Data: 2022-09-13T07:39:41.952 DEBUG:boto:Headers: {} 2022-09-13T07:39:41.952 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:41.953 DEBUG:boto:Port: 8000 2022-09-13T07:39:41.953 DEBUG:boto:Params: {} 2022-09-13T07:39:41.954 DEBUG:boto:Token: None 2022-09-13T07:39:41.955 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:41 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:41.956 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc= 2022-09-13T07:39:41.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:41 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Rs/DGMmyXH0Za7IUAXuwoy6eOqc=', 'Content-Length': '0'} 2022-09-13T07:39:41.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000751202800e7df953-00632033bd-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.004 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:42.005 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:42.005 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.006 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.007 DEBUG:boto:Method: GET 2022-09-13T07:39:42.007 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.008 DEBUG:boto:Data: 2022-09-13T07:39:42.008 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.009 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.010 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.010 DEBUG:boto:Params: {} 2022-09-13T07:39:42.011 DEBUG:boto:Token: None 2022-09-13T07:39:42.011 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/ 2022-09-13T07:39:42.012 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A= 2022-09-13T07:39:42.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A=', 'Content-Length': '0'} 2022-09-13T07:39:42.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003594749a7497cd4-00632033be-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.017 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:42.018 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:42.018 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:42.019 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.019 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.020 DEBUG:boto:Method: GET 2022-09-13T07:39:42.020 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.021 DEBUG:boto:Data: 2022-09-13T07:39:42.021 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.022 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.022 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.023 DEBUG:boto:Params: {} 2022-09-13T07:39:42.023 DEBUG:boto:Token: None 2022-09-13T07:39:42.024 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/ 2022-09-13T07:39:42.025 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A= 2022-09-13T07:39:42.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A=', 'Content-Length': '0'} 2022-09-13T07:39:42.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b902316f19f0e20b-00632033be-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.029 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:42.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:42.030 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.031 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.031 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.032 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.032 DEBUG:boto:Method: GET 2022-09-13T07:39:42.033 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.033 DEBUG:boto:Data: 2022-09-13T07:39:42.034 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.034 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.035 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.036 DEBUG:boto:Params: {} 2022-09-13T07:39:42.036 DEBUG:boto:Token: None 2022-09-13T07:39:42.037 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.038 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cPF6gs9SBZW2P14fKeBOd6gzq0I= 2022-09-13T07:39:42.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cPF6gs9SBZW2P14fKeBOd6gzq0I=', 'Content-Length': '0'} 2022-09-13T07:39:42.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011d15ce4c90e5d1f-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.043 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.043 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.044 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.044 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.045 DEBUG:boto:Method: GET 2022-09-13T07:39:42.046 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.046 DEBUG:boto:Data: 2022-09-13T07:39:42.059 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.060 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.060 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.061 DEBUG:boto:Params: {} 2022-09-13T07:39:42.062 DEBUG:boto:Token: None 2022-09-13T07:39:42.062 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.063 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cPF6gs9SBZW2P14fKeBOd6gzq0I= 2022-09-13T07:39:42.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cPF6gs9SBZW2P14fKeBOd6gzq0I=', 'Content-Length': '0'} 2022-09-13T07:39:42.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031f1e4b6b7f636c7-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.069 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.070 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.070 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.071 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.072 DEBUG:boto:Method: HEAD 2022-09-13T07:39:42.073 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.073 DEBUG:boto:Data: 2022-09-13T07:39:42.074 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.074 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.075 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.075 DEBUG:boto:Params: {} 2022-09-13T07:39:42.076 DEBUG:boto:Token: None 2022-09-13T07:39:42.076 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.077 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:evzyXIvP2J3t3Ah7JeIUcSwT5Vw= 2022-09-13T07:39:42.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:evzyXIvP2J3t3Ah7JeIUcSwT5Vw=', 'Content-Length': '0'} 2022-09-13T07:39:42.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003eb833e763cf3b0c-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.081 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.082 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.082 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.083 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.083 DEBUG:boto:Method: HEAD 2022-09-13T07:39:42.084 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.084 DEBUG:boto:Data: 2022-09-13T07:39:42.085 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.085 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.086 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.086 DEBUG:boto:Params: {} 2022-09-13T07:39:42.087 DEBUG:boto:Token: None 2022-09-13T07:39:42.087 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.088 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:evzyXIvP2J3t3Ah7JeIUcSwT5Vw= 2022-09-13T07:39:42.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:evzyXIvP2J3t3Ah7JeIUcSwT5Vw=', 'Content-Length': '0'} 2022-09-13T07:39:42.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075864eccb3276f0d-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:42.092 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.093 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.093 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.094 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.094 DEBUG:boto:Method: GET 2022-09-13T07:39:42.095 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.095 DEBUG:boto:Data: 2022-09-13T07:39:42.096 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.096 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.097 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.097 DEBUG:boto:Params: {} 2022-09-13T07:39:42.098 DEBUG:boto:Token: None 2022-09-13T07:39:42.099 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.099 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cPF6gs9SBZW2P14fKeBOd6gzq0I= 2022-09-13T07:39:42.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cPF6gs9SBZW2P14fKeBOd6gzq0I=', 'Content-Length': '0'} 2022-09-13T07:39:42.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063955640b44f9e88-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.103 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.103 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.104 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.104 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.105 DEBUG:boto:Method: GET 2022-09-13T07:39:42.106 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.106 DEBUG:boto:Data: 2022-09-13T07:39:42.107 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.107 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.108 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.108 DEBUG:boto:Params: {} 2022-09-13T07:39:42.109 DEBUG:boto:Token: None 2022-09-13T07:39:42.109 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:42.110 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cPF6gs9SBZW2P14fKeBOd6gzq0I= 2022-09-13T07:39:42.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cPF6gs9SBZW2P14fKeBOd6gzq0I=', 'Content-Length': '0'} 2022-09-13T07:39:42.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000205642247cc82ede-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.158 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:42.159 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:42.159 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.160 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.160 DEBUG:boto:Method: GET 2022-09-13T07:39:42.161 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.162 DEBUG:boto:Data: 2022-09-13T07:39:42.162 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.163 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.163 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.164 DEBUG:boto:Params: {} 2022-09-13T07:39:42.164 DEBUG:boto:Token: None 2022-09-13T07:39:42.165 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/ 2022-09-13T07:39:42.166 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A= 2022-09-13T07:39:42.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A=', 'Content-Length': '0'} 2022-09-13T07:39:42.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067cb367afda06343-00632033be-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.172 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:42.173 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:42.174 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:42.175 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.175 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.176 DEBUG:boto:Method: GET 2022-09-13T07:39:42.177 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.177 DEBUG:boto:Data: 2022-09-13T07:39:42.178 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.179 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.179 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.180 DEBUG:boto:Params: {} 2022-09-13T07:39:42.181 DEBUG:boto:Token: None 2022-09-13T07:39:42.181 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/ 2022-09-13T07:39:42.182 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A= 2022-09-13T07:39:42.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A=', 'Content-Length': '0'} 2022-09-13T07:39:42.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091022097014d10ee-00632033be-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.187 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:42.188 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.188 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.189 DEBUG:boto:Method: HEAD 2022-09-13T07:39:42.189 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:42.190 DEBUG:boto:Data: 2022-09-13T07:39:42.190 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.191 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.191 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.192 DEBUG:boto:Params: {} 2022-09-13T07:39:42.192 DEBUG:boto:Token: None 2022-09-13T07:39:42.193 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj 2022-09-13T07:39:42.194 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MaSu6m3RFaukQjG2Pumt0HOCq9M= 2022-09-13T07:39:42.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MaSu6m3RFaukQjG2Pumt0HOCq9M=', 'Content-Length': '0'} 2022-09-13T07:39:42.198 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9e38a43cb89e195-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.199 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.199 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.200 DEBUG:boto:Method: HEAD 2022-09-13T07:39:42.200 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:42.201 DEBUG:boto:Data: 2022-09-13T07:39:42.201 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.202 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.203 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.203 DEBUG:boto:Params: {} 2022-09-13T07:39:42.204 DEBUG:boto:Token: None 2022-09-13T07:39:42.204 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj 2022-09-13T07:39:42.205 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MaSu6m3RFaukQjG2Pumt0HOCq9M= 2022-09-13T07:39:42.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MaSu6m3RFaukQjG2Pumt0HOCq9M=', 'Content-Length': '0'} 2022-09-13T07:39:42.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045c966d7bbd94ef8-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:42.214 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.215 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.215 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.216 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.216 DEBUG:boto:Method: GET 2022-09-13T07:39:42.217 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.217 DEBUG:boto:Data: 2022-09-13T07:39:42.218 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.218 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.219 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.219 DEBUG:boto:Params: {} 2022-09-13T07:39:42.220 DEBUG:boto:Token: None 2022-09-13T07:39:42.221 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.221 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ry7aTWZ+kJAkgCMVFPdz9IS1HWU= 2022-09-13T07:39:42.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ry7aTWZ+kJAkgCMVFPdz9IS1HWU=', 'Content-Length': '0'} 2022-09-13T07:39:42.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf6d5c9dcc047137-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.266 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.267 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.268 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.269 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.270 DEBUG:boto:Method: GET 2022-09-13T07:39:42.271 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.272 DEBUG:boto:Data: 2022-09-13T07:39:42.273 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.274 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.275 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.276 DEBUG:boto:Params: {} 2022-09-13T07:39:42.277 DEBUG:boto:Token: None 2022-09-13T07:39:42.278 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.279 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ry7aTWZ+kJAkgCMVFPdz9IS1HWU= 2022-09-13T07:39:42.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ry7aTWZ+kJAkgCMVFPdz9IS1HWU=', 'Content-Length': '0'} 2022-09-13T07:39:42.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000647643d90ee95704-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.285 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:42.286 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:42.287 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.288 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.289 DEBUG:boto:Method: GET 2022-09-13T07:39:42.290 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.291 DEBUG:boto:Data: 2022-09-13T07:39:42.291 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.293 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.293 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.294 DEBUG:boto:Params: {} 2022-09-13T07:39:42.296 DEBUG:boto:Token: None 2022-09-13T07:39:42.297 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/ 2022-09-13T07:39:42.298 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A= 2022-09-13T07:39:42.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A=', 'Content-Length': '0'} 2022-09-13T07:39:42.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5cd7dc334c53347-00632033be-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.304 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:42.306 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:42.306 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:42.308 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.309 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.309 DEBUG:boto:Method: GET 2022-09-13T07:39:42.310 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.311 DEBUG:boto:Data: 2022-09-13T07:39:42.312 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.313 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.314 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.315 DEBUG:boto:Params: {} 2022-09-13T07:39:42.317 DEBUG:boto:Token: None 2022-09-13T07:39:42.318 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/ 2022-09-13T07:39:42.318 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A= 2022-09-13T07:39:42.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A=', 'Content-Length': '0'} 2022-09-13T07:39:42.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4943e098e9b91a9-00632033be-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.323 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:42.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:42.326 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.327 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.328 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.328 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.329 DEBUG:boto:Method: GET 2022-09-13T07:39:42.330 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.330 DEBUG:boto:Data: 2022-09-13T07:39:42.331 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.331 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.332 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.333 DEBUG:boto:Params: {} 2022-09-13T07:39:42.333 DEBUG:boto:Token: None 2022-09-13T07:39:42.334 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.335 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VYqNizudBeY3007t7G1yAOMlCzs= 2022-09-13T07:39:42.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VYqNizudBeY3007t7G1yAOMlCzs=', 'Content-Length': '0'} 2022-09-13T07:39:42.338 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000087780fc998cc8d2-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.339 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.339 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.340 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.340 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.341 DEBUG:boto:Method: GET 2022-09-13T07:39:42.342 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.343 DEBUG:boto:Data: 2022-09-13T07:39:42.343 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.344 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.345 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.345 DEBUG:boto:Params: {} 2022-09-13T07:39:42.346 DEBUG:boto:Token: None 2022-09-13T07:39:42.346 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.347 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VYqNizudBeY3007t7G1yAOMlCzs= 2022-09-13T07:39:42.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VYqNizudBeY3007t7G1yAOMlCzs=', 'Content-Length': '0'} 2022-09-13T07:39:42.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0878697c11026bc-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.394 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.395 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.396 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.396 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.397 DEBUG:boto:Method: HEAD 2022-09-13T07:39:42.397 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.398 DEBUG:boto:Data: 2022-09-13T07:39:42.398 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.399 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.399 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.400 DEBUG:boto:Params: {} 2022-09-13T07:39:42.401 DEBUG:boto:Token: None 2022-09-13T07:39:42.401 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.402 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4gScB+Hp5TbcqAuaH+Ypq9s+CNo= 2022-09-13T07:39:42.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4gScB+Hp5TbcqAuaH+Ypq9s+CNo=', 'Content-Length': '0'} 2022-09-13T07:39:42.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000941397d6d45f4c94-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.405 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.406 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.406 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.407 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.407 DEBUG:boto:Method: HEAD 2022-09-13T07:39:42.408 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.409 DEBUG:boto:Data: 2022-09-13T07:39:42.410 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.411 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.411 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.412 DEBUG:boto:Params: {} 2022-09-13T07:39:42.413 DEBUG:boto:Token: None 2022-09-13T07:39:42.415 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.416 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4gScB+Hp5TbcqAuaH+Ypq9s+CNo= 2022-09-13T07:39:42.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4gScB+Hp5TbcqAuaH+Ypq9s+CNo=', 'Content-Length': '0'} 2022-09-13T07:39:42.419 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de0daf66b0b9dd08-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:42.422 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.424 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.425 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.426 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.427 DEBUG:boto:Method: GET 2022-09-13T07:39:42.429 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.430 DEBUG:boto:Data: 2022-09-13T07:39:42.430 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.431 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.432 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.432 DEBUG:boto:Params: {} 2022-09-13T07:39:42.433 DEBUG:boto:Token: None 2022-09-13T07:39:42.434 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.435 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VYqNizudBeY3007t7G1yAOMlCzs= 2022-09-13T07:39:42.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VYqNizudBeY3007t7G1yAOMlCzs=', 'Content-Length': '0'} 2022-09-13T07:39:42.439 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce70b85e5af10b0f-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.440 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.440 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.441 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.442 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.443 DEBUG:boto:Method: GET 2022-09-13T07:39:42.444 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.445 DEBUG:boto:Data: 2022-09-13T07:39:42.446 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.447 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.448 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.449 DEBUG:boto:Params: {} 2022-09-13T07:39:42.450 DEBUG:boto:Token: None 2022-09-13T07:39:42.450 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:42.451 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VYqNizudBeY3007t7G1yAOMlCzs= 2022-09-13T07:39:42.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VYqNizudBeY3007t7G1yAOMlCzs=', 'Content-Length': '0'} 2022-09-13T07:39:42.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a43b3939c78bc80-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.456 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:42.457 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:42.458 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.459 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.459 DEBUG:boto:Method: GET 2022-09-13T07:39:42.461 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.462 DEBUG:boto:Data: 2022-09-13T07:39:42.463 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.463 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.464 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.465 DEBUG:boto:Params: {} 2022-09-13T07:39:42.466 DEBUG:boto:Token: None 2022-09-13T07:39:42.467 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/ 2022-09-13T07:39:42.467 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A= 2022-09-13T07:39:42.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A=', 'Content-Length': '0'} 2022-09-13T07:39:42.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e966a6ceff2442f-00632033be-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.514 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:42.515 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:42.516 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:42.517 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.517 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.518 DEBUG:boto:Method: GET 2022-09-13T07:39:42.518 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.519 DEBUG:boto:Data: 2022-09-13T07:39:42.520 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.521 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.522 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.523 DEBUG:boto:Params: {} 2022-09-13T07:39:42.524 DEBUG:boto:Token: None 2022-09-13T07:39:42.525 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/ 2022-09-13T07:39:42.526 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A= 2022-09-13T07:39:42.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A=', 'Content-Length': '0'} 2022-09-13T07:39:42.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006952ac0fbff35f41-00632033be-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.531 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:42.532 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.533 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.533 DEBUG:boto:Method: HEAD 2022-09-13T07:39:42.534 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:42.534 DEBUG:boto:Data: 2022-09-13T07:39:42.535 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.535 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.536 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.537 DEBUG:boto:Params: {} 2022-09-13T07:39:42.537 DEBUG:boto:Token: None 2022-09-13T07:39:42.538 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj 2022-09-13T07:39:42.538 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MaSu6m3RFaukQjG2Pumt0HOCq9M= 2022-09-13T07:39:42.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MaSu6m3RFaukQjG2Pumt0HOCq9M=', 'Content-Length': '0'} 2022-09-13T07:39:42.543 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000994565c15123d731-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.544 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.544 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.545 DEBUG:boto:Method: HEAD 2022-09-13T07:39:42.545 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:42.546 DEBUG:boto:Data: 2022-09-13T07:39:42.547 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.547 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.548 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.548 DEBUG:boto:Params: {} 2022-09-13T07:39:42.549 DEBUG:boto:Token: None 2022-09-13T07:39:42.550 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj 2022-09-13T07:39:42.551 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MaSu6m3RFaukQjG2Pumt0HOCq9M= 2022-09-13T07:39:42.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MaSu6m3RFaukQjG2Pumt0HOCq9M=', 'Content-Length': '0'} 2022-09-13T07:39:42.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe7f20b89e81f7a9-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:42.557 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.557 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.558 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.558 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.559 DEBUG:boto:Method: GET 2022-09-13T07:39:42.560 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.560 DEBUG:boto:Data: 2022-09-13T07:39:42.561 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.562 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.562 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.563 DEBUG:boto:Params: {} 2022-09-13T07:39:42.564 DEBUG:boto:Token: None 2022-09-13T07:39:42.564 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.565 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ry7aTWZ+kJAkgCMVFPdz9IS1HWU= 2022-09-13T07:39:42.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ry7aTWZ+kJAkgCMVFPdz9IS1HWU=', 'Content-Length': '0'} 2022-09-13T07:39:42.568 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005edf19919694821d-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.569 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.570 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.571 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.571 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.572 DEBUG:boto:Method: GET 2022-09-13T07:39:42.573 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.574 DEBUG:boto:Data: 2022-09-13T07:39:42.575 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.576 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.577 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.578 DEBUG:boto:Params: {} 2022-09-13T07:39:42.578 DEBUG:boto:Token: None 2022-09-13T07:39:42.579 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.580 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ry7aTWZ+kJAkgCMVFPdz9IS1HWU= 2022-09-13T07:39:42.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ry7aTWZ+kJAkgCMVFPdz9IS1HWU=', 'Content-Length': '0'} 2022-09-13T07:39:42.582 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030b60fcabeb626f1-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.626 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:42.627 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:42.627 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.628 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.628 DEBUG:boto:Method: GET 2022-09-13T07:39:42.629 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.630 DEBUG:boto:Data: 2022-09-13T07:39:42.630 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.631 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.631 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.632 DEBUG:boto:Params: {} 2022-09-13T07:39:42.632 DEBUG:boto:Token: None 2022-09-13T07:39:42.633 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/ 2022-09-13T07:39:42.634 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A= 2022-09-13T07:39:42.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A=', 'Content-Length': '0'} 2022-09-13T07:39:42.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044af4b178e1a0857-00632033be-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.639 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:42.640 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:42.640 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:42.641 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.641 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.642 DEBUG:boto:Method: GET 2022-09-13T07:39:42.642 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.643 DEBUG:boto:Data: 2022-09-13T07:39:42.643 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.644 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.644 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.645 DEBUG:boto:Params: {} 2022-09-13T07:39:42.645 DEBUG:boto:Token: None 2022-09-13T07:39:42.646 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/ 2022-09-13T07:39:42.647 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A= 2022-09-13T07:39:42.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A=', 'Content-Length': '0'} 2022-09-13T07:39:42.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5e1f78594fd946e-00632033be-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.651 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:42.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:42.653 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.653 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.654 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.655 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.655 DEBUG:boto:Method: GET 2022-09-13T07:39:42.656 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.656 DEBUG:boto:Data: 2022-09-13T07:39:42.657 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.657 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.658 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.658 DEBUG:boto:Params: {} 2022-09-13T07:39:42.659 DEBUG:boto:Token: None 2022-09-13T07:39:42.660 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.660 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ue9x9G5SBI3l0yMuSKKE27FtonA= 2022-09-13T07:39:42.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ue9x9G5SBI3l0yMuSKKE27FtonA=', 'Content-Length': '0'} 2022-09-13T07:39:42.663 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000001dde9d518cf226-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.706 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.707 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.708 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.709 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.710 DEBUG:boto:Method: GET 2022-09-13T07:39:42.711 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.711 DEBUG:boto:Data: 2022-09-13T07:39:42.712 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.712 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.713 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.713 DEBUG:boto:Params: {} 2022-09-13T07:39:42.714 DEBUG:boto:Token: None 2022-09-13T07:39:42.715 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.715 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ue9x9G5SBI3l0yMuSKKE27FtonA= 2022-09-13T07:39:42.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ue9x9G5SBI3l0yMuSKKE27FtonA=', 'Content-Length': '0'} 2022-09-13T07:39:42.718 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000afc961275cf00d6b-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.719 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.719 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.720 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.720 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.721 DEBUG:boto:Method: HEAD 2022-09-13T07:39:42.722 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.723 DEBUG:boto:Data: 2022-09-13T07:39:42.723 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.724 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.725 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.726 DEBUG:boto:Params: {} 2022-09-13T07:39:42.727 DEBUG:boto:Token: None 2022-09-13T07:39:42.728 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.729 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rXY1EVAQBy/egBXab47zKQoXpSo= 2022-09-13T07:39:42.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rXY1EVAQBy/egBXab47zKQoXpSo=', 'Content-Length': '0'} 2022-09-13T07:39:42.733 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e455305c98630c95-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.734 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.735 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.736 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.737 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.737 DEBUG:boto:Method: HEAD 2022-09-13T07:39:42.738 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.739 DEBUG:boto:Data: 2022-09-13T07:39:42.740 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.741 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.742 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.743 DEBUG:boto:Params: {} 2022-09-13T07:39:42.744 DEBUG:boto:Token: None 2022-09-13T07:39:42.745 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.746 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rXY1EVAQBy/egBXab47zKQoXpSo= 2022-09-13T07:39:42.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rXY1EVAQBy/egBXab47zKQoXpSo=', 'Content-Length': '0'} 2022-09-13T07:39:42.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b67b4637d4a739d-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:42.752 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.752 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.753 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.754 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.755 DEBUG:boto:Method: GET 2022-09-13T07:39:42.756 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.757 DEBUG:boto:Data: 2022-09-13T07:39:42.758 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.758 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.759 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.759 DEBUG:boto:Params: {} 2022-09-13T07:39:42.760 DEBUG:boto:Token: None 2022-09-13T07:39:42.760 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.761 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ue9x9G5SBI3l0yMuSKKE27FtonA= 2022-09-13T07:39:42.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ue9x9G5SBI3l0yMuSKKE27FtonA=', 'Content-Length': '0'} 2022-09-13T07:39:42.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f3bcd18213e3add-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.764 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.765 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.765 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.766 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.766 DEBUG:boto:Method: GET 2022-09-13T07:39:42.767 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.767 DEBUG:boto:Data: 2022-09-13T07:39:42.768 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.768 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.769 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.770 DEBUG:boto:Params: {} 2022-09-13T07:39:42.770 DEBUG:boto:Token: None 2022-09-13T07:39:42.771 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:42.771 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ue9x9G5SBI3l0yMuSKKE27FtonA= 2022-09-13T07:39:42.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ue9x9G5SBI3l0yMuSKKE27FtonA=', 'Content-Length': '0'} 2022-09-13T07:39:42.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000986f2c53f3d4342e-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.775 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:42.776 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:42.776 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.777 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.777 DEBUG:boto:Method: GET 2022-09-13T07:39:42.778 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.778 DEBUG:boto:Data: 2022-09-13T07:39:42.779 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.780 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.780 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.781 DEBUG:boto:Params: {} 2022-09-13T07:39:42.782 DEBUG:boto:Token: None 2022-09-13T07:39:42.783 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/ 2022-09-13T07:39:42.784 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A= 2022-09-13T07:39:42.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A=', 'Content-Length': '0'} 2022-09-13T07:39:42.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099e5ed3da4dfb753-00632033be-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.790 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:42.791 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:42.792 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:42.792 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.793 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.794 DEBUG:boto:Method: GET 2022-09-13T07:39:42.794 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.795 DEBUG:boto:Data: 2022-09-13T07:39:42.795 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.796 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.796 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.797 DEBUG:boto:Params: {} 2022-09-13T07:39:42.797 DEBUG:boto:Token: None 2022-09-13T07:39:42.798 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/ 2022-09-13T07:39:42.798 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A= 2022-09-13T07:39:42.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A=', 'Content-Length': '0'} 2022-09-13T07:39:42.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068097812eb302173-00632033be-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.846 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:42.848 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.849 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.849 DEBUG:boto:Method: HEAD 2022-09-13T07:39:42.850 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:42.851 DEBUG:boto:Data: 2022-09-13T07:39:42.852 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.853 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.853 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.854 DEBUG:boto:Params: {} 2022-09-13T07:39:42.856 DEBUG:boto:Token: None 2022-09-13T07:39:42.857 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj 2022-09-13T07:39:42.857 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MaSu6m3RFaukQjG2Pumt0HOCq9M= 2022-09-13T07:39:42.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MaSu6m3RFaukQjG2Pumt0HOCq9M=', 'Content-Length': '0'} 2022-09-13T07:39:42.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096672f8fc72cd571-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.863 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.863 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.864 DEBUG:boto:Method: HEAD 2022-09-13T07:39:42.865 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:42.865 DEBUG:boto:Data: 2022-09-13T07:39:42.866 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.866 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.867 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.867 DEBUG:boto:Params: {} 2022-09-13T07:39:42.868 DEBUG:boto:Token: None 2022-09-13T07:39:42.868 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj 2022-09-13T07:39:42.869 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MaSu6m3RFaukQjG2Pumt0HOCq9M= 2022-09-13T07:39:42.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MaSu6m3RFaukQjG2Pumt0HOCq9M=', 'Content-Length': '0'} 2022-09-13T07:39:42.873 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014454e8ed33c38c7-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:42.875 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.875 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.876 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.876 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.877 DEBUG:boto:Method: GET 2022-09-13T07:39:42.877 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.878 DEBUG:boto:Data: 2022-09-13T07:39:42.878 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.879 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.879 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.880 DEBUG:boto:Params: {} 2022-09-13T07:39:42.880 DEBUG:boto:Token: None 2022-09-13T07:39:42.881 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.882 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ry7aTWZ+kJAkgCMVFPdz9IS1HWU= 2022-09-13T07:39:42.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ry7aTWZ+kJAkgCMVFPdz9IS1HWU=', 'Content-Length': '0'} 2022-09-13T07:39:42.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000386683c1c30f193a-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.885 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.886 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.887 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.887 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.888 DEBUG:boto:Method: GET 2022-09-13T07:39:42.888 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.889 DEBUG:boto:Data: 2022-09-13T07:39:42.889 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.890 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.890 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.891 DEBUG:boto:Params: {} 2022-09-13T07:39:42.892 DEBUG:boto:Token: None 2022-09-13T07:39:42.892 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:42.893 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ry7aTWZ+kJAkgCMVFPdz9IS1HWU= 2022-09-13T07:39:42.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ry7aTWZ+kJAkgCMVFPdz9IS1HWU=', 'Content-Length': '0'} 2022-09-13T07:39:42.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a14918ed07005cd-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.898 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:42.899 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:42.900 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.901 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.902 DEBUG:boto:Method: GET 2022-09-13T07:39:42.903 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.904 DEBUG:boto:Data: 2022-09-13T07:39:42.905 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.906 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.907 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.907 DEBUG:boto:Params: {} 2022-09-13T07:39:42.908 DEBUG:boto:Token: None 2022-09-13T07:39:42.909 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/ 2022-09-13T07:39:42.909 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A= 2022-09-13T07:39:42.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A=', 'Content-Length': '0'} 2022-09-13T07:39:42.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b30aa27c326124a-00632033be-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.914 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:42.916 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:42.917 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:42.918 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.919 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.920 DEBUG:boto:Method: GET 2022-09-13T07:39:42.921 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:42.922 DEBUG:boto:Data: 2022-09-13T07:39:42.923 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.924 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.925 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.926 DEBUG:boto:Params: {} 2022-09-13T07:39:42.927 DEBUG:boto:Token: None 2022-09-13T07:39:42.928 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/ 2022-09-13T07:39:42.930 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A= 2022-09-13T07:39:42.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8Cxu+qq3qYSY2AL8QrjaDwfIX5A=', 'Content-Length': '0'} 2022-09-13T07:39:42.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab93ee5d4bbf3b10-00632033be-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.936 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:42.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:42.939 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.940 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.941 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:42.942 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:42.943 DEBUG:boto:Method: GET 2022-09-13T07:39:42.944 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:42.945 DEBUG:boto:Data: 2022-09-13T07:39:42.946 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.947 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.948 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.949 DEBUG:boto:Params: {} 2022-09-13T07:39:42.950 DEBUG:boto:Token: None 2022-09-13T07:39:42.951 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:42.952 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ua/Rh0STiMnfGI9u7ePw+QXEbTU= 2022-09-13T07:39:42.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ua/Rh0STiMnfGI9u7ePw+QXEbTU=', 'Content-Length': '0'} 2022-09-13T07:39:42.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070116729ccd40b56-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.958 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.959 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.960 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:42.961 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:42.963 DEBUG:boto:Method: GET 2022-09-13T07:39:42.964 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:42.965 DEBUG:boto:Data: 2022-09-13T07:39:42.966 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.967 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:42.968 DEBUG:boto:Port: 8000 2022-09-13T07:39:42.970 DEBUG:boto:Params: {} 2022-09-13T07:39:42.971 DEBUG:boto:Token: None 2022-09-13T07:39:42.972 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:42.973 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ua/Rh0STiMnfGI9u7ePw+QXEbTU= 2022-09-13T07:39:42.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ua/Rh0STiMnfGI9u7ePw+QXEbTU=', 'Content-Length': '0'} 2022-09-13T07:39:42.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ef1561eb0d68c6a-00632033be-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.979 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.980 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.981 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:42.982 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:42.983 DEBUG:boto:Method: HEAD 2022-09-13T07:39:42.984 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:42.985 DEBUG:boto:Data: 2022-09-13T07:39:42.986 DEBUG:boto:Headers: {} 2022-09-13T07:39:42.986 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:42.987 DEBUG:boto:Port: 8001 2022-09-13T07:39:42.988 DEBUG:boto:Params: {} 2022-09-13T07:39:42.989 DEBUG:boto:Token: None 2022-09-13T07:39:42.990 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:42 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:42.991 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fApf5C1UCrKYgrfafLwXwU7ZjiA= 2022-09-13T07:39:42.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:42 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fApf5C1UCrKYgrfafLwXwU7ZjiA=', 'Content-Length': '0'} 2022-09-13T07:39:42.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2d4839b70c96212-00632033be-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:42.996 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:42.997 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:42.998 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:42.999 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:42.999 DEBUG:boto:Method: HEAD 2022-09-13T07:39:43.000 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:43.000 DEBUG:boto:Data: 2022-09-13T07:39:43.001 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.001 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:43.002 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.002 DEBUG:boto:Params: {} 2022-09-13T07:39:43.003 DEBUG:boto:Token: None 2022-09-13T07:39:43.003 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:43.004 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hrqpN9+vB0GUAvx/tW7JtGzkrNk= 2022-09-13T07:39:43.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hrqpN9+vB0GUAvx/tW7JtGzkrNk=', 'Content-Length': '0'} 2022-09-13T07:39:43.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015166c2911d49621-00632033bf-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:43.008 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.008 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.009 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:43.009 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:43.010 DEBUG:boto:Method: GET 2022-09-13T07:39:43.010 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:43.011 DEBUG:boto:Data: 2022-09-13T07:39:43.012 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.012 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.013 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.013 DEBUG:boto:Params: {} 2022-09-13T07:39:43.014 DEBUG:boto:Token: None 2022-09-13T07:39:43.014 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:43.015 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MpXLEyMEFNI1KtT3+KlKWJtNwUY= 2022-09-13T07:39:43.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MpXLEyMEFNI1KtT3+KlKWJtNwUY=', 'Content-Length': '0'} 2022-09-13T07:39:43.027 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd261d1b2dec4ba8-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.074 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.075 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.075 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:43.076 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:43.076 DEBUG:boto:Method: GET 2022-09-13T07:39:43.077 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:43.078 DEBUG:boto:Data: 2022-09-13T07:39:43.079 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.080 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:43.081 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.082 DEBUG:boto:Params: {} 2022-09-13T07:39:43.083 DEBUG:boto:Token: None 2022-09-13T07:39:43.084 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:43.085 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MpXLEyMEFNI1KtT3+KlKWJtNwUY= 2022-09-13T07:39:43.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MpXLEyMEFNI1KtT3+KlKWJtNwUY=', 'Content-Length': '0'} 2022-09-13T07:39:43.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011a5da9f78d1245b-00632033bf-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.089 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.090 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.090 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.091 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.092 DEBUG:boto:Method: GET 2022-09-13T07:39:43.092 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.093 DEBUG:boto:Data: 2022-09-13T07:39:43.093 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.094 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.094 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.095 DEBUG:boto:Params: {} 2022-09-13T07:39:43.095 DEBUG:boto:Token: None 2022-09-13T07:39:43.096 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/ 2022-09-13T07:39:43.097 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U= 2022-09-13T07:39:43.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U=', 'Content-Length': '0'} 2022-09-13T07:39:43.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061b4af0db3d02edd-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.101 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.102 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.102 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.103 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.103 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.104 DEBUG:boto:Method: GET 2022-09-13T07:39:43.104 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.105 DEBUG:boto:Data: 2022-09-13T07:39:43.105 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.106 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:43.106 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.107 DEBUG:boto:Params: {} 2022-09-13T07:39:43.107 DEBUG:boto:Token: None 2022-09-13T07:39:43.108 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/ 2022-09-13T07:39:43.109 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U= 2022-09-13T07:39:43.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U=', 'Content-Length': '0'} 2022-09-13T07:39:43.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000188e941e0f028582-00632033bf-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.114 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.115 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.116 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.117 DEBUG:boto:Method: HEAD 2022-09-13T07:39:43.118 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:43.118 DEBUG:boto:Data: 2022-09-13T07:39:43.119 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.120 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.121 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.121 DEBUG:boto:Params: {} 2022-09-13T07:39:43.122 DEBUG:boto:Token: None 2022-09-13T07:39:43.123 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj 2022-09-13T07:39:43.123 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t9Fn7c86AQG7c16SMIZIB1GaAbM= 2022-09-13T07:39:43.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t9Fn7c86AQG7c16SMIZIB1GaAbM=', 'Content-Length': '0'} 2022-09-13T07:39:43.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdbfe8e10da78582-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.128 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.129 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.129 DEBUG:boto:Method: HEAD 2022-09-13T07:39:43.130 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:43.131 DEBUG:boto:Data: 2022-09-13T07:39:43.131 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.132 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:43.132 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.132 DEBUG:boto:Params: {} 2022-09-13T07:39:43.133 DEBUG:boto:Token: None 2022-09-13T07:39:43.134 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj 2022-09-13T07:39:43.134 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t9Fn7c86AQG7c16SMIZIB1GaAbM= 2022-09-13T07:39:43.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t9Fn7c86AQG7c16SMIZIB1GaAbM=', 'Content-Length': '0'} 2022-09-13T07:39:43.141 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d6c9de376336ade-00632033bf-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:43.143 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.144 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.145 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.146 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.147 DEBUG:boto:Method: GET 2022-09-13T07:39:43.148 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.149 DEBUG:boto:Data: 2022-09-13T07:39:43.150 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.151 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.152 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.153 DEBUG:boto:Params: {} 2022-09-13T07:39:43.154 DEBUG:boto:Token: None 2022-09-13T07:39:43.155 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.156 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY= 2022-09-13T07:39:43.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY=', 'Content-Length': '0'} 2022-09-13T07:39:43.159 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7abaf2df247f6e2-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.160 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.161 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.162 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.163 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.164 DEBUG:boto:Method: GET 2022-09-13T07:39:43.165 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.165 DEBUG:boto:Data: 2022-09-13T07:39:43.166 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.166 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:43.167 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.168 DEBUG:boto:Params: {} 2022-09-13T07:39:43.169 DEBUG:boto:Token: None 2022-09-13T07:39:43.170 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.171 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY= 2022-09-13T07:39:43.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY=', 'Content-Length': '0'} 2022-09-13T07:39:43.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025cc4a5cf71bbff3-00632033bf-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.174 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.175 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.176 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.177 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.178 DEBUG:boto:Method: GET 2022-09-13T07:39:43.179 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.179 DEBUG:boto:Data: 2022-09-13T07:39:43.180 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.180 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.181 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.181 DEBUG:boto:Params: {} 2022-09-13T07:39:43.182 DEBUG:boto:Token: None 2022-09-13T07:39:43.183 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/ 2022-09-13T07:39:43.183 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U= 2022-09-13T07:39:43.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U=', 'Content-Length': '0'} 2022-09-13T07:39:43.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5fbe82fa2a4c7cc-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.188 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.189 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.189 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.190 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.191 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.191 DEBUG:boto:Method: GET 2022-09-13T07:39:43.192 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.192 DEBUG:boto:Data: 2022-09-13T07:39:43.193 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.193 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:43.194 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.194 DEBUG:boto:Params: {} 2022-09-13T07:39:43.195 DEBUG:boto:Token: None 2022-09-13T07:39:43.195 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/ 2022-09-13T07:39:43.196 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U= 2022-09-13T07:39:43.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U=', 'Content-Length': '0'} 2022-09-13T07:39:43.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000577df47c3e85c1d4-00632033bf-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.201 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.202 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-19 zones={test-zone3, test-zone1} 2022-09-13T07:39:43.202 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-19 zones={test-zone3, test-zone2} 2022-09-13T07:39:43.203 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.203 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.204 DEBUG:boto:Method: HEAD 2022-09-13T07:39:43.204 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:39:43.205 DEBUG:boto:Data: 2022-09-13T07:39:43.206 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.207 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.208 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.210 DEBUG:boto:Params: {} 2022-09-13T07:39:43.211 DEBUG:boto:Token: None 2022-09-13T07:39:43.211 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/ 2022-09-13T07:39:43.212 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wY5aTjNnY6LdMaHcNY9VKs2Erzo= 2022-09-13T07:39:43.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wY5aTjNnY6LdMaHcNY9VKs2Erzo=', 'Content-Length': '0'} 2022-09-13T07:39:43.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004ac004addde7b79-00632033bf-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.216 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.216 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.217 DEBUG:boto:Method: HEAD 2022-09-13T07:39:43.217 DEBUG:boto:Path: /yxvvuh-19/ 2022-09-13T07:39:43.218 DEBUG:boto:Data: 2022-09-13T07:39:43.218 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.219 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.219 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.220 DEBUG:boto:Params: {} 2022-09-13T07:39:43.221 DEBUG:boto:Token: None 2022-09-13T07:39:43.221 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/ 2022-09-13T07:39:43.222 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wY5aTjNnY6LdMaHcNY9VKs2Erzo= 2022-09-13T07:39:43.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wY5aTjNnY6LdMaHcNY9VKs2Erzo=', 'Content-Length': '0'} 2022-09-13T07:39:43.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000324ff52b7592d7fa-00632033bf-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.228 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:39:43.229 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.229 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.230 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:43.231 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:43.232 DEBUG:boto:Method: GET 2022-09-13T07:39:43.232 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:43.233 DEBUG:boto:Data: 2022-09-13T07:39:43.234 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.235 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.236 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.236 DEBUG:boto:Params: {} 2022-09-13T07:39:43.237 DEBUG:boto:Token: None 2022-09-13T07:39:43.237 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/?versions 2022-09-13T07:39:43.238 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tR9axxwEn0F3D9OLMkdDpWoJx/I= 2022-09-13T07:39:43.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tR9axxwEn0F3D9OLMkdDpWoJx/I=', 'Content-Length': '0'} 2022-09-13T07:39:43.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f69048949adcc778-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.244 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEn7MftyiCP0pYMPcqn5sySJi4UkpaiZtrue2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmwBtbzZwRR4A0HEc28dvfcEWdzuIZDfalse2022-09-13T07:37:30.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7false2022-09-13T07:37:30.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTL9xVVVkIODKu69Lcj5HlhVjbyjGE3ffalse2022-09-13T07:37:30.515Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU.elqMxtYPda6ZsKEKDP833Q93.DHGxfalse2022-09-13T07:37:30.547Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji8qJKwt5yOnRkOjqtadujgkvccCo7V6false2022-09-13T07:37:30.592Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WBfalse2022-09-13T07:37:30.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtLDy1U4ZTxWriIUMx30zfLE7zPcVUt5false2022-09-13T07:37:30.424Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjulPqaKM9GsnZBkMYt.uabOf.VJ5P.5mfalse2022-09-13T07:37:30.465Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eAfalse2022-09-13T07:37:30.308Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnEgHn6dtpr-DjlqQutDDilFO8cCU9tVfalse2022-09-13T07:37:30.261Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzbAEfbqMIp9ReKOn2t8gGBeNLMCc26hfalse2022-09-13T07:37:30.341Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkKoYsAdAKf7lBJBYkG7TA7YHa0Qh7adfalse2022-09-13T07:37:30.152Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1false2022-09-13T07:37:30.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxwhfwIrEIbdFug3DCFDiVhftQi2CUm6false2022-09-13T07:37:30.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN1ijg0L2Sa1qSFcwgB7aYlf3fysHL.ofalse2022-09-13T07:37:30.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6pfalse2022-09-13T07:37:30.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjr3j0LS1bjhh8xQBCJCWMQPOkDqpqjuMfalse2022-09-13T07:37:30.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUFIYwFqhiCTkSFbmDIDHjWDiTsQToDGfalse2022-09-13T07:37:30.001Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjei7viyB1s5vEJjb-DgvRL3K6yeA94ngfalse2022-09-13T07:37:29.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHnhTbZbus4hq.TBh5.hecwSTeKVCaRfalse2022-09-13T07:37:30.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8bP-bkYz6JpHUV25JyCvc0fDB8aZwslfalse2022-09-13T07:37:29.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcPIo6H3tnk3IHLM4mBy4H8IXyB8CUPCfalse2022-09-13T07:37:29.842Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-false2022-09-13T07:37:29.815Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3VqUFgxLYLsisMjs5Rrmm15ZQudhjyfalse2022-09-13T07:37:29.688Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4false2022-09-13T07:37:29.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxshrAtab2xKYxtLaiLBmYWJxRHGcH2rfalse2022-09-13T07:37:29.770Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.Ifalse2022-09-13T07:37:29.573Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC-XByO2K9qUOxsnzB35XnbS7CZIHKxHfalse2022-09-13T07:37:29.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaErb35kohBMBuUIVZDyjKh5q.cT.bAJfalse2022-09-13T07:37:29.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.249 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:39:43.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.270 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:39:43.270 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.271 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.271 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:43.272 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:43.272 DEBUG:boto:Method: GET 2022-09-13T07:39:43.273 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:43.273 DEBUG:boto:Data: 2022-09-13T07:39:43.274 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.274 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.275 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.276 DEBUG:boto:Params: {} 2022-09-13T07:39:43.277 DEBUG:boto:Token: None 2022-09-13T07:39:43.278 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/?versions 2022-09-13T07:39:43.279 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tR9axxwEn0F3D9OLMkdDpWoJx/I= 2022-09-13T07:39:43.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tR9axxwEn0F3D9OLMkdDpWoJx/I=', 'Content-Length': '0'} 2022-09-13T07:39:43.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a54908d326a33cb8-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.284 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEn7MftyiCP0pYMPcqn5sySJi4UkpaiZtrue2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmwBtbzZwRR4A0HEc28dvfcEWdzuIZDfalse2022-09-13T07:37:30.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7false2022-09-13T07:37:30.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTL9xVVVkIODKu69Lcj5HlhVjbyjGE3ffalse2022-09-13T07:37:30.515Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU.elqMxtYPda6ZsKEKDP833Q93.DHGxfalse2022-09-13T07:37:30.547Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji8qJKwt5yOnRkOjqtadujgkvccCo7V6false2022-09-13T07:37:30.592Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WBfalse2022-09-13T07:37:30.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtLDy1U4ZTxWriIUMx30zfLE7zPcVUt5false2022-09-13T07:37:30.424Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjulPqaKM9GsnZBkMYt.uabOf.VJ5P.5mfalse2022-09-13T07:37:30.465Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eAfalse2022-09-13T07:37:30.308Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnEgHn6dtpr-DjlqQutDDilFO8cCU9tVfalse2022-09-13T07:37:30.261Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzbAEfbqMIp9ReKOn2t8gGBeNLMCc26hfalse2022-09-13T07:37:30.341Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkKoYsAdAKf7lBJBYkG7TA7YHa0Qh7adfalse2022-09-13T07:37:30.152Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1false2022-09-13T07:37:30.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxwhfwIrEIbdFug3DCFDiVhftQi2CUm6false2022-09-13T07:37:30.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN1ijg0L2Sa1qSFcwgB7aYlf3fysHL.ofalse2022-09-13T07:37:30.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6pfalse2022-09-13T07:37:30.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjr3j0LS1bjhh8xQBCJCWMQPOkDqpqjuMfalse2022-09-13T07:37:30.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUFIYwFqhiCTkSFbmDIDHjWDiTsQToDGfalse2022-09-13T07:37:30.001Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjei7viyB1s5vEJjb-DgvRL3K6yeA94ngfalse2022-09-13T07:37:29.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHnhTbZbus4hq.TBh5.hecwSTeKVCaRfalse2022-09-13T07:37:30.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8bP-bkYz6JpHUV25JyCvc0fDB8aZwslfalse2022-09-13T07:37:29.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcPIo6H3tnk3IHLM4mBy4H8IXyB8CUPCfalse2022-09-13T07:37:29.842Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-false2022-09-13T07:37:29.815Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3VqUFgxLYLsisMjs5Rrmm15ZQudhjyfalse2022-09-13T07:37:29.688Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4false2022-09-13T07:37:29.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxshrAtab2xKYxtLaiLBmYWJxRHGcH2rfalse2022-09-13T07:37:29.770Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.Ifalse2022-09-13T07:37:29.573Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC-XByO2K9qUOxsnzB35XnbS7CZIHKxHfalse2022-09-13T07:37:29.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaErb35kohBMBuUIVZDyjKh5q.cT.bAJfalse2022-09-13T07:37:29.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.287 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-13T07:39:43.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:39:43.305 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.305 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.306 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:43.306 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:43.307 DEBUG:boto:Method: GET 2022-09-13T07:39:43.308 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:43.308 DEBUG:boto:Data: 2022-09-13T07:39:43.309 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.309 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.310 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.310 DEBUG:boto:Params: {} 2022-09-13T07:39:43.311 DEBUG:boto:Token: None 2022-09-13T07:39:43.311 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/?versions 2022-09-13T07:39:43.312 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tR9axxwEn0F3D9OLMkdDpWoJx/I= 2022-09-13T07:39:43.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tR9axxwEn0F3D9OLMkdDpWoJx/I=', 'Content-Length': '0'} 2022-09-13T07:39:43.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6eff5072bc27381-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.317 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEn7MftyiCP0pYMPcqn5sySJi4UkpaiZtrue2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmwBtbzZwRR4A0HEc28dvfcEWdzuIZDfalse2022-09-13T07:37:30.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7false2022-09-13T07:37:30.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTL9xVVVkIODKu69Lcj5HlhVjbyjGE3ffalse2022-09-13T07:37:30.515Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU.elqMxtYPda6ZsKEKDP833Q93.DHGxfalse2022-09-13T07:37:30.547Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji8qJKwt5yOnRkOjqtadujgkvccCo7V6false2022-09-13T07:37:30.592Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WBfalse2022-09-13T07:37:30.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtLDy1U4ZTxWriIUMx30zfLE7zPcVUt5false2022-09-13T07:37:30.424Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjulPqaKM9GsnZBkMYt.uabOf.VJ5P.5mfalse2022-09-13T07:37:30.465Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eAfalse2022-09-13T07:37:30.308Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnEgHn6dtpr-DjlqQutDDilFO8cCU9tVfalse2022-09-13T07:37:30.261Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzbAEfbqMIp9ReKOn2t8gGBeNLMCc26hfalse2022-09-13T07:37:30.341Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkKoYsAdAKf7lBJBYkG7TA7YHa0Qh7adfalse2022-09-13T07:37:30.152Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1false2022-09-13T07:37:30.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxwhfwIrEIbdFug3DCFDiVhftQi2CUm6false2022-09-13T07:37:30.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN1ijg0L2Sa1qSFcwgB7aYlf3fysHL.ofalse2022-09-13T07:37:30.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6pfalse2022-09-13T07:37:30.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjr3j0LS1bjhh8xQBCJCWMQPOkDqpqjuMfalse2022-09-13T07:37:30.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUFIYwFqhiCTkSFbmDIDHjWDiTsQToDGfalse2022-09-13T07:37:30.001Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjei7viyB1s5vEJjb-DgvRL3K6yeA94ngfalse2022-09-13T07:37:29.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHnhTbZbus4hq.TBh5.hecwSTeKVCaRfalse2022-09-13T07:37:30.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8bP-bkYz6JpHUV25JyCvc0fDB8aZwslfalse2022-09-13T07:37:29.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcPIo6H3tnk3IHLM4mBy4H8IXyB8CUPCfalse2022-09-13T07:37:29.842Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-false2022-09-13T07:37:29.815Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3VqUFgxLYLsisMjs5Rrmm15ZQudhjyfalse2022-09-13T07:37:29.688Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4false2022-09-13T07:37:29.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxshrAtab2xKYxtLaiLBmYWJxRHGcH2rfalse2022-09-13T07:37:29.770Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.Ifalse2022-09-13T07:37:29.573Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC-XByO2K9qUOxsnzB35XnbS7CZIHKxHfalse2022-09-13T07:37:29.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaErb35kohBMBuUIVZDyjKh5q.cT.bAJfalse2022-09-13T07:37:29.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.319 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.319 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.320 DEBUG:boto:path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:43.320 DEBUG:boto:auth_path=/yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:43.321 DEBUG:boto:Method: GET 2022-09-13T07:39:43.322 DEBUG:boto:Path: /yxvvuh-19/?versions&max-keys=999 2022-09-13T07:39:43.322 DEBUG:boto:Data: 2022-09-13T07:39:43.323 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.323 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.324 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.324 DEBUG:boto:Params: {} 2022-09-13T07:39:43.325 DEBUG:boto:Token: None 2022-09-13T07:39:43.326 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/?versions 2022-09-13T07:39:43.326 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tR9axxwEn0F3D9OLMkdDpWoJx/I= 2022-09-13T07:39:43.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tR9axxwEn0F3D9OLMkdDpWoJx/I=', 'Content-Length': '0'} 2022-09-13T07:39:43.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ee307187a43f4b0-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.333 DEBUG:boto:b'yxvvuh-19999falsedummyFDANku1GQSwU5HbWax2gCLnve9-nkdktrue2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEn7MftyiCP0pYMPcqn5sySJi4UkpaiZtrue2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmwBtbzZwRR4A0HEc28dvfcEWdzuIZDfalse2022-09-13T07:37:30.687Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7false2022-09-13T07:37:30.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTL9xVVVkIODKu69Lcj5HlhVjbyjGE3ffalse2022-09-13T07:37:30.515Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU.elqMxtYPda6ZsKEKDP833Q93.DHGxfalse2022-09-13T07:37:30.547Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobji8qJKwt5yOnRkOjqtadujgkvccCo7V6false2022-09-13T07:37:30.592Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WBfalse2022-09-13T07:37:30.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtLDy1U4ZTxWriIUMx30zfLE7zPcVUt5false2022-09-13T07:37:30.424Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjulPqaKM9GsnZBkMYt.uabOf.VJ5P.5mfalse2022-09-13T07:37:30.465Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eAfalse2022-09-13T07:37:30.308Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnEgHn6dtpr-DjlqQutDDilFO8cCU9tVfalse2022-09-13T07:37:30.261Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzbAEfbqMIp9ReKOn2t8gGBeNLMCc26hfalse2022-09-13T07:37:30.341Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkKoYsAdAKf7lBJBYkG7TA7YHa0Qh7adfalse2022-09-13T07:37:30.152Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1false2022-09-13T07:37:30.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxwhfwIrEIbdFug3DCFDiVhftQi2CUm6false2022-09-13T07:37:30.219Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN1ijg0L2Sa1qSFcwgB7aYlf3fysHL.ofalse2022-09-13T07:37:30.059Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6pfalse2022-09-13T07:37:30.091Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjr3j0LS1bjhh8xQBCJCWMQPOkDqpqjuMfalse2022-09-13T07:37:30.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUFIYwFqhiCTkSFbmDIDHjWDiTsQToDGfalse2022-09-13T07:37:30.001Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjei7viyB1s5vEJjb-DgvRL3K6yeA94ngfalse2022-09-13T07:37:29.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHnhTbZbus4hq.TBh5.hecwSTeKVCaRfalse2022-09-13T07:37:30.028Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8bP-bkYz6JpHUV25JyCvc0fDB8aZwslfalse2022-09-13T07:37:29.909Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcPIo6H3tnk3IHLM4mBy4H8IXyB8CUPCfalse2022-09-13T07:37:29.842Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-false2022-09-13T07:37:29.815Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3VqUFgxLYLsisMjs5Rrmm15ZQudhjyfalse2022-09-13T07:37:29.688Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4false2022-09-13T07:37:29.728Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxshrAtab2xKYxtLaiLBmYWJxRHGcH2rfalse2022-09-13T07:37:29.770Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.Ifalse2022-09-13T07:37:29.573Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC-XByO2K9qUOxsnzB35XnbS7CZIHKxHfalse2022-09-13T07:37:29.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaErb35kohBMBuUIVZDyjKh5q.cT.bAJfalse2022-09-13T07:37:29.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:39:43.336 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:43.337 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:43.337 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.338 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.338 DEBUG:boto:Method: GET 2022-09-13T07:39:43.339 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.339 DEBUG:boto:Data: 2022-09-13T07:39:43.340 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.340 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.341 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.342 DEBUG:boto:Params: {} 2022-09-13T07:39:43.343 DEBUG:boto:Token: None 2022-09-13T07:39:43.344 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.345 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf01XdT30/wMo7t0LILI6DFvWCg= 2022-09-13T07:39:43.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf01XdT30/wMo7t0LILI6DFvWCg=', 'Content-Length': '0'} 2022-09-13T07:39:43.349 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000039c39ea8ccbd1666-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.350 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:43.351 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:43.352 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.353 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.353 DEBUG:boto:Method: GET 2022-09-13T07:39:43.354 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.354 DEBUG:boto:Data: 2022-09-13T07:39:43.355 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.355 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.356 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.356 DEBUG:boto:Params: {} 2022-09-13T07:39:43.357 DEBUG:boto:Token: None 2022-09-13T07:39:43.358 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.358 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf01XdT30/wMo7t0LILI6DFvWCg= 2022-09-13T07:39:43.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf01XdT30/wMo7t0LILI6DFvWCg=', 'Content-Length': '0'} 2022-09-13T07:39:43.361 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007f3fed5dfef381f0-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.362 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:43.362 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:43.363 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.363 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.364 DEBUG:boto:Method: HEAD 2022-09-13T07:39:43.364 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.365 DEBUG:boto:Data: 2022-09-13T07:39:43.365 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.366 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.366 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.367 DEBUG:boto:Params: {} 2022-09-13T07:39:43.367 DEBUG:boto:Token: None 2022-09-13T07:39:43.368 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.368 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bHaLg8CGQLV+bw43QqwvT5M0M6E= 2022-09-13T07:39:43.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bHaLg8CGQLV+bw43QqwvT5M0M6E=', 'Content-Length': '0'} 2022-09-13T07:39:43.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad7f17102f8c99c6-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.379 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:43.379 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:43.380 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.380 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.381 DEBUG:boto:Method: HEAD 2022-09-13T07:39:43.381 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.382 DEBUG:boto:Data: 2022-09-13T07:39:43.382 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.383 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.383 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.384 DEBUG:boto:Params: {} 2022-09-13T07:39:43.384 DEBUG:boto:Token: None 2022-09-13T07:39:43.385 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.385 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bHaLg8CGQLV+bw43QqwvT5M0M6E= 2022-09-13T07:39:43.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bHaLg8CGQLV+bw43QqwvT5M0M6E=', 'Content-Length': '0'} 2022-09-13T07:39:43.388 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c3620756a41abb67-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:39:43.389 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:43.390 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:43.390 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.391 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.391 DEBUG:boto:Method: GET 2022-09-13T07:39:43.392 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.392 DEBUG:boto:Data: 2022-09-13T07:39:43.393 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.393 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.394 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.394 DEBUG:boto:Params: {} 2022-09-13T07:39:43.395 DEBUG:boto:Token: None 2022-09-13T07:39:43.395 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.396 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf01XdT30/wMo7t0LILI6DFvWCg= 2022-09-13T07:39:43.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf01XdT30/wMo7t0LILI6DFvWCg=', 'Content-Length': '0'} 2022-09-13T07:39:43.399 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a0f0f27bd1d3c89-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.400 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:43.400 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:43.401 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.402 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.403 DEBUG:boto:Method: GET 2022-09-13T07:39:43.403 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.404 DEBUG:boto:Data: 2022-09-13T07:39:43.404 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.405 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.405 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.406 DEBUG:boto:Params: {} 2022-09-13T07:39:43.406 DEBUG:boto:Token: None 2022-09-13T07:39:43.407 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.408 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf01XdT30/wMo7t0LILI6DFvWCg= 2022-09-13T07:39:43.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf01XdT30/wMo7t0LILI6DFvWCg=', 'Content-Length': '0'} 2022-09-13T07:39:43.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000629f1d236de90284-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.411 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.412 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.412 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:43.413 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:43.414 DEBUG:boto:Method: GET 2022-09-13T07:39:43.414 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:39:43.415 DEBUG:boto:Data: 2022-09-13T07:39:43.415 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.416 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.416 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.417 DEBUG:boto:Params: {} 2022-09-13T07:39:43.417 DEBUG:boto:Token: None 2022-09-13T07:39:43.418 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/ 2022-09-13T07:39:43.419 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U= 2022-09-13T07:39:43.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U=', 'Content-Length': '0'} 2022-09-13T07:39:43.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf4d1ddac29483b8-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.466 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.468 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.469 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.469 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:43.470 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:43.470 DEBUG:boto:Method: GET 2022-09-13T07:39:43.471 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:39:43.472 DEBUG:boto:Data: 2022-09-13T07:39:43.472 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.473 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.474 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.475 DEBUG:boto:Params: {} 2022-09-13T07:39:43.476 DEBUG:boto:Token: None 2022-09-13T07:39:43.477 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/ 2022-09-13T07:39:43.478 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U= 2022-09-13T07:39:43.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U=', 'Content-Length': '0'} 2022-09-13T07:39:43.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026b6deaa44ce0625-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.484 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.485 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:43.486 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:43.487 DEBUG:boto:Method: HEAD 2022-09-13T07:39:43.488 DEBUG:boto:Path: /yxvvuh-19/dummy 2022-09-13T07:39:43.489 DEBUG:boto:Data: 2022-09-13T07:39:43.489 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.490 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.491 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.492 DEBUG:boto:Params: {} 2022-09-13T07:39:43.493 DEBUG:boto:Token: None 2022-09-13T07:39:43.494 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/dummy 2022-09-13T07:39:43.495 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tRS1bAIySONikV1+OhVxGIf0EOU= 2022-09-13T07:39:43.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tRS1bAIySONikV1+OhVxGIf0EOU=', 'Content-Length': '0'} 2022-09-13T07:39:43.499 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fdf7ebad4747bf4b-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.500 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:43.501 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:43.502 DEBUG:boto:Method: HEAD 2022-09-13T07:39:43.503 DEBUG:boto:Path: /yxvvuh-19/dummy 2022-09-13T07:39:43.504 DEBUG:boto:Data: 2022-09-13T07:39:43.505 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.506 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.507 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.507 DEBUG:boto:Params: {} 2022-09-13T07:39:43.508 DEBUG:boto:Token: None 2022-09-13T07:39:43.509 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/dummy 2022-09-13T07:39:43.510 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tRS1bAIySONikV1+OhVxGIf0EOU= 2022-09-13T07:39:43.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tRS1bAIySONikV1+OhVxGIf0EOU=', 'Content-Length': '0'} 2022-09-13T07:39:43.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e1d7e028c01f3eff-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-13T07:39:43.517 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:43.518 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:43.519 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.519 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.520 DEBUG:boto:Method: GET 2022-09-13T07:39:43.521 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.522 DEBUG:boto:Data: 2022-09-13T07:39:43.523 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.524 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.525 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.526 DEBUG:boto:Params: {} 2022-09-13T07:39:43.527 DEBUG:boto:Token: None 2022-09-13T07:39:43.528 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.529 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf01XdT30/wMo7t0LILI6DFvWCg= 2022-09-13T07:39:43.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf01XdT30/wMo7t0LILI6DFvWCg=', 'Content-Length': '0'} 2022-09-13T07:39:43.533 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000014cdce82d42b11db-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.534 DEBUG:boto:path=/yxvvuh-19/dummy 2022-09-13T07:39:43.535 DEBUG:boto:auth_path=/yxvvuh-19/dummy 2022-09-13T07:39:43.536 DEBUG:boto:path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.537 DEBUG:boto:auth_path=/yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.538 DEBUG:boto:Method: GET 2022-09-13T07:39:43.539 DEBUG:boto:Path: /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.540 DEBUG:boto:Data: 2022-09-13T07:39:43.541 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.542 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.543 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.544 DEBUG:boto:Params: {} 2022-09-13T07:39:43.545 DEBUG:boto:Token: None 2022-09-13T07:39:43.546 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/dummy?versionId=FDANku1GQSwU5HbWax2gCLnve9-nkdk 2022-09-13T07:39:43.547 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Bf01XdT30/wMo7t0LILI6DFvWCg= 2022-09-13T07:39:43.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Bf01XdT30/wMo7t0LILI6DFvWCg=', 'Content-Length': '0'} 2022-09-13T07:39:43.551 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:36:37 GMT'), ('x-amz-version-id', 'FDANku1GQSwU5HbWax2gCLnve9-nkdk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000001d82c422722d49e-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.552 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.553 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.554 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:43.555 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:43.556 DEBUG:boto:Method: GET 2022-09-13T07:39:43.557 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:39:43.558 DEBUG:boto:Data: 2022-09-13T07:39:43.558 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.559 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.560 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.561 DEBUG:boto:Params: {} 2022-09-13T07:39:43.562 DEBUG:boto:Token: None 2022-09-13T07:39:43.563 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/ 2022-09-13T07:39:43.564 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U= 2022-09-13T07:39:43.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U=', 'Content-Length': '0'} 2022-09-13T07:39:43.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008b7db339a5101fa-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.574 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.575 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.576 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.577 DEBUG:boto:path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:43.578 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=dummy 2022-09-13T07:39:43.579 DEBUG:boto:Method: GET 2022-09-13T07:39:43.580 DEBUG:boto:Path: /yxvvuh-19/?prefix=dummy 2022-09-13T07:39:43.580 DEBUG:boto:Data: 2022-09-13T07:39:43.581 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.582 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.583 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.584 DEBUG:boto:Params: {} 2022-09-13T07:39:43.585 DEBUG:boto:Token: None 2022-09-13T07:39:43.586 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/ 2022-09-13T07:39:43.587 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U= 2022-09-13T07:39:43.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U=', 'Content-Length': '0'} 2022-09-13T07:39:43.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d20bf9a2d837fa0f-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.634 DEBUG:boto:b'yxvvuh-19dummy1000falsedummy2022-09-13T07:36:37.664Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:43.635 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.636 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.637 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.637 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.638 DEBUG:boto:Method: GET 2022-09-13T07:39:43.639 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.639 DEBUG:boto:Data: 2022-09-13T07:39:43.640 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.641 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.642 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.643 DEBUG:boto:Params: {} 2022-09-13T07:39:43.644 DEBUG:boto:Token: None 2022-09-13T07:39:43.645 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.646 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY= 2022-09-13T07:39:43.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY=', 'Content-Length': '0'} 2022-09-13T07:39:43.650 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056fabe8073f83275-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.652 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.653 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.653 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.654 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.654 DEBUG:boto:Method: GET 2022-09-13T07:39:43.655 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.655 DEBUG:boto:Data: 2022-09-13T07:39:43.656 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.656 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.657 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.657 DEBUG:boto:Params: {} 2022-09-13T07:39:43.658 DEBUG:boto:Token: None 2022-09-13T07:39:43.659 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.659 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY= 2022-09-13T07:39:43.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY=', 'Content-Length': '0'} 2022-09-13T07:39:43.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044c944cee064dc57-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.663 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.664 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.664 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.665 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.665 DEBUG:boto:Method: HEAD 2022-09-13T07:39:43.666 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.666 DEBUG:boto:Data: 2022-09-13T07:39:43.667 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.667 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.668 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.668 DEBUG:boto:Params: {} 2022-09-13T07:39:43.669 DEBUG:boto:Token: None 2022-09-13T07:39:43.669 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.670 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TkbUR7C+DxrGJroJ9BkoQ704YIQ= 2022-09-13T07:39:43.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TkbUR7C+DxrGJroJ9BkoQ704YIQ=', 'Content-Length': '0'} 2022-09-13T07:39:43.673 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd0955f36bb141fa-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.673 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.674 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.674 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.675 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.676 DEBUG:boto:Method: HEAD 2022-09-13T07:39:43.677 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.678 DEBUG:boto:Data: 2022-09-13T07:39:43.679 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.679 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.680 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.680 DEBUG:boto:Params: {} 2022-09-13T07:39:43.681 DEBUG:boto:Token: None 2022-09-13T07:39:43.682 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.682 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TkbUR7C+DxrGJroJ9BkoQ704YIQ= 2022-09-13T07:39:43.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TkbUR7C+DxrGJroJ9BkoQ704YIQ=', 'Content-Length': '0'} 2022-09-13T07:39:43.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041bff8943db2e0d1-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:43.687 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.687 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.688 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.688 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.689 DEBUG:boto:Method: GET 2022-09-13T07:39:43.689 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.690 DEBUG:boto:Data: 2022-09-13T07:39:43.691 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.692 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.693 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.694 DEBUG:boto:Params: {} 2022-09-13T07:39:43.695 DEBUG:boto:Token: None 2022-09-13T07:39:43.695 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.696 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY= 2022-09-13T07:39:43.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY=', 'Content-Length': '0'} 2022-09-13T07:39:43.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003117dde26771c0a0-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.746 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.747 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.748 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.749 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.750 DEBUG:boto:Method: GET 2022-09-13T07:39:43.751 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.752 DEBUG:boto:Data: 2022-09-13T07:39:43.753 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.754 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.755 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.756 DEBUG:boto:Params: {} 2022-09-13T07:39:43.757 DEBUG:boto:Token: None 2022-09-13T07:39:43.758 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.760 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY= 2022-09-13T07:39:43.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY=', 'Content-Length': '0'} 2022-09-13T07:39:43.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fbab98638ebe3c3-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.765 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.765 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.766 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.767 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.767 DEBUG:boto:Method: GET 2022-09-13T07:39:43.768 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.769 DEBUG:boto:Data: 2022-09-13T07:39:43.770 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.771 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.772 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.773 DEBUG:boto:Params: {} 2022-09-13T07:39:43.774 DEBUG:boto:Token: None 2022-09-13T07:39:43.775 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/ 2022-09-13T07:39:43.776 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U= 2022-09-13T07:39:43.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U=', 'Content-Length': '0'} 2022-09-13T07:39:43.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac5aeb9ed8dd36f1-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.781 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.782 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.783 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.784 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.784 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.785 DEBUG:boto:Method: GET 2022-09-13T07:39:43.785 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.786 DEBUG:boto:Data: 2022-09-13T07:39:43.786 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.787 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.788 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.788 DEBUG:boto:Params: {} 2022-09-13T07:39:43.789 DEBUG:boto:Token: None 2022-09-13T07:39:43.790 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/ 2022-09-13T07:39:43.790 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U= 2022-09-13T07:39:43.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U=', 'Content-Length': '0'} 2022-09-13T07:39:43.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf04592890e4e40a-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.795 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.796 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.796 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.797 DEBUG:boto:Method: HEAD 2022-09-13T07:39:43.797 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:43.798 DEBUG:boto:Data: 2022-09-13T07:39:43.799 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.799 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.800 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.801 DEBUG:boto:Params: {} 2022-09-13T07:39:43.802 DEBUG:boto:Token: None 2022-09-13T07:39:43.803 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj 2022-09-13T07:39:43.804 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t9Fn7c86AQG7c16SMIZIB1GaAbM= 2022-09-13T07:39:43.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t9Fn7c86AQG7c16SMIZIB1GaAbM=', 'Content-Length': '0'} 2022-09-13T07:39:43.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aaf384a287463b03-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.810 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.811 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.811 DEBUG:boto:Method: HEAD 2022-09-13T07:39:43.812 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:43.812 DEBUG:boto:Data: 2022-09-13T07:39:43.813 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.813 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.814 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.814 DEBUG:boto:Params: {} 2022-09-13T07:39:43.815 DEBUG:boto:Token: None 2022-09-13T07:39:43.816 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj 2022-09-13T07:39:43.816 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t9Fn7c86AQG7c16SMIZIB1GaAbM= 2022-09-13T07:39:43.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t9Fn7c86AQG7c16SMIZIB1GaAbM=', 'Content-Length': '0'} 2022-09-13T07:39:43.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056e61834dce77e0d-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:43.822 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.823 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.823 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.824 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.824 DEBUG:boto:Method: GET 2022-09-13T07:39:43.825 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.826 DEBUG:boto:Data: 2022-09-13T07:39:43.826 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.827 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.827 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.828 DEBUG:boto:Params: {} 2022-09-13T07:39:43.829 DEBUG:boto:Token: None 2022-09-13T07:39:43.829 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.830 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY= 2022-09-13T07:39:43.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY=', 'Content-Length': '0'} 2022-09-13T07:39:43.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df5615e71c1b40fe-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.834 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.835 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.836 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.836 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.837 DEBUG:boto:Method: GET 2022-09-13T07:39:43.837 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.838 DEBUG:boto:Data: 2022-09-13T07:39:43.838 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.839 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.839 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.840 DEBUG:boto:Params: {} 2022-09-13T07:39:43.841 DEBUG:boto:Token: None 2022-09-13T07:39:43.842 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:43.843 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY= 2022-09-13T07:39:43.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:elUqvOzaJz5u3m9H4mS9yzrT9OY=', 'Content-Length': '0'} 2022-09-13T07:39:43.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006174ff3c0a6f278d-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.890 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.891 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.891 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.892 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.893 DEBUG:boto:Method: GET 2022-09-13T07:39:43.893 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.894 DEBUG:boto:Data: 2022-09-13T07:39:43.894 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.895 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.895 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.896 DEBUG:boto:Params: {} 2022-09-13T07:39:43.897 DEBUG:boto:Token: None 2022-09-13T07:39:43.897 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/ 2022-09-13T07:39:43.898 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U= 2022-09-13T07:39:43.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U=', 'Content-Length': '0'} 2022-09-13T07:39:43.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000285e3a05e2ee8470-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.903 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.903 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:43.904 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:43.905 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.905 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.906 DEBUG:boto:Method: GET 2022-09-13T07:39:43.906 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:43.907 DEBUG:boto:Data: 2022-09-13T07:39:43.907 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.908 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.909 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.909 DEBUG:boto:Params: {} 2022-09-13T07:39:43.910 DEBUG:boto:Token: None 2022-09-13T07:39:43.911 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/ 2022-09-13T07:39:43.912 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U= 2022-09-13T07:39:43.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x6lUpFrNQGB/hYYy0yfkyT1KG+U=', 'Content-Length': '0'} 2022-09-13T07:39:43.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bca8e4eef5fb48ca-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.918 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:43.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:43.920 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.920 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.921 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.921 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.922 DEBUG:boto:Method: GET 2022-09-13T07:39:43.923 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.923 DEBUG:boto:Data: 2022-09-13T07:39:43.924 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.924 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.925 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.925 DEBUG:boto:Params: {} 2022-09-13T07:39:43.926 DEBUG:boto:Token: None 2022-09-13T07:39:43.927 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.927 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oChtslWWrhrLD+xxP8fu576YPzA= 2022-09-13T07:39:43.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oChtslWWrhrLD+xxP8fu576YPzA=', 'Content-Length': '0'} 2022-09-13T07:39:43.931 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b159734209f592c-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.933 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.934 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.935 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.935 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.936 DEBUG:boto:Method: GET 2022-09-13T07:39:43.936 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.937 DEBUG:boto:Data: 2022-09-13T07:39:43.937 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.938 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.938 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.939 DEBUG:boto:Params: {} 2022-09-13T07:39:43.940 DEBUG:boto:Token: None 2022-09-13T07:39:43.941 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.941 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oChtslWWrhrLD+xxP8fu576YPzA= 2022-09-13T07:39:43.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oChtslWWrhrLD+xxP8fu576YPzA=', 'Content-Length': '0'} 2022-09-13T07:39:43.944 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a12c918f6560f2d8-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.945 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.946 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.946 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.947 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.947 DEBUG:boto:Method: HEAD 2022-09-13T07:39:43.948 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.948 DEBUG:boto:Data: 2022-09-13T07:39:43.949 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.949 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.950 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.950 DEBUG:boto:Params: {} 2022-09-13T07:39:43.951 DEBUG:boto:Token: None 2022-09-13T07:39:43.951 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.952 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:E9TRW+gPQH8BwqXgJ72ppbir5LM= 2022-09-13T07:39:43.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:E9TRW+gPQH8BwqXgJ72ppbir5LM=', 'Content-Length': '0'} 2022-09-13T07:39:43.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007031be67e2d1fd63-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.956 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.956 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.957 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.957 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.958 DEBUG:boto:Method: HEAD 2022-09-13T07:39:43.958 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.959 DEBUG:boto:Data: 2022-09-13T07:39:43.959 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.960 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:43.960 DEBUG:boto:Port: 8000 2022-09-13T07:39:43.961 DEBUG:boto:Params: {} 2022-09-13T07:39:43.962 DEBUG:boto:Token: None 2022-09-13T07:39:43.962 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.963 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:E9TRW+gPQH8BwqXgJ72ppbir5LM= 2022-09-13T07:39:43.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:E9TRW+gPQH8BwqXgJ72ppbir5LM=', 'Content-Length': '0'} 2022-09-13T07:39:43.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3c77b530bc7e496-00632033bf-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:43.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:43.967 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:43.968 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:43.968 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.969 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.969 DEBUG:boto:Method: GET 2022-09-13T07:39:43.970 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.970 DEBUG:boto:Data: 2022-09-13T07:39:43.971 DEBUG:boto:Headers: {} 2022-09-13T07:39:43.972 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:43.973 DEBUG:boto:Port: 8001 2022-09-13T07:39:43.974 DEBUG:boto:Params: {} 2022-09-13T07:39:43.975 DEBUG:boto:Token: None 2022-09-13T07:39:43.976 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:43 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:43.977 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oChtslWWrhrLD+xxP8fu576YPzA= 2022-09-13T07:39:43.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:43 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oChtslWWrhrLD+xxP8fu576YPzA=', 'Content-Length': '0'} 2022-09-13T07:39:43.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b00e958a5ad109c-00632033bf-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.022 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.024 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.024 DEBUG:boto:path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:44.025 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:44.026 DEBUG:boto:Method: GET 2022-09-13T07:39:44.027 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:44.028 DEBUG:boto:Data: 2022-09-13T07:39:44.029 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.030 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.031 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.031 DEBUG:boto:Params: {} 2022-09-13T07:39:44.033 DEBUG:boto:Token: None 2022-09-13T07:39:44.034 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD 2022-09-13T07:39:44.035 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tzMHNWmOxHhxtsKhMV+LHk6f/XA= 2022-09-13T07:39:44.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tzMHNWmOxHhxtsKhMV+LHk6f/XA=', 'Content-Length': '0'} 2022-09-13T07:39:44.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'FmwBtbzZwRR4A0HEc28dvfcEWdzuIZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7a972c96cbba809-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.042 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.042 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.043 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.043 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.044 DEBUG:boto:Method: GET 2022-09-13T07:39:44.045 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.045 DEBUG:boto:Data: 2022-09-13T07:39:44.046 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.046 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.047 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.047 DEBUG:boto:Params: {} 2022-09-13T07:39:44.048 DEBUG:boto:Token: None 2022-09-13T07:39:44.049 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.049 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6e8ddc2c8ce504c-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.053 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.054 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.055 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.056 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.056 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.057 DEBUG:boto:Method: GET 2022-09-13T07:39:44.057 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.058 DEBUG:boto:Data: 2022-09-13T07:39:44.058 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.059 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.059 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.060 DEBUG:boto:Params: {} 2022-09-13T07:39:44.061 DEBUG:boto:Token: None 2022-09-13T07:39:44.061 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.062 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcf077e72ae1f501-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.067 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.068 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.069 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.070 DEBUG:boto:Method: HEAD 2022-09-13T07:39:44.071 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:44.072 DEBUG:boto:Data: 2022-09-13T07:39:44.073 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.074 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.075 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.076 DEBUG:boto:Params: {} 2022-09-13T07:39:44.076 DEBUG:boto:Token: None 2022-09-13T07:39:44.077 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj 2022-09-13T07:39:44.078 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4= 2022-09-13T07:39:44.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4=', 'Content-Length': '0'} 2022-09-13T07:39:44.083 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061d2880e827c6c46-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.084 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.085 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.086 DEBUG:boto:Method: HEAD 2022-09-13T07:39:44.087 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:44.088 DEBUG:boto:Data: 2022-09-13T07:39:44.089 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.090 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.090 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.091 DEBUG:boto:Params: {} 2022-09-13T07:39:44.092 DEBUG:boto:Token: None 2022-09-13T07:39:44.093 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj 2022-09-13T07:39:44.103 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4= 2022-09-13T07:39:44.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4=', 'Content-Length': '0'} 2022-09-13T07:39:44.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f40a3126421cb516-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:44.109 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.110 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.111 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.111 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.112 DEBUG:boto:Method: GET 2022-09-13T07:39:44.113 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.113 DEBUG:boto:Data: 2022-09-13T07:39:44.114 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.115 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.115 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.116 DEBUG:boto:Params: {} 2022-09-13T07:39:44.117 DEBUG:boto:Token: None 2022-09-13T07:39:44.118 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.118 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI= 2022-09-13T07:39:44.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI=', 'Content-Length': '0'} 2022-09-13T07:39:44.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d990478cb8554569-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.122 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.123 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.123 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.124 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.124 DEBUG:boto:Method: GET 2022-09-13T07:39:44.125 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.126 DEBUG:boto:Data: 2022-09-13T07:39:44.126 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.127 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.127 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.128 DEBUG:boto:Params: {} 2022-09-13T07:39:44.129 DEBUG:boto:Token: None 2022-09-13T07:39:44.130 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.131 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI= 2022-09-13T07:39:44.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI=', 'Content-Length': '0'} 2022-09-13T07:39:44.135 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000332aec43414f8dcb-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.178 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.179 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.180 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.180 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.181 DEBUG:boto:Method: GET 2022-09-13T07:39:44.181 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.182 DEBUG:boto:Data: 2022-09-13T07:39:44.183 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.183 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.184 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.184 DEBUG:boto:Params: {} 2022-09-13T07:39:44.185 DEBUG:boto:Token: None 2022-09-13T07:39:44.185 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.186 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3f1bd53b7c57c0c-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.190 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.191 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.192 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.193 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.193 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.194 DEBUG:boto:Method: GET 2022-09-13T07:39:44.194 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.195 DEBUG:boto:Data: 2022-09-13T07:39:44.195 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.196 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.196 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.197 DEBUG:boto:Params: {} 2022-09-13T07:39:44.197 DEBUG:boto:Token: None 2022-09-13T07:39:44.198 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.198 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ea4c236ab04b72b-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.203 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:44.204 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.205 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.205 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.206 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.206 DEBUG:boto:Method: GET 2022-09-13T07:39:44.207 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.207 DEBUG:boto:Data: 2022-09-13T07:39:44.208 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.209 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.209 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.210 DEBUG:boto:Params: {} 2022-09-13T07:39:44.210 DEBUG:boto:Token: None 2022-09-13T07:39:44.211 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.211 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SLrchun0XXFg2aztt4HFjKV5k3U= 2022-09-13T07:39:44.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SLrchun0XXFg2aztt4HFjKV5k3U=', 'Content-Length': '0'} 2022-09-13T07:39:44.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b98817b445860a0-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.215 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.215 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.216 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.217 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.217 DEBUG:boto:Method: GET 2022-09-13T07:39:44.218 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.218 DEBUG:boto:Data: 2022-09-13T07:39:44.219 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.219 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.220 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.220 DEBUG:boto:Params: {} 2022-09-13T07:39:44.221 DEBUG:boto:Token: None 2022-09-13T07:39:44.221 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.222 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SLrchun0XXFg2aztt4HFjKV5k3U= 2022-09-13T07:39:44.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SLrchun0XXFg2aztt4HFjKV5k3U=', 'Content-Length': '0'} 2022-09-13T07:39:44.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000559e01c09a9633b-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.226 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.226 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.227 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.227 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.228 DEBUG:boto:Method: HEAD 2022-09-13T07:39:44.228 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.229 DEBUG:boto:Data: 2022-09-13T07:39:44.229 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.230 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.230 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.231 DEBUG:boto:Params: {} 2022-09-13T07:39:44.231 DEBUG:boto:Token: None 2022-09-13T07:39:44.232 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.232 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GQh/xyuM5Zog03F/yO7XiSJ+zxY= 2022-09-13T07:39:44.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GQh/xyuM5Zog03F/yO7XiSJ+zxY=', 'Content-Length': '0'} 2022-09-13T07:39:44.235 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ee6be49a21457be-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.236 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.237 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.237 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.238 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.238 DEBUG:boto:Method: HEAD 2022-09-13T07:39:44.239 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.239 DEBUG:boto:Data: 2022-09-13T07:39:44.240 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.240 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.241 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.241 DEBUG:boto:Params: {} 2022-09-13T07:39:44.242 DEBUG:boto:Token: None 2022-09-13T07:39:44.242 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.243 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GQh/xyuM5Zog03F/yO7XiSJ+zxY= 2022-09-13T07:39:44.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GQh/xyuM5Zog03F/yO7XiSJ+zxY=', 'Content-Length': '0'} 2022-09-13T07:39:44.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd7151775f751643-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:44.247 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.248 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.248 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.249 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.249 DEBUG:boto:Method: GET 2022-09-13T07:39:44.250 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.250 DEBUG:boto:Data: 2022-09-13T07:39:44.251 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.251 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.252 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.252 DEBUG:boto:Params: {} 2022-09-13T07:39:44.253 DEBUG:boto:Token: None 2022-09-13T07:39:44.254 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.254 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SLrchun0XXFg2aztt4HFjKV5k3U= 2022-09-13T07:39:44.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SLrchun0XXFg2aztt4HFjKV5k3U=', 'Content-Length': '0'} 2022-09-13T07:39:44.257 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6c5181e10530aec-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.298 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.299 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.299 DEBUG:boto:path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.300 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.301 DEBUG:boto:Method: GET 2022-09-13T07:39:44.302 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.303 DEBUG:boto:Data: 2022-09-13T07:39:44.303 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.304 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.304 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.305 DEBUG:boto:Params: {} 2022-09-13T07:39:44.305 DEBUG:boto:Token: None 2022-09-13T07:39:44.306 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7 2022-09-13T07:39:44.307 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SLrchun0XXFg2aztt4HFjKV5k3U= 2022-09-13T07:39:44.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SLrchun0XXFg2aztt4HFjKV5k3U=', 'Content-Length': '0'} 2022-09-13T07:39:44.310 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'jgnkUBGMsWG7OrVYbmwDKGe6FpMsFA7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000912329d8f2968424-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.310 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.311 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.311 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.312 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.312 DEBUG:boto:Method: GET 2022-09-13T07:39:44.313 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.313 DEBUG:boto:Data: 2022-09-13T07:39:44.314 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.314 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.315 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.315 DEBUG:boto:Params: {} 2022-09-13T07:39:44.316 DEBUG:boto:Token: None 2022-09-13T07:39:44.317 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.317 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000153ada98c0dfd2a7-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.322 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.323 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.323 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.324 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.324 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.325 DEBUG:boto:Method: GET 2022-09-13T07:39:44.325 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.326 DEBUG:boto:Data: 2022-09-13T07:39:44.326 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.327 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.327 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.328 DEBUG:boto:Params: {} 2022-09-13T07:39:44.329 DEBUG:boto:Token: None 2022-09-13T07:39:44.329 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.330 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c37abe9d5108c961-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.334 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.335 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.335 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.336 DEBUG:boto:Method: HEAD 2022-09-13T07:39:44.336 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:44.337 DEBUG:boto:Data: 2022-09-13T07:39:44.337 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.338 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.338 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.339 DEBUG:boto:Params: {} 2022-09-13T07:39:44.339 DEBUG:boto:Token: None 2022-09-13T07:39:44.340 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj 2022-09-13T07:39:44.340 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4= 2022-09-13T07:39:44.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4=', 'Content-Length': '0'} 2022-09-13T07:39:44.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051da09143bad77fb-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.345 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.346 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.347 DEBUG:boto:Method: HEAD 2022-09-13T07:39:44.347 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:44.347 DEBUG:boto:Data: 2022-09-13T07:39:44.348 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.348 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.349 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.349 DEBUG:boto:Params: {} 2022-09-13T07:39:44.350 DEBUG:boto:Token: None 2022-09-13T07:39:44.350 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj 2022-09-13T07:39:44.351 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4= 2022-09-13T07:39:44.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4=', 'Content-Length': '0'} 2022-09-13T07:39:44.355 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000896a9ec14812cad-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:44.357 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.357 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.358 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.358 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.359 DEBUG:boto:Method: GET 2022-09-13T07:39:44.359 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.360 DEBUG:boto:Data: 2022-09-13T07:39:44.360 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.361 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.362 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.362 DEBUG:boto:Params: {} 2022-09-13T07:39:44.363 DEBUG:boto:Token: None 2022-09-13T07:39:44.363 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.364 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI= 2022-09-13T07:39:44.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI=', 'Content-Length': '0'} 2022-09-13T07:39:44.367 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cac0f123574ed09-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.368 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.368 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.369 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.370 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.370 DEBUG:boto:Method: GET 2022-09-13T07:39:44.371 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.371 DEBUG:boto:Data: 2022-09-13T07:39:44.372 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.373 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.373 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.374 DEBUG:boto:Params: {} 2022-09-13T07:39:44.374 DEBUG:boto:Token: None 2022-09-13T07:39:44.375 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.376 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI= 2022-09-13T07:39:44.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI=', 'Content-Length': '0'} 2022-09-13T07:39:44.379 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de75cfa89754f687-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.422 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.424 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.424 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.425 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.427 DEBUG:boto:Method: GET 2022-09-13T07:39:44.427 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.428 DEBUG:boto:Data: 2022-09-13T07:39:44.428 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.429 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.429 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.430 DEBUG:boto:Params: {} 2022-09-13T07:39:44.431 DEBUG:boto:Token: None 2022-09-13T07:39:44.431 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.432 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6785aa935326e63-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.440 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.441 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.442 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.443 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.444 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.444 DEBUG:boto:Method: GET 2022-09-13T07:39:44.445 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.446 DEBUG:boto:Data: 2022-09-13T07:39:44.446 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.447 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.447 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.448 DEBUG:boto:Params: {} 2022-09-13T07:39:44.449 DEBUG:boto:Token: None 2022-09-13T07:39:44.449 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.450 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000918b30ba4c2de63e-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.454 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:44.456 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.456 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.457 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.457 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.458 DEBUG:boto:Method: GET 2022-09-13T07:39:44.458 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.459 DEBUG:boto:Data: 2022-09-13T07:39:44.459 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.460 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.460 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.461 DEBUG:boto:Params: {} 2022-09-13T07:39:44.461 DEBUG:boto:Token: None 2022-09-13T07:39:44.462 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.462 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lctPJ5Ic9l4jc1efNKG/ubbtnAg= 2022-09-13T07:39:44.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lctPJ5Ic9l4jc1efNKG/ubbtnAg=', 'Content-Length': '0'} 2022-09-13T07:39:44.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000115ae76888b1b128-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.510 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.511 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.511 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.512 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.512 DEBUG:boto:Method: GET 2022-09-13T07:39:44.513 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.513 DEBUG:boto:Data: 2022-09-13T07:39:44.514 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.514 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.515 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.515 DEBUG:boto:Params: {} 2022-09-13T07:39:44.516 DEBUG:boto:Token: None 2022-09-13T07:39:44.517 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.517 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lctPJ5Ic9l4jc1efNKG/ubbtnAg= 2022-09-13T07:39:44.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lctPJ5Ic9l4jc1efNKG/ubbtnAg=', 'Content-Length': '0'} 2022-09-13T07:39:44.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab1f887899f5d90c-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.521 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.522 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.522 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.523 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.523 DEBUG:boto:Method: HEAD 2022-09-13T07:39:44.524 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.525 DEBUG:boto:Data: 2022-09-13T07:39:44.526 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.527 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.527 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.528 DEBUG:boto:Params: {} 2022-09-13T07:39:44.529 DEBUG:boto:Token: None 2022-09-13T07:39:44.531 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.532 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fhMM8IU47J3yWWLzyr6UdvUupLM= 2022-09-13T07:39:44.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fhMM8IU47J3yWWLzyr6UdvUupLM=', 'Content-Length': '0'} 2022-09-13T07:39:44.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b4feb060a5ec1de-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.536 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.537 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.538 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.539 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.540 DEBUG:boto:Method: HEAD 2022-09-13T07:39:44.541 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.541 DEBUG:boto:Data: 2022-09-13T07:39:44.542 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.543 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.544 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.545 DEBUG:boto:Params: {} 2022-09-13T07:39:44.546 DEBUG:boto:Token: None 2022-09-13T07:39:44.547 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.547 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:fhMM8IU47J3yWWLzyr6UdvUupLM= 2022-09-13T07:39:44.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:fhMM8IU47J3yWWLzyr6UdvUupLM=', 'Content-Length': '0'} 2022-09-13T07:39:44.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4b1823970f6cdd6-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:44.553 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.554 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.555 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.556 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.557 DEBUG:boto:Method: GET 2022-09-13T07:39:44.558 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.559 DEBUG:boto:Data: 2022-09-13T07:39:44.559 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.560 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.561 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.562 DEBUG:boto:Params: {} 2022-09-13T07:39:44.563 DEBUG:boto:Token: None 2022-09-13T07:39:44.564 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.565 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lctPJ5Ic9l4jc1efNKG/ubbtnAg= 2022-09-13T07:39:44.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lctPJ5Ic9l4jc1efNKG/ubbtnAg=', 'Content-Length': '0'} 2022-09-13T07:39:44.568 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000757b9a65b5be943c-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.569 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.569 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.570 DEBUG:boto:path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.571 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.572 DEBUG:boto:Method: GET 2022-09-13T07:39:44.573 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.573 DEBUG:boto:Data: 2022-09-13T07:39:44.574 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.574 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.575 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.575 DEBUG:boto:Params: {} 2022-09-13T07:39:44.576 DEBUG:boto:Token: None 2022-09-13T07:39:44.576 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f 2022-09-13T07:39:44.577 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lctPJ5Ic9l4jc1efNKG/ubbtnAg= 2022-09-13T07:39:44.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lctPJ5Ic9l4jc1efNKG/ubbtnAg=', 'Content-Length': '0'} 2022-09-13T07:39:44.580 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'TL9xVVVkIODKu69Lcj5HlhVjbyjGE3f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4c487e3925c1cfe-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.581 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.581 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.582 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.582 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.583 DEBUG:boto:Method: GET 2022-09-13T07:39:44.584 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.584 DEBUG:boto:Data: 2022-09-13T07:39:44.585 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.585 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.586 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.586 DEBUG:boto:Params: {} 2022-09-13T07:39:44.587 DEBUG:boto:Token: None 2022-09-13T07:39:44.588 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.588 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f6f5db0b3242110-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.593 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.594 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.595 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.595 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.596 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.596 DEBUG:boto:Method: GET 2022-09-13T07:39:44.597 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.597 DEBUG:boto:Data: 2022-09-13T07:39:44.598 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.599 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.600 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.601 DEBUG:boto:Params: {} 2022-09-13T07:39:44.601 DEBUG:boto:Token: None 2022-09-13T07:39:44.602 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.604 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3ea1855ac228521-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.609 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.610 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.611 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.612 DEBUG:boto:Method: HEAD 2022-09-13T07:39:44.613 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:44.614 DEBUG:boto:Data: 2022-09-13T07:39:44.614 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.615 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.615 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.616 DEBUG:boto:Params: {} 2022-09-13T07:39:44.617 DEBUG:boto:Token: None 2022-09-13T07:39:44.617 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj 2022-09-13T07:39:44.618 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4= 2022-09-13T07:39:44.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4=', 'Content-Length': '0'} 2022-09-13T07:39:44.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076d7183c6b95b921-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.623 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.624 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.625 DEBUG:boto:Method: HEAD 2022-09-13T07:39:44.625 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:44.626 DEBUG:boto:Data: 2022-09-13T07:39:44.626 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.627 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.627 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.628 DEBUG:boto:Params: {} 2022-09-13T07:39:44.628 DEBUG:boto:Token: None 2022-09-13T07:39:44.629 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj 2022-09-13T07:39:44.630 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4= 2022-09-13T07:39:44.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4=', 'Content-Length': '0'} 2022-09-13T07:39:44.634 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd9fb67b2a8f13f5-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:44.636 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.637 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.637 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.638 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.638 DEBUG:boto:Method: GET 2022-09-13T07:39:44.639 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.640 DEBUG:boto:Data: 2022-09-13T07:39:44.640 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.641 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.641 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.642 DEBUG:boto:Params: {} 2022-09-13T07:39:44.643 DEBUG:boto:Token: None 2022-09-13T07:39:44.643 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.644 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI= 2022-09-13T07:39:44.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI=', 'Content-Length': '0'} 2022-09-13T07:39:44.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4e3ba71323e6dfc-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.647 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.648 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.648 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.649 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.649 DEBUG:boto:Method: GET 2022-09-13T07:39:44.650 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.651 DEBUG:boto:Data: 2022-09-13T07:39:44.651 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.652 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.652 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.653 DEBUG:boto:Params: {} 2022-09-13T07:39:44.654 DEBUG:boto:Token: None 2022-09-13T07:39:44.655 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.655 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI= 2022-09-13T07:39:44.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI=', 'Content-Length': '0'} 2022-09-13T07:39:44.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2440f2fbb756c74-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.659 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.660 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.661 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.661 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.662 DEBUG:boto:Method: GET 2022-09-13T07:39:44.663 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.663 DEBUG:boto:Data: 2022-09-13T07:39:44.664 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.664 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.665 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.666 DEBUG:boto:Params: {} 2022-09-13T07:39:44.666 DEBUG:boto:Token: None 2022-09-13T07:39:44.667 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.668 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8b6799e44fe0df9-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.714 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.716 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.717 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.717 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.718 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.719 DEBUG:boto:Method: GET 2022-09-13T07:39:44.720 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.720 DEBUG:boto:Data: 2022-09-13T07:39:44.721 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.721 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.722 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.723 DEBUG:boto:Params: {} 2022-09-13T07:39:44.723 DEBUG:boto:Token: None 2022-09-13T07:39:44.724 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.724 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000879477dfaa445bb7-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.729 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:44.730 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.731 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.732 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.732 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.733 DEBUG:boto:Method: GET 2022-09-13T07:39:44.733 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.734 DEBUG:boto:Data: 2022-09-13T07:39:44.734 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.735 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.735 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.736 DEBUG:boto:Params: {} 2022-09-13T07:39:44.737 DEBUG:boto:Token: None 2022-09-13T07:39:44.738 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.738 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HjLqtg1WhnjkBZditz8eFF99li0= 2022-09-13T07:39:44.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HjLqtg1WhnjkBZditz8eFF99li0=', 'Content-Length': '0'} 2022-09-13T07:39:44.741 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fb321db541c84b5-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.742 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.743 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.743 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.744 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.744 DEBUG:boto:Method: GET 2022-09-13T07:39:44.745 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.746 DEBUG:boto:Data: 2022-09-13T07:39:44.746 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.747 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.747 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.748 DEBUG:boto:Params: {} 2022-09-13T07:39:44.748 DEBUG:boto:Token: None 2022-09-13T07:39:44.749 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.750 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HjLqtg1WhnjkBZditz8eFF99li0= 2022-09-13T07:39:44.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HjLqtg1WhnjkBZditz8eFF99li0=', 'Content-Length': '0'} 2022-09-13T07:39:44.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b62b20daba4a585a-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.753 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.754 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.755 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.755 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.756 DEBUG:boto:Method: HEAD 2022-09-13T07:39:44.756 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.757 DEBUG:boto:Data: 2022-09-13T07:39:44.757 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.758 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.758 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.759 DEBUG:boto:Params: {} 2022-09-13T07:39:44.759 DEBUG:boto:Token: None 2022-09-13T07:39:44.760 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.760 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iUvNEgq6OpSdBu0KSuDP1lUA9y4= 2022-09-13T07:39:44.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iUvNEgq6OpSdBu0KSuDP1lUA9y4=', 'Content-Length': '0'} 2022-09-13T07:39:44.763 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031e961a95a70adfd-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.764 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.765 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.765 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.766 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.766 DEBUG:boto:Method: HEAD 2022-09-13T07:39:44.767 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.767 DEBUG:boto:Data: 2022-09-13T07:39:44.768 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.768 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.769 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.769 DEBUG:boto:Params: {} 2022-09-13T07:39:44.770 DEBUG:boto:Token: None 2022-09-13T07:39:44.770 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.771 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iUvNEgq6OpSdBu0KSuDP1lUA9y4= 2022-09-13T07:39:44.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iUvNEgq6OpSdBu0KSuDP1lUA9y4=', 'Content-Length': '0'} 2022-09-13T07:39:44.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089d832e87a8e09a4-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:44.775 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.776 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.776 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.777 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.777 DEBUG:boto:Method: GET 2022-09-13T07:39:44.778 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.778 DEBUG:boto:Data: 2022-09-13T07:39:44.779 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.779 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.780 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.780 DEBUG:boto:Params: {} 2022-09-13T07:39:44.781 DEBUG:boto:Token: None 2022-09-13T07:39:44.782 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.783 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HjLqtg1WhnjkBZditz8eFF99li0= 2022-09-13T07:39:44.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HjLqtg1WhnjkBZditz8eFF99li0=', 'Content-Length': '0'} 2022-09-13T07:39:44.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095975c2c31edcf59-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.786 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.787 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.787 DEBUG:boto:path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.788 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.788 DEBUG:boto:Method: GET 2022-09-13T07:39:44.789 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.789 DEBUG:boto:Data: 2022-09-13T07:39:44.790 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.791 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.791 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.792 DEBUG:boto:Params: {} 2022-09-13T07:39:44.793 DEBUG:boto:Token: None 2022-09-13T07:39:44.793 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=U.elqMxtYPda6ZsKEKDP833Q93.DHGx 2022-09-13T07:39:44.794 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HjLqtg1WhnjkBZditz8eFF99li0= 2022-09-13T07:39:44.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HjLqtg1WhnjkBZditz8eFF99li0=', 'Content-Length': '0'} 2022-09-13T07:39:44.797 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'U.elqMxtYPda6ZsKEKDP833Q93.DHGx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a8d8177d6299aa3-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.798 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.799 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.799 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.800 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.801 DEBUG:boto:Method: GET 2022-09-13T07:39:44.801 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.802 DEBUG:boto:Data: 2022-09-13T07:39:44.802 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.803 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.803 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.804 DEBUG:boto:Params: {} 2022-09-13T07:39:44.804 DEBUG:boto:Token: None 2022-09-13T07:39:44.805 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.806 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1352888ed1f9c27-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.811 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.812 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.814 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.815 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.816 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.816 DEBUG:boto:Method: GET 2022-09-13T07:39:44.817 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.818 DEBUG:boto:Data: 2022-09-13T07:39:44.819 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.820 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.821 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.821 DEBUG:boto:Params: {} 2022-09-13T07:39:44.822 DEBUG:boto:Token: None 2022-09-13T07:39:44.823 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.823 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000872b4c01ba6b131f-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.828 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.829 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.829 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.830 DEBUG:boto:Method: HEAD 2022-09-13T07:39:44.830 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:44.831 DEBUG:boto:Data: 2022-09-13T07:39:44.831 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.832 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.832 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.833 DEBUG:boto:Params: {} 2022-09-13T07:39:44.834 DEBUG:boto:Token: None 2022-09-13T07:39:44.834 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj 2022-09-13T07:39:44.835 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4= 2022-09-13T07:39:44.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4=', 'Content-Length': '0'} 2022-09-13T07:39:44.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000426b5b97a2e84b04-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.840 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.841 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.842 DEBUG:boto:Method: HEAD 2022-09-13T07:39:44.842 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:44.843 DEBUG:boto:Data: 2022-09-13T07:39:44.843 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.844 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.844 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.845 DEBUG:boto:Params: {} 2022-09-13T07:39:44.846 DEBUG:boto:Token: None 2022-09-13T07:39:44.846 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj 2022-09-13T07:39:44.847 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4= 2022-09-13T07:39:44.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LVvdCz71MBGX6z8wdBIz0e5Azl4=', 'Content-Length': '0'} 2022-09-13T07:39:44.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001dc7d19ddc0796d-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:44.853 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.854 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.854 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.855 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.856 DEBUG:boto:Method: GET 2022-09-13T07:39:44.857 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.858 DEBUG:boto:Data: 2022-09-13T07:39:44.859 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.860 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.861 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.862 DEBUG:boto:Params: {} 2022-09-13T07:39:44.863 DEBUG:boto:Token: None 2022-09-13T07:39:44.864 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.865 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI= 2022-09-13T07:39:44.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI=', 'Content-Length': '0'} 2022-09-13T07:39:44.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001dac770cd4c670d8-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.870 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.871 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.871 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.872 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.872 DEBUG:boto:Method: GET 2022-09-13T07:39:44.873 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.873 DEBUG:boto:Data: 2022-09-13T07:39:44.874 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.874 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.875 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.875 DEBUG:boto:Params: {} 2022-09-13T07:39:44.876 DEBUG:boto:Token: None 2022-09-13T07:39:44.877 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:44.877 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI= 2022-09-13T07:39:44.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cCVGCgKvXInIu9aNEIgzMLRnQYI=', 'Content-Length': '0'} 2022-09-13T07:39:44.880 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8502f0e8f45740f-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.926 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.927 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.928 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.929 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.929 DEBUG:boto:Method: GET 2022-09-13T07:39:44.930 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.932 DEBUG:boto:Data: 2022-09-13T07:39:44.932 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.933 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.934 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.936 DEBUG:boto:Params: {} 2022-09-13T07:39:44.938 DEBUG:boto:Token: None 2022-09-13T07:39:44.939 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.940 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000794b57381d6ae47e-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.946 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.947 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:44.948 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:44.949 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.950 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.952 DEBUG:boto:Method: GET 2022-09-13T07:39:44.953 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:44.954 DEBUG:boto:Data: 2022-09-13T07:39:44.955 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.955 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.956 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.957 DEBUG:boto:Params: {} 2022-09-13T07:39:44.958 DEBUG:boto:Token: None 2022-09-13T07:39:44.959 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/ 2022-09-13T07:39:44.959 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA= 2022-09-13T07:39:44.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TgU1ueGF6XeAdK33iLhQPJerPsA=', 'Content-Length': '0'} 2022-09-13T07:39:44.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e64533ce79876386-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.964 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:44.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:44.966 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.966 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.967 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:44.967 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:44.968 DEBUG:boto:Method: GET 2022-09-13T07:39:44.968 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:44.969 DEBUG:boto:Data: 2022-09-13T07:39:44.969 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.970 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:44.971 DEBUG:boto:Port: 8001 2022-09-13T07:39:44.971 DEBUG:boto:Params: {} 2022-09-13T07:39:44.972 DEBUG:boto:Token: None 2022-09-13T07:39:44.973 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:44.973 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MBDVxj/LfzGwHFjDSngGy0UP/m0= 2022-09-13T07:39:44.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MBDVxj/LfzGwHFjDSngGy0UP/m0=', 'Content-Length': '0'} 2022-09-13T07:39:44.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bf0f3d0006a9a7e-00632033c0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.980 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.981 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.982 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:44.983 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:44.984 DEBUG:boto:Method: GET 2022-09-13T07:39:44.984 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:44.985 DEBUG:boto:Data: 2022-09-13T07:39:44.985 DEBUG:boto:Headers: {} 2022-09-13T07:39:44.986 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:44.987 DEBUG:boto:Port: 8000 2022-09-13T07:39:44.988 DEBUG:boto:Params: {} 2022-09-13T07:39:44.989 DEBUG:boto:Token: None 2022-09-13T07:39:44.990 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:44 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:44.991 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MBDVxj/LfzGwHFjDSngGy0UP/m0= 2022-09-13T07:39:44.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:44 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MBDVxj/LfzGwHFjDSngGy0UP/m0=', 'Content-Length': '0'} 2022-09-13T07:39:44.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f031c0cc640e2920-00632033c0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:44.996 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:44.997 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:44.998 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:44.999 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:44.999 DEBUG:boto:Method: HEAD 2022-09-13T07:39:45.000 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:45.000 DEBUG:boto:Data: 2022-09-13T07:39:45.001 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.001 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.002 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.002 DEBUG:boto:Params: {} 2022-09-13T07:39:45.003 DEBUG:boto:Token: None 2022-09-13T07:39:45.003 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:45.004 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EkWAs8jVL2HXZQSeYZEtMF1aaY0= 2022-09-13T07:39:45.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EkWAs8jVL2HXZQSeYZEtMF1aaY0=', 'Content-Length': '0'} 2022-09-13T07:39:45.007 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000396a0ac15c28de9f-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.007 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.008 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.008 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:45.009 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:45.009 DEBUG:boto:Method: HEAD 2022-09-13T07:39:45.010 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:45.010 DEBUG:boto:Data: 2022-09-13T07:39:45.011 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.011 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.012 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.012 DEBUG:boto:Params: {} 2022-09-13T07:39:45.013 DEBUG:boto:Token: None 2022-09-13T07:39:45.013 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:45.014 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EkWAs8jVL2HXZQSeYZEtMF1aaY0= 2022-09-13T07:39:45.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EkWAs8jVL2HXZQSeYZEtMF1aaY0=', 'Content-Length': '0'} 2022-09-13T07:39:45.017 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001aeb7b31c81c8fb7-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:45.020 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.020 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.021 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:45.022 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:45.023 DEBUG:boto:Method: GET 2022-09-13T07:39:45.024 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:45.025 DEBUG:boto:Data: 2022-09-13T07:39:45.026 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.026 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.027 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.027 DEBUG:boto:Params: {} 2022-09-13T07:39:45.028 DEBUG:boto:Token: None 2022-09-13T07:39:45.028 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:45.029 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Ecus89x2PlW2T9fESbf7ZbDlRXI= 2022-09-13T07:39:45.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Ecus89x2PlW2T9fESbf7ZbDlRXI=', 'Content-Length': '0'} 2022-09-13T07:39:45.032 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008feb40f77c41cd35-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.074 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.075 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.076 DEBUG:boto:path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:45.077 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:45.078 DEBUG:boto:Method: GET 2022-09-13T07:39:45.079 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:45.080 DEBUG:boto:Data: 2022-09-13T07:39:45.081 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.082 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.083 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.084 DEBUG:boto:Params: {} 2022-09-13T07:39:45.085 DEBUG:boto:Token: None 2022-09-13T07:39:45.086 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=i8qJKwt5yOnRkOjqtadujgkvccCo7V6 2022-09-13T07:39:45.087 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Ecus89x2PlW2T9fESbf7ZbDlRXI= 2022-09-13T07:39:45.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Ecus89x2PlW2T9fESbf7ZbDlRXI=', 'Content-Length': '0'} 2022-09-13T07:39:45.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'i8qJKwt5yOnRkOjqtadujgkvccCo7V6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bcccd8e16651dec-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.092 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:45.092 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:45.093 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.093 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.094 DEBUG:boto:Method: GET 2022-09-13T07:39:45.094 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.095 DEBUG:boto:Data: 2022-09-13T07:39:45.096 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.096 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.097 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.097 DEBUG:boto:Params: {} 2022-09-13T07:39:45.098 DEBUG:boto:Token: None 2022-09-13T07:39:45.099 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/ 2022-09-13T07:39:45.099 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8= 2022-09-13T07:39:45.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8=', 'Content-Length': '0'} 2022-09-13T07:39:45.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a147e2f7e7eef4a9-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.104 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:45.104 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:45.105 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:45.105 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.106 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.106 DEBUG:boto:Method: GET 2022-09-13T07:39:45.107 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.107 DEBUG:boto:Data: 2022-09-13T07:39:45.108 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.108 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.109 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.109 DEBUG:boto:Params: {} 2022-09-13T07:39:45.110 DEBUG:boto:Token: None 2022-09-13T07:39:45.111 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/ 2022-09-13T07:39:45.111 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8= 2022-09-13T07:39:45.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8=', 'Content-Length': '0'} 2022-09-13T07:39:45.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac97eee0232a6af9-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.115 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:45.116 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.116 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.117 DEBUG:boto:Method: HEAD 2022-09-13T07:39:45.117 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:45.118 DEBUG:boto:Data: 2022-09-13T07:39:45.118 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.119 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.119 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.120 DEBUG:boto:Params: {} 2022-09-13T07:39:45.120 DEBUG:boto:Token: None 2022-09-13T07:39:45.121 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj 2022-09-13T07:39:45.121 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VX/t8fehQ7T9viP9e4cpNmQV6qg= 2022-09-13T07:39:45.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VX/t8fehQ7T9viP9e4cpNmQV6qg=', 'Content-Length': '0'} 2022-09-13T07:39:45.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d4102a4f925f4d2-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.126 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.127 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.127 DEBUG:boto:Method: HEAD 2022-09-13T07:39:45.128 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:45.128 DEBUG:boto:Data: 2022-09-13T07:39:45.129 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.129 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.130 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.130 DEBUG:boto:Params: {} 2022-09-13T07:39:45.131 DEBUG:boto:Token: None 2022-09-13T07:39:45.131 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj 2022-09-13T07:39:45.132 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VX/t8fehQ7T9viP9e4cpNmQV6qg= 2022-09-13T07:39:45.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VX/t8fehQ7T9viP9e4cpNmQV6qg=', 'Content-Length': '0'} 2022-09-13T07:39:45.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc319466054889d5-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:45.137 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.138 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.138 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.139 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.139 DEBUG:boto:Method: GET 2022-09-13T07:39:45.140 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.140 DEBUG:boto:Data: 2022-09-13T07:39:45.141 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.141 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.142 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.142 DEBUG:boto:Params: {} 2022-09-13T07:39:45.143 DEBUG:boto:Token: None 2022-09-13T07:39:45.143 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.144 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WLnL1SjSbRk9El/h+h8FFCV6twk= 2022-09-13T07:39:45.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WLnL1SjSbRk9El/h+h8FFCV6twk=', 'Content-Length': '0'} 2022-09-13T07:39:45.147 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001743ddca741d158c-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.148 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.148 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.149 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.149 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.150 DEBUG:boto:Method: GET 2022-09-13T07:39:45.150 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.151 DEBUG:boto:Data: 2022-09-13T07:39:45.151 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.152 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.152 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.153 DEBUG:boto:Params: {} 2022-09-13T07:39:45.153 DEBUG:boto:Token: None 2022-09-13T07:39:45.154 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.154 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WLnL1SjSbRk9El/h+h8FFCV6twk= 2022-09-13T07:39:45.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WLnL1SjSbRk9El/h+h8FFCV6twk=', 'Content-Length': '0'} 2022-09-13T07:39:45.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b37642e36471e82-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.198 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:45.199 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:45.199 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.200 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.201 DEBUG:boto:Method: GET 2022-09-13T07:39:45.202 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.202 DEBUG:boto:Data: 2022-09-13T07:39:45.203 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.204 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.205 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.206 DEBUG:boto:Params: {} 2022-09-13T07:39:45.207 DEBUG:boto:Token: None 2022-09-13T07:39:45.208 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/ 2022-09-13T07:39:45.208 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8= 2022-09-13T07:39:45.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8=', 'Content-Length': '0'} 2022-09-13T07:39:45.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a6ea2c2014617e7-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.213 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:45.214 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:45.214 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:45.215 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.215 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.216 DEBUG:boto:Method: GET 2022-09-13T07:39:45.216 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.217 DEBUG:boto:Data: 2022-09-13T07:39:45.217 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.218 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.218 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.219 DEBUG:boto:Params: {} 2022-09-13T07:39:45.219 DEBUG:boto:Token: None 2022-09-13T07:39:45.220 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/ 2022-09-13T07:39:45.221 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8= 2022-09-13T07:39:45.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8=', 'Content-Length': '0'} 2022-09-13T07:39:45.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f08b6cf6e137526-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.225 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:45.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:45.226 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.227 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.228 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.228 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.229 DEBUG:boto:Method: GET 2022-09-13T07:39:45.230 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.230 DEBUG:boto:Data: 2022-09-13T07:39:45.231 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.232 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.232 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.233 DEBUG:boto:Params: {} 2022-09-13T07:39:45.233 DEBUG:boto:Token: None 2022-09-13T07:39:45.234 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.235 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/KKHjXi61xKBdCuk9IGbTe0IvSU= 2022-09-13T07:39:45.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/KKHjXi61xKBdCuk9IGbTe0IvSU=', 'Content-Length': '0'} 2022-09-13T07:39:45.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc8970df9480b374-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.289 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.290 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.291 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.292 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.292 DEBUG:boto:Method: GET 2022-09-13T07:39:45.293 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.294 DEBUG:boto:Data: 2022-09-13T07:39:45.295 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.296 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.297 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.298 DEBUG:boto:Params: {} 2022-09-13T07:39:45.301 DEBUG:boto:Token: None 2022-09-13T07:39:45.302 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.303 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/KKHjXi61xKBdCuk9IGbTe0IvSU= 2022-09-13T07:39:45.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/KKHjXi61xKBdCuk9IGbTe0IvSU=', 'Content-Length': '0'} 2022-09-13T07:39:45.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050cf180dd52d7788-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.310 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.311 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.312 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.312 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.313 DEBUG:boto:Method: HEAD 2022-09-13T07:39:45.313 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.314 DEBUG:boto:Data: 2022-09-13T07:39:45.315 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.315 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.316 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.317 DEBUG:boto:Params: {} 2022-09-13T07:39:45.318 DEBUG:boto:Token: None 2022-09-13T07:39:45.319 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.320 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6LzmKohhVj56qg3dZf8s4ekzcEM= 2022-09-13T07:39:45.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6LzmKohhVj56qg3dZf8s4ekzcEM=', 'Content-Length': '0'} 2022-09-13T07:39:45.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ebbe965a2c0b47f6-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.324 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.325 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.326 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.327 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.328 DEBUG:boto:Method: HEAD 2022-09-13T07:39:45.328 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.328 DEBUG:boto:Data: 2022-09-13T07:39:45.329 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.330 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.330 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.331 DEBUG:boto:Params: {} 2022-09-13T07:39:45.332 DEBUG:boto:Token: None 2022-09-13T07:39:45.332 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.333 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6LzmKohhVj56qg3dZf8s4ekzcEM= 2022-09-13T07:39:45.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6LzmKohhVj56qg3dZf8s4ekzcEM=', 'Content-Length': '0'} 2022-09-13T07:39:45.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfd50f35fa61d991-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:45.338 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.339 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.340 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.341 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.342 DEBUG:boto:Method: GET 2022-09-13T07:39:45.342 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.343 DEBUG:boto:Data: 2022-09-13T07:39:45.344 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.345 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.346 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.347 DEBUG:boto:Params: {} 2022-09-13T07:39:45.348 DEBUG:boto:Token: None 2022-09-13T07:39:45.349 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.350 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/KKHjXi61xKBdCuk9IGbTe0IvSU= 2022-09-13T07:39:45.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/KKHjXi61xKBdCuk9IGbTe0IvSU=', 'Content-Length': '0'} 2022-09-13T07:39:45.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a45046749354d8c-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.356 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.357 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.357 DEBUG:boto:path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.358 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.359 DEBUG:boto:Method: GET 2022-09-13T07:39:45.360 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.361 DEBUG:boto:Data: 2022-09-13T07:39:45.362 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.362 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.363 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.363 DEBUG:boto:Params: {} 2022-09-13T07:39:45.364 DEBUG:boto:Token: None 2022-09-13T07:39:45.365 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB 2022-09-13T07:39:45.365 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/KKHjXi61xKBdCuk9IGbTe0IvSU= 2022-09-13T07:39:45.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/KKHjXi61xKBdCuk9IGbTe0IvSU=', 'Content-Length': '0'} 2022-09-13T07:39:45.369 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '3qFRMPwv7vgGcoQgHQ0mrTC8ozSP0WB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f0a029078370d88-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.371 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:45.372 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:45.372 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.373 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.374 DEBUG:boto:Method: GET 2022-09-13T07:39:45.374 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.375 DEBUG:boto:Data: 2022-09-13T07:39:45.375 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.376 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.376 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.377 DEBUG:boto:Params: {} 2022-09-13T07:39:45.378 DEBUG:boto:Token: None 2022-09-13T07:39:45.378 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/ 2022-09-13T07:39:45.379 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8= 2022-09-13T07:39:45.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8=', 'Content-Length': '0'} 2022-09-13T07:39:45.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060ca6eac1127ad3e-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.383 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:45.384 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:45.385 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:45.386 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.386 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.387 DEBUG:boto:Method: GET 2022-09-13T07:39:45.387 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.388 DEBUG:boto:Data: 2022-09-13T07:39:45.388 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.389 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.389 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.390 DEBUG:boto:Params: {} 2022-09-13T07:39:45.391 DEBUG:boto:Token: None 2022-09-13T07:39:45.391 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/ 2022-09-13T07:39:45.392 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8= 2022-09-13T07:39:45.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8=', 'Content-Length': '0'} 2022-09-13T07:39:45.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006abf9150e2be2a27-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.396 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:45.397 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.398 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.398 DEBUG:boto:Method: HEAD 2022-09-13T07:39:45.399 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:45.399 DEBUG:boto:Data: 2022-09-13T07:39:45.400 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.400 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.401 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.401 DEBUG:boto:Params: {} 2022-09-13T07:39:45.402 DEBUG:boto:Token: None 2022-09-13T07:39:45.402 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj 2022-09-13T07:39:45.403 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VX/t8fehQ7T9viP9e4cpNmQV6qg= 2022-09-13T07:39:45.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VX/t8fehQ7T9viP9e4cpNmQV6qg=', 'Content-Length': '0'} 2022-09-13T07:39:45.407 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000327b89d0cf3c807e-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.408 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.409 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.410 DEBUG:boto:Method: HEAD 2022-09-13T07:39:45.410 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:45.411 DEBUG:boto:Data: 2022-09-13T07:39:45.412 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.413 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.414 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.415 DEBUG:boto:Params: {} 2022-09-13T07:39:45.416 DEBUG:boto:Token: None 2022-09-13T07:39:45.417 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj 2022-09-13T07:39:45.418 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VX/t8fehQ7T9viP9e4cpNmQV6qg= 2022-09-13T07:39:45.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VX/t8fehQ7T9viP9e4cpNmQV6qg=', 'Content-Length': '0'} 2022-09-13T07:39:45.423 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2e879180d2cfd3a-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:45.425 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.426 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.427 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.428 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.429 DEBUG:boto:Method: GET 2022-09-13T07:39:45.430 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.431 DEBUG:boto:Data: 2022-09-13T07:39:45.432 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.433 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.434 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.435 DEBUG:boto:Params: {} 2022-09-13T07:39:45.436 DEBUG:boto:Token: None 2022-09-13T07:39:45.437 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.438 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WLnL1SjSbRk9El/h+h8FFCV6twk= 2022-09-13T07:39:45.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WLnL1SjSbRk9El/h+h8FFCV6twk=', 'Content-Length': '0'} 2022-09-13T07:39:45.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f7009530b28def2-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.482 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.483 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.484 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.484 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.485 DEBUG:boto:Method: GET 2022-09-13T07:39:45.485 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.486 DEBUG:boto:Data: 2022-09-13T07:39:45.486 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.487 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.488 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.488 DEBUG:boto:Params: {} 2022-09-13T07:39:45.489 DEBUG:boto:Token: None 2022-09-13T07:39:45.490 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.491 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WLnL1SjSbRk9El/h+h8FFCV6twk= 2022-09-13T07:39:45.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WLnL1SjSbRk9El/h+h8FFCV6twk=', 'Content-Length': '0'} 2022-09-13T07:39:45.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2ff3b1e990b1b3b-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.495 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:45.495 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:45.496 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.496 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.497 DEBUG:boto:Method: GET 2022-09-13T07:39:45.497 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.498 DEBUG:boto:Data: 2022-09-13T07:39:45.498 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.499 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.499 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.500 DEBUG:boto:Params: {} 2022-09-13T07:39:45.500 DEBUG:boto:Token: None 2022-09-13T07:39:45.501 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/ 2022-09-13T07:39:45.501 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8= 2022-09-13T07:39:45.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8=', 'Content-Length': '0'} 2022-09-13T07:39:45.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002baa0cafa3a62950-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.506 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:45.507 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:45.509 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:45.509 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.510 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.510 DEBUG:boto:Method: GET 2022-09-13T07:39:45.511 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.511 DEBUG:boto:Data: 2022-09-13T07:39:45.512 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.512 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.513 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.513 DEBUG:boto:Params: {} 2022-09-13T07:39:45.514 DEBUG:boto:Token: None 2022-09-13T07:39:45.515 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/ 2022-09-13T07:39:45.515 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8= 2022-09-13T07:39:45.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8=', 'Content-Length': '0'} 2022-09-13T07:39:45.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b20a008e9543142-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.520 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:45.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:45.521 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.522 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.523 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.523 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.524 DEBUG:boto:Method: GET 2022-09-13T07:39:45.524 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.525 DEBUG:boto:Data: 2022-09-13T07:39:45.525 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.526 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.526 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.527 DEBUG:boto:Params: {} 2022-09-13T07:39:45.527 DEBUG:boto:Token: None 2022-09-13T07:39:45.528 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.528 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PUUCNEbrq0gBMslro1PybVxcG1Y= 2022-09-13T07:39:45.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PUUCNEbrq0gBMslro1PybVxcG1Y=', 'Content-Length': '0'} 2022-09-13T07:39:45.531 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000156eebe374c5f1e5-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.532 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.533 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.533 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.534 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.534 DEBUG:boto:Method: GET 2022-09-13T07:39:45.535 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.535 DEBUG:boto:Data: 2022-09-13T07:39:45.536 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.536 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.537 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.537 DEBUG:boto:Params: {} 2022-09-13T07:39:45.538 DEBUG:boto:Token: None 2022-09-13T07:39:45.538 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.539 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PUUCNEbrq0gBMslro1PybVxcG1Y= 2022-09-13T07:39:45.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PUUCNEbrq0gBMslro1PybVxcG1Y=', 'Content-Length': '0'} 2022-09-13T07:39:45.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c653e5b442ceddf3-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.543 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.544 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.544 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.545 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.546 DEBUG:boto:Method: HEAD 2022-09-13T07:39:45.547 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.548 DEBUG:boto:Data: 2022-09-13T07:39:45.549 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.549 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.550 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.551 DEBUG:boto:Params: {} 2022-09-13T07:39:45.552 DEBUG:boto:Token: None 2022-09-13T07:39:45.552 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.553 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UZkWN+GCVOnvJVmJ129YjamjuNQ= 2022-09-13T07:39:45.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UZkWN+GCVOnvJVmJ129YjamjuNQ=', 'Content-Length': '0'} 2022-09-13T07:39:45.557 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e9238eae9657868-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.558 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.559 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.560 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.561 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.562 DEBUG:boto:Method: HEAD 2022-09-13T07:39:45.563 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.563 DEBUG:boto:Data: 2022-09-13T07:39:45.564 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.565 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.566 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.567 DEBUG:boto:Params: {} 2022-09-13T07:39:45.568 DEBUG:boto:Token: None 2022-09-13T07:39:45.569 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.570 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UZkWN+GCVOnvJVmJ129YjamjuNQ= 2022-09-13T07:39:45.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UZkWN+GCVOnvJVmJ129YjamjuNQ=', 'Content-Length': '0'} 2022-09-13T07:39:45.574 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef51835163c671ac-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:45.576 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.577 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.578 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.579 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.580 DEBUG:boto:Method: GET 2022-09-13T07:39:45.581 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.582 DEBUG:boto:Data: 2022-09-13T07:39:45.583 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.583 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.584 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.585 DEBUG:boto:Params: {} 2022-09-13T07:39:45.586 DEBUG:boto:Token: None 2022-09-13T07:39:45.587 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.589 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PUUCNEbrq0gBMslro1PybVxcG1Y= 2022-09-13T07:39:45.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PUUCNEbrq0gBMslro1PybVxcG1Y=', 'Content-Length': '0'} 2022-09-13T07:39:45.592 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a49267017d0a40ee-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.594 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.595 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.595 DEBUG:boto:path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.596 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.597 DEBUG:boto:Method: GET 2022-09-13T07:39:45.598 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.599 DEBUG:boto:Data: 2022-09-13T07:39:45.599 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.600 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.600 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.601 DEBUG:boto:Params: {} 2022-09-13T07:39:45.601 DEBUG:boto:Token: None 2022-09-13T07:39:45.602 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5 2022-09-13T07:39:45.602 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PUUCNEbrq0gBMslro1PybVxcG1Y= 2022-09-13T07:39:45.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PUUCNEbrq0gBMslro1PybVxcG1Y=', 'Content-Length': '0'} 2022-09-13T07:39:45.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'tLDy1U4ZTxWriIUMx30zfLE7zPcVUt5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006badabbaff160bbf-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.606 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:45.607 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:45.607 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.608 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.608 DEBUG:boto:Method: GET 2022-09-13T07:39:45.609 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.609 DEBUG:boto:Data: 2022-09-13T07:39:45.610 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.610 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.611 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.611 DEBUG:boto:Params: {} 2022-09-13T07:39:45.612 DEBUG:boto:Token: None 2022-09-13T07:39:45.612 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/ 2022-09-13T07:39:45.613 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8= 2022-09-13T07:39:45.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8=', 'Content-Length': '0'} 2022-09-13T07:39:45.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000910ff68719c6b85c-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.617 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:45.618 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:45.619 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:45.619 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.619 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.620 DEBUG:boto:Method: GET 2022-09-13T07:39:45.621 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.621 DEBUG:boto:Data: 2022-09-13T07:39:45.621 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.622 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.623 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.623 DEBUG:boto:Params: {} 2022-09-13T07:39:45.624 DEBUG:boto:Token: None 2022-09-13T07:39:45.624 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/ 2022-09-13T07:39:45.625 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8= 2022-09-13T07:39:45.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8=', 'Content-Length': '0'} 2022-09-13T07:39:45.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000547cd21acb99ea10-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.629 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:45.630 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.630 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.631 DEBUG:boto:Method: HEAD 2022-09-13T07:39:45.631 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:45.632 DEBUG:boto:Data: 2022-09-13T07:39:45.632 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.633 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.633 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.634 DEBUG:boto:Params: {} 2022-09-13T07:39:45.634 DEBUG:boto:Token: None 2022-09-13T07:39:45.635 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj 2022-09-13T07:39:45.635 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VX/t8fehQ7T9viP9e4cpNmQV6qg= 2022-09-13T07:39:45.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VX/t8fehQ7T9viP9e4cpNmQV6qg=', 'Content-Length': '0'} 2022-09-13T07:39:45.639 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003156049ac7395bb5-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.640 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.641 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.641 DEBUG:boto:Method: HEAD 2022-09-13T07:39:45.642 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:45.642 DEBUG:boto:Data: 2022-09-13T07:39:45.643 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.643 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.643 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.644 DEBUG:boto:Params: {} 2022-09-13T07:39:45.645 DEBUG:boto:Token: None 2022-09-13T07:39:45.645 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj 2022-09-13T07:39:45.646 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VX/t8fehQ7T9viP9e4cpNmQV6qg= 2022-09-13T07:39:45.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VX/t8fehQ7T9viP9e4cpNmQV6qg=', 'Content-Length': '0'} 2022-09-13T07:39:45.651 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a87af42b9e85fa4-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:45.652 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.653 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.654 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.654 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.655 DEBUG:boto:Method: GET 2022-09-13T07:39:45.655 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.656 DEBUG:boto:Data: 2022-09-13T07:39:45.656 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.657 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.657 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.658 DEBUG:boto:Params: {} 2022-09-13T07:39:45.659 DEBUG:boto:Token: None 2022-09-13T07:39:45.660 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.660 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WLnL1SjSbRk9El/h+h8FFCV6twk= 2022-09-13T07:39:45.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WLnL1SjSbRk9El/h+h8FFCV6twk=', 'Content-Length': '0'} 2022-09-13T07:39:45.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e16c03d0ef308520-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.666 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.667 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.668 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.669 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.670 DEBUG:boto:Method: GET 2022-09-13T07:39:45.671 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.671 DEBUG:boto:Data: 2022-09-13T07:39:45.672 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.673 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.674 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.675 DEBUG:boto:Params: {} 2022-09-13T07:39:45.676 DEBUG:boto:Token: None 2022-09-13T07:39:45.677 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:45.678 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WLnL1SjSbRk9El/h+h8FFCV6twk= 2022-09-13T07:39:45.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WLnL1SjSbRk9El/h+h8FFCV6twk=', 'Content-Length': '0'} 2022-09-13T07:39:45.683 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000590049dcdfe2d2c5-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.685 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:45.686 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:45.687 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.688 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.689 DEBUG:boto:Method: GET 2022-09-13T07:39:45.690 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.691 DEBUG:boto:Data: 2022-09-13T07:39:45.692 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.693 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.694 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.695 DEBUG:boto:Params: {} 2022-09-13T07:39:45.696 DEBUG:boto:Token: None 2022-09-13T07:39:45.697 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/ 2022-09-13T07:39:45.699 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8= 2022-09-13T07:39:45.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8=', 'Content-Length': '0'} 2022-09-13T07:39:45.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faf32791778cb81a-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.705 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:45.706 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:45.706 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:45.707 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.708 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.708 DEBUG:boto:Method: GET 2022-09-13T07:39:45.709 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:45.709 DEBUG:boto:Data: 2022-09-13T07:39:45.710 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.710 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:45.711 DEBUG:boto:Port: 8000 2022-09-13T07:39:45.712 DEBUG:boto:Params: {} 2022-09-13T07:39:45.712 DEBUG:boto:Token: None 2022-09-13T07:39:45.713 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/ 2022-09-13T07:39:45.713 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8= 2022-09-13T07:39:45.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:l2a8TLZFPLGkCY/ajg+YbpVr7C8=', 'Content-Length': '0'} 2022-09-13T07:39:45.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013d3293449201ca9-00632033c1-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.774 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:45.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:45.776 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.777 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.778 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:45.779 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:45.780 DEBUG:boto:Method: GET 2022-09-13T07:39:45.780 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:45.781 DEBUG:boto:Data: 2022-09-13T07:39:45.781 DEBUG:boto:Headers: {} 2022-09-13T07:39:45.782 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:45.782 DEBUG:boto:Port: 8001 2022-09-13T07:39:45.783 DEBUG:boto:Params: {} 2022-09-13T07:39:45.783 DEBUG:boto:Token: None 2022-09-13T07:39:45.784 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:45 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:45.785 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wq0slyyrKaEHTiiM/WsTNaj38JQ= 2022-09-13T07:39:45.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:45 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wq0slyyrKaEHTiiM/WsTNaj38JQ=', 'Content-Length': '0'} 2022-09-13T07:39:45.795 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002901c3b541a8dead-00632033c1-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:45.798 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:45.805 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:45.806 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:45.806 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:45.807 DEBUG:boto:Method: GET 2022-09-13T07:39:45.809 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.795 DEBUG:boto:Data: 2022-09-13T07:39:46.796 DEBUG:boto:Headers: {} 2022-09-13T07:39:46.797 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:46.798 DEBUG:boto:Port: 8000 2022-09-13T07:39:46.799 DEBUG:boto:Params: {} 2022-09-13T07:39:46.800 DEBUG:boto:Token: None 2022-09-13T07:39:46.801 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:46 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.802 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i6A3PWkp7Rtj2SwNk6mAaYH9140= 2022-09-13T07:39:46.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i6A3PWkp7Rtj2SwNk6mAaYH9140=', 'Content-Length': '0'} 2022-09-13T07:39:46.808 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fc2c92abb403e31-00632033c2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:46.809 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:46.810 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:46.811 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.811 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.812 DEBUG:boto:Method: HEAD 2022-09-13T07:39:46.813 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.813 DEBUG:boto:Data: 2022-09-13T07:39:46.814 DEBUG:boto:Headers: {} 2022-09-13T07:39:46.814 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:46.815 DEBUG:boto:Port: 8001 2022-09-13T07:39:46.815 DEBUG:boto:Params: {} 2022-09-13T07:39:46.816 DEBUG:boto:Token: None 2022-09-13T07:39:46.817 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:46 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.818 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Lmq2K44iAw+6TTCt8pL5lQzQdCw= 2022-09-13T07:39:46.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Lmq2K44iAw+6TTCt8pL5lQzQdCw=', 'Content-Length': '0'} 2022-09-13T07:39:46.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000604eabad989913ae-00632033c2-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:46.826 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:46.827 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:46.828 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.828 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.829 DEBUG:boto:Method: HEAD 2022-09-13T07:39:46.829 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.830 DEBUG:boto:Data: 2022-09-13T07:39:46.830 DEBUG:boto:Headers: {} 2022-09-13T07:39:46.831 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:46.832 DEBUG:boto:Port: 8000 2022-09-13T07:39:46.833 DEBUG:boto:Params: {} 2022-09-13T07:39:46.834 DEBUG:boto:Token: None 2022-09-13T07:39:46.835 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:46 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.836 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Lmq2K44iAw+6TTCt8pL5lQzQdCw= 2022-09-13T07:39:46.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Lmq2K44iAw+6TTCt8pL5lQzQdCw=', 'Content-Length': '0'} 2022-09-13T07:39:46.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da57f57aa00e5702-00632033c2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:46.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:46.845 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:46.846 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:46.847 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.847 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.848 DEBUG:boto:Method: GET 2022-09-13T07:39:46.849 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.850 DEBUG:boto:Data: 2022-09-13T07:39:46.851 DEBUG:boto:Headers: {} 2022-09-13T07:39:46.852 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:46.853 DEBUG:boto:Port: 8001 2022-09-13T07:39:46.855 DEBUG:boto:Params: {} 2022-09-13T07:39:46.856 DEBUG:boto:Token: None 2022-09-13T07:39:46.857 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:46 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.858 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i6A3PWkp7Rtj2SwNk6mAaYH9140= 2022-09-13T07:39:46.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i6A3PWkp7Rtj2SwNk6mAaYH9140=', 'Content-Length': '0'} 2022-09-13T07:39:46.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003eb3e61e2ec2e3ee-00632033c2-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:46.906 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:46.908 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:46.909 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.911 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.911 DEBUG:boto:Method: GET 2022-09-13T07:39:46.912 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.912 DEBUG:boto:Data: 2022-09-13T07:39:46.913 DEBUG:boto:Headers: {} 2022-09-13T07:39:46.913 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:46.914 DEBUG:boto:Port: 8000 2022-09-13T07:39:46.915 DEBUG:boto:Params: {} 2022-09-13T07:39:46.916 DEBUG:boto:Token: None 2022-09-13T07:39:46.918 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:46 GMT /yxvvuh-19/obj?versionId=ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m 2022-09-13T07:39:46.918 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i6A3PWkp7Rtj2SwNk6mAaYH9140= 2022-09-13T07:39:46.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i6A3PWkp7Rtj2SwNk6mAaYH9140=', 'Content-Length': '0'} 2022-09-13T07:39:46.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'ulPqaKM9GsnZBkMYt.uabOf.VJ5P.5m'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ec5d56601653f79-00632033c2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:46.929 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:46.931 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:46.932 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:46.933 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:46.934 DEBUG:boto:Method: GET 2022-09-13T07:39:46.935 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:46.936 DEBUG:boto:Data: 2022-09-13T07:39:46.937 DEBUG:boto:Headers: {} 2022-09-13T07:39:46.938 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:46.939 DEBUG:boto:Port: 8001 2022-09-13T07:39:46.940 DEBUG:boto:Params: {} 2022-09-13T07:39:46.942 DEBUG:boto:Token: None 2022-09-13T07:39:46.942 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:46 GMT /yxvvuh-19/ 2022-09-13T07:39:46.943 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2E8HBmuSr0VcxlfcHEJc7MKXncA= 2022-09-13T07:39:46.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2E8HBmuSr0VcxlfcHEJc7MKXncA=', 'Content-Length': '0'} 2022-09-13T07:39:46.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005021d28ef8876320-00632033c2-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:46.951 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:46.952 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:46.953 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:46.953 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:46.955 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:46.957 DEBUG:boto:Method: GET 2022-09-13T07:39:46.958 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:46.960 DEBUG:boto:Data: 2022-09-13T07:39:46.961 DEBUG:boto:Headers: {} 2022-09-13T07:39:46.962 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:46.964 DEBUG:boto:Port: 8000 2022-09-13T07:39:46.965 DEBUG:boto:Params: {} 2022-09-13T07:39:46.966 DEBUG:boto:Token: None 2022-09-13T07:39:46.967 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:46 GMT /yxvvuh-19/ 2022-09-13T07:39:46.968 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2E8HBmuSr0VcxlfcHEJc7MKXncA= 2022-09-13T07:39:46.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2E8HBmuSr0VcxlfcHEJc7MKXncA=', 'Content-Length': '0'} 2022-09-13T07:39:46.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000530903139d2f39e7-00632033c2-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:46.973 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:46.975 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:46.976 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:46.976 DEBUG:boto:Method: HEAD 2022-09-13T07:39:46.977 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:46.978 DEBUG:boto:Data: 2022-09-13T07:39:46.979 DEBUG:boto:Headers: {} 2022-09-13T07:39:46.979 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:46.980 DEBUG:boto:Port: 8001 2022-09-13T07:39:46.981 DEBUG:boto:Params: {} 2022-09-13T07:39:46.982 DEBUG:boto:Token: None 2022-09-13T07:39:46.983 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:46 GMT /yxvvuh-19/obj 2022-09-13T07:39:46.984 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YWdyAayyDVAIqOV9J2e5BIHk+go= 2022-09-13T07:39:46.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YWdyAayyDVAIqOV9J2e5BIHk+go=', 'Content-Length': '0'} 2022-09-13T07:39:46.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000187314b8f4df0334-00632033c2-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:46.991 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:46.992 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:46.992 DEBUG:boto:Method: HEAD 2022-09-13T07:39:46.993 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:46.994 DEBUG:boto:Data: 2022-09-13T07:39:46.994 DEBUG:boto:Headers: {} 2022-09-13T07:39:46.995 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:46.996 DEBUG:boto:Port: 8000 2022-09-13T07:39:46.997 DEBUG:boto:Params: {} 2022-09-13T07:39:46.998 DEBUG:boto:Token: None 2022-09-13T07:39:46.999 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:46 GMT /yxvvuh-19/obj 2022-09-13T07:39:47.000 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YWdyAayyDVAIqOV9J2e5BIHk+go= 2022-09-13T07:39:47.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YWdyAayyDVAIqOV9J2e5BIHk+go=', 'Content-Length': '0'} 2022-09-13T07:39:47.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077a7d65179316b09-00632033c2-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:47.006 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.007 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.008 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.008 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.009 DEBUG:boto:Method: GET 2022-09-13T07:39:47.009 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.010 DEBUG:boto:Data: 2022-09-13T07:39:47.011 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.011 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.012 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.013 DEBUG:boto:Params: {} 2022-09-13T07:39:47.013 DEBUG:boto:Token: None 2022-09-13T07:39:47.014 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.015 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wHmQN5yKSWjJXSMtZzQ92/SBQm0= 2022-09-13T07:39:47.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wHmQN5yKSWjJXSMtZzQ92/SBQm0=', 'Content-Length': '0'} 2022-09-13T07:39:47.018 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e5ec4a6e6070022-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.067 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.068 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.069 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.070 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.071 DEBUG:boto:Method: GET 2022-09-13T07:39:47.072 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.073 DEBUG:boto:Data: 2022-09-13T07:39:47.074 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.074 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.075 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.076 DEBUG:boto:Params: {} 2022-09-13T07:39:47.076 DEBUG:boto:Token: None 2022-09-13T07:39:47.077 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.078 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wHmQN5yKSWjJXSMtZzQ92/SBQm0= 2022-09-13T07:39:47.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wHmQN5yKSWjJXSMtZzQ92/SBQm0=', 'Content-Length': '0'} 2022-09-13T07:39:47.087 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a27757980854cdc6-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.089 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:47.090 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:47.091 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.092 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.092 DEBUG:boto:Method: GET 2022-09-13T07:39:47.093 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.094 DEBUG:boto:Data: 2022-09-13T07:39:47.095 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.096 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.097 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.098 DEBUG:boto:Params: {} 2022-09-13T07:39:47.099 DEBUG:boto:Token: None 2022-09-13T07:39:47.100 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/ 2022-09-13T07:39:47.101 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw= 2022-09-13T07:39:47.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw=', 'Content-Length': '0'} 2022-09-13T07:39:47.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab3518c4a908731a-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.116 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:47.117 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:47.118 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:47.120 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.121 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.123 DEBUG:boto:Method: GET 2022-09-13T07:39:47.124 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.125 DEBUG:boto:Data: 2022-09-13T07:39:47.126 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.127 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.128 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.130 DEBUG:boto:Params: {} 2022-09-13T07:39:47.131 DEBUG:boto:Token: None 2022-09-13T07:39:47.132 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/ 2022-09-13T07:39:47.133 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw= 2022-09-13T07:39:47.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw=', 'Content-Length': '0'} 2022-09-13T07:39:47.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000112e0428036dfed5-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.145 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:47.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:47.148 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.149 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.150 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.151 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.152 DEBUG:boto:Method: GET 2022-09-13T07:39:47.153 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.154 DEBUG:boto:Data: 2022-09-13T07:39:47.155 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.155 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.156 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.157 DEBUG:boto:Params: {} 2022-09-13T07:39:47.158 DEBUG:boto:Token: None 2022-09-13T07:39:47.159 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.161 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/XzvHlmIA/SizJUQj5remFHBlB0= 2022-09-13T07:39:47.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/XzvHlmIA/SizJUQj5remFHBlB0=', 'Content-Length': '0'} 2022-09-13T07:39:47.168 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b346346b3f20eea7-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.171 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.172 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.173 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.174 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.175 DEBUG:boto:Method: GET 2022-09-13T07:39:47.176 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.177 DEBUG:boto:Data: 2022-09-13T07:39:47.178 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.179 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.180 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.181 DEBUG:boto:Params: {} 2022-09-13T07:39:47.182 DEBUG:boto:Token: None 2022-09-13T07:39:47.183 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.184 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/XzvHlmIA/SizJUQj5remFHBlB0= 2022-09-13T07:39:47.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/XzvHlmIA/SizJUQj5remFHBlB0=', 'Content-Length': '0'} 2022-09-13T07:39:47.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a95e9c085f3c21f-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.193 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.194 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.195 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.196 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.197 DEBUG:boto:Method: HEAD 2022-09-13T07:39:47.198 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.199 DEBUG:boto:Data: 2022-09-13T07:39:47.200 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.200 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.201 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.202 DEBUG:boto:Params: {} 2022-09-13T07:39:47.203 DEBUG:boto:Token: None 2022-09-13T07:39:47.204 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.205 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4aDEChagVDnkVfNnoEqCcmALXN4= 2022-09-13T07:39:47.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4aDEChagVDnkVfNnoEqCcmALXN4=', 'Content-Length': '0'} 2022-09-13T07:39:47.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d909751893bd8a0-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.217 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.218 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.219 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.220 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.221 DEBUG:boto:Method: HEAD 2022-09-13T07:39:47.222 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.223 DEBUG:boto:Data: 2022-09-13T07:39:47.223 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.224 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.224 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.225 DEBUG:boto:Params: {} 2022-09-13T07:39:47.226 DEBUG:boto:Token: None 2022-09-13T07:39:47.226 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.227 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4aDEChagVDnkVfNnoEqCcmALXN4= 2022-09-13T07:39:47.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4aDEChagVDnkVfNnoEqCcmALXN4=', 'Content-Length': '0'} 2022-09-13T07:39:47.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a140cc66c5563bb8-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:47.232 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.233 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.233 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.234 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.234 DEBUG:boto:Method: GET 2022-09-13T07:39:47.235 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.235 DEBUG:boto:Data: 2022-09-13T07:39:47.236 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.236 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.237 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.237 DEBUG:boto:Params: {} 2022-09-13T07:39:47.238 DEBUG:boto:Token: None 2022-09-13T07:39:47.239 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.240 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/XzvHlmIA/SizJUQj5remFHBlB0= 2022-09-13T07:39:47.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/XzvHlmIA/SizJUQj5remFHBlB0=', 'Content-Length': '0'} 2022-09-13T07:39:47.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003327baa31dfe13e2-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.291 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.293 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.294 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.295 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.295 DEBUG:boto:Method: GET 2022-09-13T07:39:47.296 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.297 DEBUG:boto:Data: 2022-09-13T07:39:47.298 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.299 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.300 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.301 DEBUG:boto:Params: {} 2022-09-13T07:39:47.302 DEBUG:boto:Token: None 2022-09-13T07:39:47.303 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA 2022-09-13T07:39:47.304 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/XzvHlmIA/SizJUQj5remFHBlB0= 2022-09-13T07:39:47.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/XzvHlmIA/SizJUQj5remFHBlB0=', 'Content-Length': '0'} 2022-09-13T07:39:47.308 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', '8Q.mkCe9CuoAMoz-3WZm3yrTXxrG9eA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016bcf14dc2190385-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.309 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:47.310 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:47.311 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.311 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.312 DEBUG:boto:Method: GET 2022-09-13T07:39:47.313 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.313 DEBUG:boto:Data: 2022-09-13T07:39:47.314 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.315 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.316 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.316 DEBUG:boto:Params: {} 2022-09-13T07:39:47.317 DEBUG:boto:Token: None 2022-09-13T07:39:47.318 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/ 2022-09-13T07:39:47.319 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw= 2022-09-13T07:39:47.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw=', 'Content-Length': '0'} 2022-09-13T07:39:47.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c8a114086beb5b8-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.327 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:47.328 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:47.329 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:47.330 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.331 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.331 DEBUG:boto:Method: GET 2022-09-13T07:39:47.332 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.332 DEBUG:boto:Data: 2022-09-13T07:39:47.333 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.334 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.334 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.335 DEBUG:boto:Params: {} 2022-09-13T07:39:47.336 DEBUG:boto:Token: None 2022-09-13T07:39:47.338 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/ 2022-09-13T07:39:47.339 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw= 2022-09-13T07:39:47.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw=', 'Content-Length': '0'} 2022-09-13T07:39:47.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001332d5d6901ee8d8-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.344 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:47.345 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.346 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.347 DEBUG:boto:Method: HEAD 2022-09-13T07:39:47.347 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:47.348 DEBUG:boto:Data: 2022-09-13T07:39:47.348 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.349 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.349 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.350 DEBUG:boto:Params: {} 2022-09-13T07:39:47.350 DEBUG:boto:Token: None 2022-09-13T07:39:47.351 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj 2022-09-13T07:39:47.352 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:JyvT3WN0PygsqgIK1ljkFIUkEXc= 2022-09-13T07:39:47.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JyvT3WN0PygsqgIK1ljkFIUkEXc=', 'Content-Length': '0'} 2022-09-13T07:39:47.358 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e1fb082cc7c5f01-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.360 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.361 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.362 DEBUG:boto:Method: HEAD 2022-09-13T07:39:47.363 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:47.363 DEBUG:boto:Data: 2022-09-13T07:39:47.364 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.365 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.366 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.367 DEBUG:boto:Params: {} 2022-09-13T07:39:47.367 DEBUG:boto:Token: None 2022-09-13T07:39:47.369 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj 2022-09-13T07:39:47.369 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:JyvT3WN0PygsqgIK1ljkFIUkEXc= 2022-09-13T07:39:47.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JyvT3WN0PygsqgIK1ljkFIUkEXc=', 'Content-Length': '0'} 2022-09-13T07:39:47.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004efc5e8df95756a9-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:47.378 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.378 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.379 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.379 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.380 DEBUG:boto:Method: GET 2022-09-13T07:39:47.380 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.382 DEBUG:boto:Data: 2022-09-13T07:39:47.382 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.383 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.383 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.384 DEBUG:boto:Params: {} 2022-09-13T07:39:47.385 DEBUG:boto:Token: None 2022-09-13T07:39:47.385 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.386 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wHmQN5yKSWjJXSMtZzQ92/SBQm0= 2022-09-13T07:39:47.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wHmQN5yKSWjJXSMtZzQ92/SBQm0=', 'Content-Length': '0'} 2022-09-13T07:39:47.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2067c26c76a9c12-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.433 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.434 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.434 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.435 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.435 DEBUG:boto:Method: GET 2022-09-13T07:39:47.436 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.436 DEBUG:boto:Data: 2022-09-13T07:39:47.437 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.437 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.438 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.439 DEBUG:boto:Params: {} 2022-09-13T07:39:47.439 DEBUG:boto:Token: None 2022-09-13T07:39:47.440 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.440 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wHmQN5yKSWjJXSMtZzQ92/SBQm0= 2022-09-13T07:39:47.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wHmQN5yKSWjJXSMtZzQ92/SBQm0=', 'Content-Length': '0'} 2022-09-13T07:39:47.447 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000647b1a080a2591-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.449 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:47.450 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:47.451 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.452 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.452 DEBUG:boto:Method: GET 2022-09-13T07:39:47.453 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.454 DEBUG:boto:Data: 2022-09-13T07:39:47.454 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.455 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.456 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.456 DEBUG:boto:Params: {} 2022-09-13T07:39:47.457 DEBUG:boto:Token: None 2022-09-13T07:39:47.459 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/ 2022-09-13T07:39:47.459 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw= 2022-09-13T07:39:47.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw=', 'Content-Length': '0'} 2022-09-13T07:39:47.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dee9eca023d0dce-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.465 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:47.466 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:47.467 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:47.468 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.470 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.471 DEBUG:boto:Method: GET 2022-09-13T07:39:47.472 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.474 DEBUG:boto:Data: 2022-09-13T07:39:47.475 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.476 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.477 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.478 DEBUG:boto:Params: {} 2022-09-13T07:39:47.480 DEBUG:boto:Token: None 2022-09-13T07:39:47.481 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/ 2022-09-13T07:39:47.482 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw= 2022-09-13T07:39:47.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw=', 'Content-Length': '0'} 2022-09-13T07:39:47.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000409e5c8b37232a0e-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.534 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:47.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:47.536 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.537 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.538 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.538 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.539 DEBUG:boto:Method: GET 2022-09-13T07:39:47.540 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.540 DEBUG:boto:Data: 2022-09-13T07:39:47.541 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.542 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.542 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.543 DEBUG:boto:Params: {} 2022-09-13T07:39:47.544 DEBUG:boto:Token: None 2022-09-13T07:39:47.544 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.545 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ez5/MiG3SIhVrD2ybA6+GYsrLAA= 2022-09-13T07:39:47.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ez5/MiG3SIhVrD2ybA6+GYsrLAA=', 'Content-Length': '0'} 2022-09-13T07:39:47.548 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d123fb5d5e0ae88a-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.549 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.550 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.551 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.551 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.552 DEBUG:boto:Method: GET 2022-09-13T07:39:47.553 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.553 DEBUG:boto:Data: 2022-09-13T07:39:47.554 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.555 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.555 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.556 DEBUG:boto:Params: {} 2022-09-13T07:39:47.556 DEBUG:boto:Token: None 2022-09-13T07:39:47.557 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.557 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ez5/MiG3SIhVrD2ybA6+GYsrLAA= 2022-09-13T07:39:47.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ez5/MiG3SIhVrD2ybA6+GYsrLAA=', 'Content-Length': '0'} 2022-09-13T07:39:47.561 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000254f9a6005026951-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.562 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.563 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.563 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.564 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.564 DEBUG:boto:Method: HEAD 2022-09-13T07:39:47.565 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.565 DEBUG:boto:Data: 2022-09-13T07:39:47.566 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.566 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.567 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.567 DEBUG:boto:Params: {} 2022-09-13T07:39:47.568 DEBUG:boto:Token: None 2022-09-13T07:39:47.568 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.569 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pQ0nAsWbK5kadOxucGSMRwuChGU= 2022-09-13T07:39:47.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pQ0nAsWbK5kadOxucGSMRwuChGU=', 'Content-Length': '0'} 2022-09-13T07:39:47.573 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bd10c8f43dbd0a2-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.574 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.575 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.575 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.576 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.576 DEBUG:boto:Method: HEAD 2022-09-13T07:39:47.577 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.577 DEBUG:boto:Data: 2022-09-13T07:39:47.578 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.578 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.579 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.580 DEBUG:boto:Params: {} 2022-09-13T07:39:47.580 DEBUG:boto:Token: None 2022-09-13T07:39:47.581 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.582 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pQ0nAsWbK5kadOxucGSMRwuChGU= 2022-09-13T07:39:47.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pQ0nAsWbK5kadOxucGSMRwuChGU=', 'Content-Length': '0'} 2022-09-13T07:39:47.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000692905b8e3cb0696-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:47.594 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.595 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.596 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.597 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.597 DEBUG:boto:Method: GET 2022-09-13T07:39:47.598 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.599 DEBUG:boto:Data: 2022-09-13T07:39:47.599 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.600 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.601 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.602 DEBUG:boto:Params: {} 2022-09-13T07:39:47.603 DEBUG:boto:Token: None 2022-09-13T07:39:47.603 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.604 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ez5/MiG3SIhVrD2ybA6+GYsrLAA= 2022-09-13T07:39:47.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ez5/MiG3SIhVrD2ybA6+GYsrLAA=', 'Content-Length': '0'} 2022-09-13T07:39:47.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c40fb6aeb71d327-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.610 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.612 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.613 DEBUG:boto:path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.614 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.615 DEBUG:boto:Method: GET 2022-09-13T07:39:47.616 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.617 DEBUG:boto:Data: 2022-09-13T07:39:47.618 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.619 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.620 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.621 DEBUG:boto:Params: {} 2022-09-13T07:39:47.622 DEBUG:boto:Token: None 2022-09-13T07:39:47.623 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=nEgHn6dtpr-DjlqQutDDilFO8cCU9tV 2022-09-13T07:39:47.624 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ez5/MiG3SIhVrD2ybA6+GYsrLAA= 2022-09-13T07:39:47.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ez5/MiG3SIhVrD2ybA6+GYsrLAA=', 'Content-Length': '0'} 2022-09-13T07:39:47.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'nEgHn6dtpr-DjlqQutDDilFO8cCU9tV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ec05903312694a5-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.628 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:47.629 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:47.630 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.630 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.631 DEBUG:boto:Method: GET 2022-09-13T07:39:47.631 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.632 DEBUG:boto:Data: 2022-09-13T07:39:47.633 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.633 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.634 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.635 DEBUG:boto:Params: {} 2022-09-13T07:39:47.636 DEBUG:boto:Token: None 2022-09-13T07:39:47.637 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/ 2022-09-13T07:39:47.637 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw= 2022-09-13T07:39:47.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw=', 'Content-Length': '0'} 2022-09-13T07:39:47.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095cd56e2e0d3abd1-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.642 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:47.643 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:47.644 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:47.644 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.645 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.646 DEBUG:boto:Method: GET 2022-09-13T07:39:47.646 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.647 DEBUG:boto:Data: 2022-09-13T07:39:47.647 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.648 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.648 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.649 DEBUG:boto:Params: {} 2022-09-13T07:39:47.649 DEBUG:boto:Token: None 2022-09-13T07:39:47.650 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/ 2022-09-13T07:39:47.651 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw= 2022-09-13T07:39:47.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw=', 'Content-Length': '0'} 2022-09-13T07:39:47.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a955ad75ca89ff58-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.655 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:47.656 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.657 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.657 DEBUG:boto:Method: HEAD 2022-09-13T07:39:47.658 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:47.659 DEBUG:boto:Data: 2022-09-13T07:39:47.659 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.660 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.660 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.661 DEBUG:boto:Params: {} 2022-09-13T07:39:47.661 DEBUG:boto:Token: None 2022-09-13T07:39:47.662 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj 2022-09-13T07:39:47.663 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:JyvT3WN0PygsqgIK1ljkFIUkEXc= 2022-09-13T07:39:47.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JyvT3WN0PygsqgIK1ljkFIUkEXc=', 'Content-Length': '0'} 2022-09-13T07:39:47.668 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b58ca6d5f40a8345-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.670 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.670 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.671 DEBUG:boto:Method: HEAD 2022-09-13T07:39:47.672 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:47.672 DEBUG:boto:Data: 2022-09-13T07:39:47.673 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.673 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.674 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.674 DEBUG:boto:Params: {} 2022-09-13T07:39:47.675 DEBUG:boto:Token: None 2022-09-13T07:39:47.676 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj 2022-09-13T07:39:47.676 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:JyvT3WN0PygsqgIK1ljkFIUkEXc= 2022-09-13T07:39:47.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JyvT3WN0PygsqgIK1ljkFIUkEXc=', 'Content-Length': '0'} 2022-09-13T07:39:47.681 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001daeb0d378fbb7bb-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:47.684 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.685 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.686 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.686 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.687 DEBUG:boto:Method: GET 2022-09-13T07:39:47.688 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.688 DEBUG:boto:Data: 2022-09-13T07:39:47.689 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.690 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.691 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.692 DEBUG:boto:Params: {} 2022-09-13T07:39:47.693 DEBUG:boto:Token: None 2022-09-13T07:39:47.694 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.695 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wHmQN5yKSWjJXSMtZzQ92/SBQm0= 2022-09-13T07:39:47.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wHmQN5yKSWjJXSMtZzQ92/SBQm0=', 'Content-Length': '0'} 2022-09-13T07:39:47.698 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec7efaf5d72431d4-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.746 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.747 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.748 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.748 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.749 DEBUG:boto:Method: GET 2022-09-13T07:39:47.750 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.750 DEBUG:boto:Data: 2022-09-13T07:39:47.751 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.752 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.752 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.753 DEBUG:boto:Params: {} 2022-09-13T07:39:47.754 DEBUG:boto:Token: None 2022-09-13T07:39:47.755 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:47.756 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wHmQN5yKSWjJXSMtZzQ92/SBQm0= 2022-09-13T07:39:47.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wHmQN5yKSWjJXSMtZzQ92/SBQm0=', 'Content-Length': '0'} 2022-09-13T07:39:47.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095a1490548ff511d-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.760 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:47.761 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:47.762 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.763 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.764 DEBUG:boto:Method: GET 2022-09-13T07:39:47.765 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.767 DEBUG:boto:Data: 2022-09-13T07:39:47.767 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.768 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.768 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.769 DEBUG:boto:Params: {} 2022-09-13T07:39:47.769 DEBUG:boto:Token: None 2022-09-13T07:39:47.770 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/ 2022-09-13T07:39:47.771 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw= 2022-09-13T07:39:47.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw=', 'Content-Length': '0'} 2022-09-13T07:39:47.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b2910cf9d7a0c1b-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.775 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:47.776 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:47.777 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:47.778 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.778 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.779 DEBUG:boto:Method: GET 2022-09-13T07:39:47.780 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.780 DEBUG:boto:Data: 2022-09-13T07:39:47.781 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.781 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.782 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.783 DEBUG:boto:Params: {} 2022-09-13T07:39:47.783 DEBUG:boto:Token: None 2022-09-13T07:39:47.784 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/ 2022-09-13T07:39:47.785 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw= 2022-09-13T07:39:47.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw=', 'Content-Length': '0'} 2022-09-13T07:39:47.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000404cae475359fe12-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.790 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:47.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:47.791 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.792 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.793 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.794 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.794 DEBUG:boto:Method: GET 2022-09-13T07:39:47.795 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.796 DEBUG:boto:Data: 2022-09-13T07:39:47.796 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.797 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.797 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.798 DEBUG:boto:Params: {} 2022-09-13T07:39:47.799 DEBUG:boto:Token: None 2022-09-13T07:39:47.799 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.800 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+/Nxi4raL4wUEt9di+ZOoZyznXg= 2022-09-13T07:39:47.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+/Nxi4raL4wUEt9di+ZOoZyznXg=', 'Content-Length': '0'} 2022-09-13T07:39:47.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9d78673a21d5769-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.806 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.807 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.808 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.808 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.809 DEBUG:boto:Method: GET 2022-09-13T07:39:47.809 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.810 DEBUG:boto:Data: 2022-09-13T07:39:47.810 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.811 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.812 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.812 DEBUG:boto:Params: {} 2022-09-13T07:39:47.813 DEBUG:boto:Token: None 2022-09-13T07:39:47.814 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.815 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+/Nxi4raL4wUEt9di+ZOoZyznXg= 2022-09-13T07:39:47.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+/Nxi4raL4wUEt9di+ZOoZyznXg=', 'Content-Length': '0'} 2022-09-13T07:39:47.818 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067e1ca9d11f49b2e-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.820 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.821 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.821 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.822 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.823 DEBUG:boto:Method: HEAD 2022-09-13T07:39:47.824 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.825 DEBUG:boto:Data: 2022-09-13T07:39:47.826 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.827 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.827 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.829 DEBUG:boto:Params: {} 2022-09-13T07:39:47.830 DEBUG:boto:Token: None 2022-09-13T07:39:47.831 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.832 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tTnaCUm/cPAiY9Q2fdYptSB4nMc= 2022-09-13T07:39:47.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tTnaCUm/cPAiY9Q2fdYptSB4nMc=', 'Content-Length': '0'} 2022-09-13T07:39:47.836 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085c9cec0f124db90-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.837 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.838 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.838 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.839 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.840 DEBUG:boto:Method: HEAD 2022-09-13T07:39:47.840 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.840 DEBUG:boto:Data: 2022-09-13T07:39:47.841 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.841 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.842 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.842 DEBUG:boto:Params: {} 2022-09-13T07:39:47.843 DEBUG:boto:Token: None 2022-09-13T07:39:47.844 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.845 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tTnaCUm/cPAiY9Q2fdYptSB4nMc= 2022-09-13T07:39:47.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tTnaCUm/cPAiY9Q2fdYptSB4nMc=', 'Content-Length': '0'} 2022-09-13T07:39:47.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f2e339127bebe31-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:47.851 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.852 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.853 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.854 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.855 DEBUG:boto:Method: GET 2022-09-13T07:39:47.856 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.857 DEBUG:boto:Data: 2022-09-13T07:39:47.858 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.859 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.860 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.861 DEBUG:boto:Params: {} 2022-09-13T07:39:47.862 DEBUG:boto:Token: None 2022-09-13T07:39:47.863 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.864 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+/Nxi4raL4wUEt9di+ZOoZyznXg= 2022-09-13T07:39:47.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+/Nxi4raL4wUEt9di+ZOoZyznXg=', 'Content-Length': '0'} 2022-09-13T07:39:47.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094e6589869ee09db-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.915 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:47.916 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:47.917 DEBUG:boto:path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.917 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.919 DEBUG:boto:Method: GET 2022-09-13T07:39:47.920 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.920 DEBUG:boto:Data: 2022-09-13T07:39:47.921 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.922 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.923 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.924 DEBUG:boto:Params: {} 2022-09-13T07:39:47.925 DEBUG:boto:Token: None 2022-09-13T07:39:47.925 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/obj?versionId=zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h 2022-09-13T07:39:47.926 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+/Nxi4raL4wUEt9di+ZOoZyznXg= 2022-09-13T07:39:47.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+/Nxi4raL4wUEt9di+ZOoZyznXg=', 'Content-Length': '0'} 2022-09-13T07:39:47.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'zbAEfbqMIp9ReKOn2t8gGBeNLMCc26h'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff304dd57bce981b-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.931 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:47.932 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:47.933 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.933 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.934 DEBUG:boto:Method: GET 2022-09-13T07:39:47.935 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.935 DEBUG:boto:Data: 2022-09-13T07:39:47.936 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.936 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:47.937 DEBUG:boto:Port: 8001 2022-09-13T07:39:47.937 DEBUG:boto:Params: {} 2022-09-13T07:39:47.938 DEBUG:boto:Token: None 2022-09-13T07:39:47.938 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/ 2022-09-13T07:39:47.939 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw= 2022-09-13T07:39:47.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw=', 'Content-Length': '0'} 2022-09-13T07:39:47.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa015f79d843c2d6-00632033c3-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:47.945 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:47.946 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:47.947 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:47.947 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.948 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.949 DEBUG:boto:Method: GET 2022-09-13T07:39:47.949 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:47.950 DEBUG:boto:Data: 2022-09-13T07:39:47.951 DEBUG:boto:Headers: {} 2022-09-13T07:39:47.951 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:47.952 DEBUG:boto:Port: 8000 2022-09-13T07:39:47.952 DEBUG:boto:Params: {} 2022-09-13T07:39:47.953 DEBUG:boto:Token: None 2022-09-13T07:39:47.954 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:47 GMT /yxvvuh-19/ 2022-09-13T07:39:47.954 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw= 2022-09-13T07:39:47.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+vKnTowIUVrXw4WKKabJ+SLp+Jw=', 'Content-Length': '0'} 2022-09-13T07:39:47.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000168e65dce349e1ab-00632033c3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.006 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:48.009 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.010 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.012 DEBUG:boto:Method: HEAD 2022-09-13T07:39:48.014 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:48.015 DEBUG:boto:Data: 2022-09-13T07:39:48.016 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.017 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.018 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.019 DEBUG:boto:Params: {} 2022-09-13T07:39:48.020 DEBUG:boto:Token: None 2022-09-13T07:39:48.022 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj 2022-09-13T07:39:48.023 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU= 2022-09-13T07:39:48.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU=', 'Content-Length': '0'} 2022-09-13T07:39:48.037 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e2d54812b5a7b9a-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.040 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.041 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.042 DEBUG:boto:Method: HEAD 2022-09-13T07:39:48.043 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:48.043 DEBUG:boto:Data: 2022-09-13T07:39:48.044 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.044 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.045 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.045 DEBUG:boto:Params: {} 2022-09-13T07:39:48.046 DEBUG:boto:Token: None 2022-09-13T07:39:48.047 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj 2022-09-13T07:39:48.047 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU= 2022-09-13T07:39:48.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU=', 'Content-Length': '0'} 2022-09-13T07:39:48.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b501842f19170970-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:48.055 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.056 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.057 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.057 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.058 DEBUG:boto:Method: GET 2022-09-13T07:39:48.058 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.059 DEBUG:boto:Data: 2022-09-13T07:39:48.059 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.060 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.060 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.061 DEBUG:boto:Params: {} 2022-09-13T07:39:48.062 DEBUG:boto:Token: None 2022-09-13T07:39:48.062 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.063 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I= 2022-09-13T07:39:48.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I=', 'Content-Length': '0'} 2022-09-13T07:39:48.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a6901e39e52f71e-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.067 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.068 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.068 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.069 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.070 DEBUG:boto:Method: GET 2022-09-13T07:39:48.070 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.071 DEBUG:boto:Data: 2022-09-13T07:39:48.072 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.072 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.075 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.078 DEBUG:boto:Params: {} 2022-09-13T07:39:48.079 DEBUG:boto:Token: None 2022-09-13T07:39:48.080 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.081 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I= 2022-09-13T07:39:48.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I=', 'Content-Length': '0'} 2022-09-13T07:39:48.084 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dec029c238ee0eb2-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.085 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:48.086 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:48.087 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.088 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.088 DEBUG:boto:Method: GET 2022-09-13T07:39:48.089 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.090 DEBUG:boto:Data: 2022-09-13T07:39:48.091 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.091 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.092 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.092 DEBUG:boto:Params: {} 2022-09-13T07:39:48.093 DEBUG:boto:Token: None 2022-09-13T07:39:48.094 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/ 2022-09-13T07:39:48.094 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4= 2022-09-13T07:39:48.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4=', 'Content-Length': '0'} 2022-09-13T07:39:48.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d74280c74135f91-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.099 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:48.100 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:48.100 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:48.101 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.101 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.102 DEBUG:boto:Method: GET 2022-09-13T07:39:48.102 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.103 DEBUG:boto:Data: 2022-09-13T07:39:48.103 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.104 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.104 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.105 DEBUG:boto:Params: {} 2022-09-13T07:39:48.105 DEBUG:boto:Token: None 2022-09-13T07:39:48.106 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/ 2022-09-13T07:39:48.107 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4= 2022-09-13T07:39:48.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4=', 'Content-Length': '0'} 2022-09-13T07:39:48.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2c6e38473ed565e-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.111 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:48.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:48.113 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.114 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.115 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.115 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.116 DEBUG:boto:Method: GET 2022-09-13T07:39:48.116 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.117 DEBUG:boto:Data: 2022-09-13T07:39:48.117 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.118 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.119 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.119 DEBUG:boto:Params: {} 2022-09-13T07:39:48.120 DEBUG:boto:Token: None 2022-09-13T07:39:48.120 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.121 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FJ8PHPeQaQ485oetjePQJXgt+DQ= 2022-09-13T07:39:48.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FJ8PHPeQaQ485oetjePQJXgt+DQ=', 'Content-Length': '0'} 2022-09-13T07:39:48.124 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb02f25b10e16c51-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.174 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.175 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.176 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.177 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.178 DEBUG:boto:Method: GET 2022-09-13T07:39:48.179 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.179 DEBUG:boto:Data: 2022-09-13T07:39:48.180 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.181 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.183 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.184 DEBUG:boto:Params: {} 2022-09-13T07:39:48.185 DEBUG:boto:Token: None 2022-09-13T07:39:48.186 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.187 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FJ8PHPeQaQ485oetjePQJXgt+DQ= 2022-09-13T07:39:48.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FJ8PHPeQaQ485oetjePQJXgt+DQ=', 'Content-Length': '0'} 2022-09-13T07:39:48.191 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6fd683cf19a111c-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.192 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.192 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.193 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.193 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.194 DEBUG:boto:Method: HEAD 2022-09-13T07:39:48.194 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.195 DEBUG:boto:Data: 2022-09-13T07:39:48.195 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.196 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.196 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.197 DEBUG:boto:Params: {} 2022-09-13T07:39:48.198 DEBUG:boto:Token: None 2022-09-13T07:39:48.198 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.199 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9Ss7Kr+Lhv13fMVvLsknm7xJ45Y= 2022-09-13T07:39:48.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9Ss7Kr+Lhv13fMVvLsknm7xJ45Y=', 'Content-Length': '0'} 2022-09-13T07:39:48.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c60aaeedb69d74c7-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.202 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.203 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.204 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.204 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.205 DEBUG:boto:Method: HEAD 2022-09-13T07:39:48.206 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.207 DEBUG:boto:Data: 2022-09-13T07:39:48.207 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.208 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.208 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.209 DEBUG:boto:Params: {} 2022-09-13T07:39:48.210 DEBUG:boto:Token: None 2022-09-13T07:39:48.211 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.212 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9Ss7Kr+Lhv13fMVvLsknm7xJ45Y= 2022-09-13T07:39:48.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9Ss7Kr+Lhv13fMVvLsknm7xJ45Y=', 'Content-Length': '0'} 2022-09-13T07:39:48.215 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b72a639aa33ba33a-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:48.218 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.219 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.220 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.220 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.221 DEBUG:boto:Method: GET 2022-09-13T07:39:48.222 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.223 DEBUG:boto:Data: 2022-09-13T07:39:48.224 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.225 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.226 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.227 DEBUG:boto:Params: {} 2022-09-13T07:39:48.228 DEBUG:boto:Token: None 2022-09-13T07:39:48.229 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.230 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FJ8PHPeQaQ485oetjePQJXgt+DQ= 2022-09-13T07:39:48.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FJ8PHPeQaQ485oetjePQJXgt+DQ=', 'Content-Length': '0'} 2022-09-13T07:39:48.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf4714f1aa46b558-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.234 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.235 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.236 DEBUG:boto:path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.237 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.238 DEBUG:boto:Method: GET 2022-09-13T07:39:48.239 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.240 DEBUG:boto:Data: 2022-09-13T07:39:48.241 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.242 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.243 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.244 DEBUG:boto:Params: {} 2022-09-13T07:39:48.245 DEBUG:boto:Token: None 2022-09-13T07:39:48.246 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad 2022-09-13T07:39:48.247 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FJ8PHPeQaQ485oetjePQJXgt+DQ= 2022-09-13T07:39:48.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FJ8PHPeQaQ485oetjePQJXgt+DQ=', 'Content-Length': '0'} 2022-09-13T07:39:48.251 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'kKoYsAdAKf7lBJBYkG7TA7YHa0Qh7ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4356cc9ae1ff8e7-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.253 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:48.254 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:48.255 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.256 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.257 DEBUG:boto:Method: GET 2022-09-13T07:39:48.258 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.259 DEBUG:boto:Data: 2022-09-13T07:39:48.260 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.261 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.262 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.263 DEBUG:boto:Params: {} 2022-09-13T07:39:48.264 DEBUG:boto:Token: None 2022-09-13T07:39:48.265 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/ 2022-09-13T07:39:48.266 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4= 2022-09-13T07:39:48.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4=', 'Content-Length': '0'} 2022-09-13T07:39:48.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058e2952923d4bc44-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.273 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:48.274 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:48.275 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:48.276 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.277 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.278 DEBUG:boto:Method: GET 2022-09-13T07:39:48.279 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.280 DEBUG:boto:Data: 2022-09-13T07:39:48.281 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.282 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.282 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.283 DEBUG:boto:Params: {} 2022-09-13T07:39:48.284 DEBUG:boto:Token: None 2022-09-13T07:39:48.284 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/ 2022-09-13T07:39:48.285 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4= 2022-09-13T07:39:48.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4=', 'Content-Length': '0'} 2022-09-13T07:39:48.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e575ac338b25dee3-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.334 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:48.335 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.336 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.336 DEBUG:boto:Method: HEAD 2022-09-13T07:39:48.337 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:48.338 DEBUG:boto:Data: 2022-09-13T07:39:48.338 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.339 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.340 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.340 DEBUG:boto:Params: {} 2022-09-13T07:39:48.341 DEBUG:boto:Token: None 2022-09-13T07:39:48.342 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj 2022-09-13T07:39:48.343 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU= 2022-09-13T07:39:48.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU=', 'Content-Length': '0'} 2022-09-13T07:39:48.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000938d896d143ee366-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.348 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.348 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.349 DEBUG:boto:Method: HEAD 2022-09-13T07:39:48.349 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:48.350 DEBUG:boto:Data: 2022-09-13T07:39:48.351 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.351 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.352 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.353 DEBUG:boto:Params: {} 2022-09-13T07:39:48.354 DEBUG:boto:Token: None 2022-09-13T07:39:48.355 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj 2022-09-13T07:39:48.356 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU= 2022-09-13T07:39:48.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU=', 'Content-Length': '0'} 2022-09-13T07:39:48.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000faa379b39a8c1d42-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:48.362 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.363 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.363 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.364 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.364 DEBUG:boto:Method: GET 2022-09-13T07:39:48.365 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.365 DEBUG:boto:Data: 2022-09-13T07:39:48.366 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.366 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.367 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.368 DEBUG:boto:Params: {} 2022-09-13T07:39:48.368 DEBUG:boto:Token: None 2022-09-13T07:39:48.369 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.370 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I= 2022-09-13T07:39:48.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I=', 'Content-Length': '0'} 2022-09-13T07:39:48.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d14295f990129c3-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.373 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.374 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.375 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.375 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.376 DEBUG:boto:Method: GET 2022-09-13T07:39:48.376 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.377 DEBUG:boto:Data: 2022-09-13T07:39:48.377 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.378 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.378 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.379 DEBUG:boto:Params: {} 2022-09-13T07:39:48.380 DEBUG:boto:Token: None 2022-09-13T07:39:48.380 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.381 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I= 2022-09-13T07:39:48.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I=', 'Content-Length': '0'} 2022-09-13T07:39:48.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000056777f5824c9b4-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.384 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:48.385 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:48.386 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.387 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.387 DEBUG:boto:Method: GET 2022-09-13T07:39:48.388 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.388 DEBUG:boto:Data: 2022-09-13T07:39:48.389 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.389 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.390 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.390 DEBUG:boto:Params: {} 2022-09-13T07:39:48.391 DEBUG:boto:Token: None 2022-09-13T07:39:48.392 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/ 2022-09-13T07:39:48.392 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4= 2022-09-13T07:39:48.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4=', 'Content-Length': '0'} 2022-09-13T07:39:48.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c81fe9802dd27e4-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.397 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:48.398 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:48.399 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:48.399 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.400 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.401 DEBUG:boto:Method: GET 2022-09-13T07:39:48.401 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.402 DEBUG:boto:Data: 2022-09-13T07:39:48.403 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.403 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.404 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.405 DEBUG:boto:Params: {} 2022-09-13T07:39:48.405 DEBUG:boto:Token: None 2022-09-13T07:39:48.406 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/ 2022-09-13T07:39:48.406 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4= 2022-09-13T07:39:48.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4=', 'Content-Length': '0'} 2022-09-13T07:39:48.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee1986bf6358b838-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.411 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:48.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:48.413 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.414 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.414 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.415 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.415 DEBUG:boto:Method: GET 2022-09-13T07:39:48.416 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.416 DEBUG:boto:Data: 2022-09-13T07:39:48.417 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.417 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.418 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.419 DEBUG:boto:Params: {} 2022-09-13T07:39:48.420 DEBUG:boto:Token: None 2022-09-13T07:39:48.420 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.421 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:795Q65tu+3XZeG3Izx0qoqC21Mg= 2022-09-13T07:39:48.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:795Q65tu+3XZeG3Izx0qoqC21Mg=', 'Content-Length': '0'} 2022-09-13T07:39:48.424 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010d927de7e2cb4bf-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.425 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.425 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.426 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.426 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.427 DEBUG:boto:Method: GET 2022-09-13T07:39:48.427 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.429 DEBUG:boto:Data: 2022-09-13T07:39:48.430 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.431 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.432 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.433 DEBUG:boto:Params: {} 2022-09-13T07:39:48.434 DEBUG:boto:Token: None 2022-09-13T07:39:48.435 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.437 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:795Q65tu+3XZeG3Izx0qoqC21Mg= 2022-09-13T07:39:48.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:795Q65tu+3XZeG3Izx0qoqC21Mg=', 'Content-Length': '0'} 2022-09-13T07:39:48.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f9edbb4ee5b9097-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.486 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.487 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.487 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.488 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.489 DEBUG:boto:Method: HEAD 2022-09-13T07:39:48.489 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.490 DEBUG:boto:Data: 2022-09-13T07:39:48.490 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.491 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.493 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.493 DEBUG:boto:Params: {} 2022-09-13T07:39:48.494 DEBUG:boto:Token: None 2022-09-13T07:39:48.495 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.495 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WveFksCSGtj06yhyeR90G0rJGqE= 2022-09-13T07:39:48.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WveFksCSGtj06yhyeR90G0rJGqE=', 'Content-Length': '0'} 2022-09-13T07:39:48.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e640236de0af0476-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.498 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.499 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.500 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.501 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.501 DEBUG:boto:Method: HEAD 2022-09-13T07:39:48.502 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.503 DEBUG:boto:Data: 2022-09-13T07:39:48.503 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.504 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.504 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.505 DEBUG:boto:Params: {} 2022-09-13T07:39:48.506 DEBUG:boto:Token: None 2022-09-13T07:39:48.507 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.507 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WveFksCSGtj06yhyeR90G0rJGqE= 2022-09-13T07:39:48.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WveFksCSGtj06yhyeR90G0rJGqE=', 'Content-Length': '0'} 2022-09-13T07:39:48.510 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000380eec3a5732f535-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:48.511 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.512 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.512 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.513 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.513 DEBUG:boto:Method: GET 2022-09-13T07:39:48.514 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.514 DEBUG:boto:Data: 2022-09-13T07:39:48.515 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.515 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.516 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.516 DEBUG:boto:Params: {} 2022-09-13T07:39:48.517 DEBUG:boto:Token: None 2022-09-13T07:39:48.517 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.518 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:795Q65tu+3XZeG3Izx0qoqC21Mg= 2022-09-13T07:39:48.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:795Q65tu+3XZeG3Izx0qoqC21Mg=', 'Content-Length': '0'} 2022-09-13T07:39:48.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b35a79b324708be6-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.522 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.522 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.523 DEBUG:boto:path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.523 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.524 DEBUG:boto:Method: GET 2022-09-13T07:39:48.524 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.525 DEBUG:boto:Data: 2022-09-13T07:39:48.525 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.526 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.526 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.527 DEBUG:boto:Params: {} 2022-09-13T07:39:48.527 DEBUG:boto:Token: None 2022-09-13T07:39:48.528 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1 2022-09-13T07:39:48.528 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:795Q65tu+3XZeG3Izx0qoqC21Mg= 2022-09-13T07:39:48.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:795Q65tu+3XZeG3Izx0qoqC21Mg=', 'Content-Length': '0'} 2022-09-13T07:39:48.531 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'sa9HOuzMyWC3pQ-OSnkrC71AYS1rQZ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008919360687f1498c-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.532 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:48.533 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:48.533 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.534 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.534 DEBUG:boto:Method: GET 2022-09-13T07:39:48.535 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.535 DEBUG:boto:Data: 2022-09-13T07:39:48.536 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.536 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.537 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.537 DEBUG:boto:Params: {} 2022-09-13T07:39:48.538 DEBUG:boto:Token: None 2022-09-13T07:39:48.538 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/ 2022-09-13T07:39:48.539 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4= 2022-09-13T07:39:48.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4=', 'Content-Length': '0'} 2022-09-13T07:39:48.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009102b7f64ef1c4d-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.544 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:48.544 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:48.545 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:48.545 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.546 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.546 DEBUG:boto:Method: GET 2022-09-13T07:39:48.547 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.547 DEBUG:boto:Data: 2022-09-13T07:39:48.548 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.548 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.549 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.549 DEBUG:boto:Params: {} 2022-09-13T07:39:48.550 DEBUG:boto:Token: None 2022-09-13T07:39:48.551 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/ 2022-09-13T07:39:48.551 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4= 2022-09-13T07:39:48.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4=', 'Content-Length': '0'} 2022-09-13T07:39:48.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7f71b19604f4fe2-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.556 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:48.556 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.557 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.557 DEBUG:boto:Method: HEAD 2022-09-13T07:39:48.558 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:48.558 DEBUG:boto:Data: 2022-09-13T07:39:48.559 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.559 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.560 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.560 DEBUG:boto:Params: {} 2022-09-13T07:39:48.561 DEBUG:boto:Token: None 2022-09-13T07:39:48.561 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj 2022-09-13T07:39:48.562 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU= 2022-09-13T07:39:48.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU=', 'Content-Length': '0'} 2022-09-13T07:39:48.566 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc4cde2b63ebdd1f-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.567 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.568 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.568 DEBUG:boto:Method: HEAD 2022-09-13T07:39:48.569 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:48.569 DEBUG:boto:Data: 2022-09-13T07:39:48.570 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.570 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.571 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.572 DEBUG:boto:Params: {} 2022-09-13T07:39:48.573 DEBUG:boto:Token: None 2022-09-13T07:39:48.573 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj 2022-09-13T07:39:48.574 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU= 2022-09-13T07:39:48.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU=', 'Content-Length': '0'} 2022-09-13T07:39:48.578 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a6191a80dbd6f45-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:48.580 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.581 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.581 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.582 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.582 DEBUG:boto:Method: GET 2022-09-13T07:39:48.583 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.583 DEBUG:boto:Data: 2022-09-13T07:39:48.584 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.584 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.585 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.585 DEBUG:boto:Params: {} 2022-09-13T07:39:48.586 DEBUG:boto:Token: None 2022-09-13T07:39:48.587 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.587 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I= 2022-09-13T07:39:48.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I=', 'Content-Length': '0'} 2022-09-13T07:39:48.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee5fbe34e605e0f7-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.638 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.639 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.639 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.640 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.641 DEBUG:boto:Method: GET 2022-09-13T07:39:48.641 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.642 DEBUG:boto:Data: 2022-09-13T07:39:48.642 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.643 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.644 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.645 DEBUG:boto:Params: {} 2022-09-13T07:39:48.648 DEBUG:boto:Token: None 2022-09-13T07:39:48.648 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.649 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I= 2022-09-13T07:39:48.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I=', 'Content-Length': '0'} 2022-09-13T07:39:48.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f13ba6bb4199928-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.654 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:48.655 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:48.655 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.656 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.657 DEBUG:boto:Method: GET 2022-09-13T07:39:48.657 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.658 DEBUG:boto:Data: 2022-09-13T07:39:48.658 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.659 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.660 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.660 DEBUG:boto:Params: {} 2022-09-13T07:39:48.661 DEBUG:boto:Token: None 2022-09-13T07:39:48.662 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/ 2022-09-13T07:39:48.663 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4= 2022-09-13T07:39:48.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4=', 'Content-Length': '0'} 2022-09-13T07:39:48.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cc167ff2820e209-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.667 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:48.668 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:48.669 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:48.669 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.670 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.671 DEBUG:boto:Method: GET 2022-09-13T07:39:48.671 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.672 DEBUG:boto:Data: 2022-09-13T07:39:48.673 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.673 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.674 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.674 DEBUG:boto:Params: {} 2022-09-13T07:39:48.675 DEBUG:boto:Token: None 2022-09-13T07:39:48.676 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/ 2022-09-13T07:39:48.676 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4= 2022-09-13T07:39:48.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4=', 'Content-Length': '0'} 2022-09-13T07:39:48.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ff21b7b400d9d9f-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.681 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:48.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:48.683 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.684 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.685 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.686 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.687 DEBUG:boto:Method: GET 2022-09-13T07:39:48.688 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.688 DEBUG:boto:Data: 2022-09-13T07:39:48.689 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.689 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.690 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.691 DEBUG:boto:Params: {} 2022-09-13T07:39:48.692 DEBUG:boto:Token: None 2022-09-13T07:39:48.693 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.694 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tZkU3C+s8M5QvJbSe4/sEDy/Dlg= 2022-09-13T07:39:48.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tZkU3C+s8M5QvJbSe4/sEDy/Dlg=', 'Content-Length': '0'} 2022-09-13T07:39:48.698 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060a35fbd893950a5-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.699 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.700 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.701 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.702 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.703 DEBUG:boto:Method: GET 2022-09-13T07:39:48.704 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.705 DEBUG:boto:Data: 2022-09-13T07:39:48.706 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.707 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.708 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.709 DEBUG:boto:Params: {} 2022-09-13T07:39:48.710 DEBUG:boto:Token: None 2022-09-13T07:39:48.714 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.715 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tZkU3C+s8M5QvJbSe4/sEDy/Dlg= 2022-09-13T07:39:48.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tZkU3C+s8M5QvJbSe4/sEDy/Dlg=', 'Content-Length': '0'} 2022-09-13T07:39:48.718 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000906f8158cf6e4e6c-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.720 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.720 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.721 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.722 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.723 DEBUG:boto:Method: HEAD 2022-09-13T07:39:48.724 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.725 DEBUG:boto:Data: 2022-09-13T07:39:48.725 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.726 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.727 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.727 DEBUG:boto:Params: {} 2022-09-13T07:39:48.728 DEBUG:boto:Token: None 2022-09-13T07:39:48.729 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.729 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GM6CayAGP3cUFBu5FeNgWA9901w= 2022-09-13T07:39:48.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GM6CayAGP3cUFBu5FeNgWA9901w=', 'Content-Length': '0'} 2022-09-13T07:39:48.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d04f82f0e707c0f-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.733 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.733 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.734 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.734 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.735 DEBUG:boto:Method: HEAD 2022-09-13T07:39:48.735 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.736 DEBUG:boto:Data: 2022-09-13T07:39:48.737 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.737 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.738 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.738 DEBUG:boto:Params: {} 2022-09-13T07:39:48.739 DEBUG:boto:Token: None 2022-09-13T07:39:48.739 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.740 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:GM6CayAGP3cUFBu5FeNgWA9901w= 2022-09-13T07:39:48.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:GM6CayAGP3cUFBu5FeNgWA9901w=', 'Content-Length': '0'} 2022-09-13T07:39:48.743 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bb970d8e85a07d0-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:48.744 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.745 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.745 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.746 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.746 DEBUG:boto:Method: GET 2022-09-13T07:39:48.747 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.747 DEBUG:boto:Data: 2022-09-13T07:39:48.748 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.748 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.749 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.749 DEBUG:boto:Params: {} 2022-09-13T07:39:48.750 DEBUG:boto:Token: None 2022-09-13T07:39:48.750 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.751 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tZkU3C+s8M5QvJbSe4/sEDy/Dlg= 2022-09-13T07:39:48.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tZkU3C+s8M5QvJbSe4/sEDy/Dlg=', 'Content-Length': '0'} 2022-09-13T07:39:48.754 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a84f3968314989e4-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.798 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.799 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.800 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.800 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.801 DEBUG:boto:Method: GET 2022-09-13T07:39:48.802 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.803 DEBUG:boto:Data: 2022-09-13T07:39:48.804 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.804 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.805 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.805 DEBUG:boto:Params: {} 2022-09-13T07:39:48.806 DEBUG:boto:Token: None 2022-09-13T07:39:48.806 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=xwhfwIrEIbdFug3DCFDiVhftQi2CUm6 2022-09-13T07:39:48.807 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tZkU3C+s8M5QvJbSe4/sEDy/Dlg= 2022-09-13T07:39:48.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tZkU3C+s8M5QvJbSe4/sEDy/Dlg=', 'Content-Length': '0'} 2022-09-13T07:39:48.810 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'xwhfwIrEIbdFug3DCFDiVhftQi2CUm6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b09524a7dce152e1-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.811 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:48.811 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:48.812 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.812 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.813 DEBUG:boto:Method: GET 2022-09-13T07:39:48.813 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.814 DEBUG:boto:Data: 2022-09-13T07:39:48.815 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.815 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.816 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.816 DEBUG:boto:Params: {} 2022-09-13T07:39:48.817 DEBUG:boto:Token: None 2022-09-13T07:39:48.817 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/ 2022-09-13T07:39:48.819 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4= 2022-09-13T07:39:48.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4=', 'Content-Length': '0'} 2022-09-13T07:39:48.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098c787dcc435c523-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.823 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:48.824 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:48.825 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:48.825 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.826 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.827 DEBUG:boto:Method: GET 2022-09-13T07:39:48.827 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.828 DEBUG:boto:Data: 2022-09-13T07:39:48.828 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.829 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.829 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.830 DEBUG:boto:Params: {} 2022-09-13T07:39:48.831 DEBUG:boto:Token: None 2022-09-13T07:39:48.831 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/ 2022-09-13T07:39:48.832 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4= 2022-09-13T07:39:48.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4=', 'Content-Length': '0'} 2022-09-13T07:39:48.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001046a47ada96cee6-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.836 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:48.837 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.837 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.838 DEBUG:boto:Method: HEAD 2022-09-13T07:39:48.839 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:48.839 DEBUG:boto:Data: 2022-09-13T07:39:48.839 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.840 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.840 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.841 DEBUG:boto:Params: {} 2022-09-13T07:39:48.841 DEBUG:boto:Token: None 2022-09-13T07:39:48.842 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj 2022-09-13T07:39:48.843 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU= 2022-09-13T07:39:48.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU=', 'Content-Length': '0'} 2022-09-13T07:39:48.847 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000053c264ccafc7ea1-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.848 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.848 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.849 DEBUG:boto:Method: HEAD 2022-09-13T07:39:48.849 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:48.850 DEBUG:boto:Data: 2022-09-13T07:39:48.850 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.851 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.851 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.852 DEBUG:boto:Params: {} 2022-09-13T07:39:48.853 DEBUG:boto:Token: None 2022-09-13T07:39:48.853 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj 2022-09-13T07:39:48.854 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU= 2022-09-13T07:39:48.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z4d5bQsTHzHB5Aqpjw8cP3ttNcU=', 'Content-Length': '0'} 2022-09-13T07:39:48.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011f33333238039cd-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:48.860 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.860 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.861 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.861 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.862 DEBUG:boto:Method: GET 2022-09-13T07:39:48.863 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.864 DEBUG:boto:Data: 2022-09-13T07:39:48.864 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.865 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.865 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.866 DEBUG:boto:Params: {} 2022-09-13T07:39:48.866 DEBUG:boto:Token: None 2022-09-13T07:39:48.867 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.867 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I= 2022-09-13T07:39:48.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I=', 'Content-Length': '0'} 2022-09-13T07:39:48.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000397a85b34fd63ea4-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.914 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.915 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.918 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.919 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.919 DEBUG:boto:Method: GET 2022-09-13T07:39:48.920 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.921 DEBUG:boto:Data: 2022-09-13T07:39:48.922 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.923 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.924 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.924 DEBUG:boto:Params: {} 2022-09-13T07:39:48.925 DEBUG:boto:Token: None 2022-09-13T07:39:48.926 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:48.927 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I= 2022-09-13T07:39:48.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:J+k51oGr4ORPCT1sPdU+MZypd3I=', 'Content-Length': '0'} 2022-09-13T07:39:48.931 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ccb00d3941ac668-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.932 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:48.933 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:48.934 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.935 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.936 DEBUG:boto:Method: GET 2022-09-13T07:39:48.936 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.937 DEBUG:boto:Data: 2022-09-13T07:39:48.937 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.938 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.938 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.939 DEBUG:boto:Params: {} 2022-09-13T07:39:48.940 DEBUG:boto:Token: None 2022-09-13T07:39:48.940 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/ 2022-09-13T07:39:48.941 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4= 2022-09-13T07:39:48.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4=', 'Content-Length': '0'} 2022-09-13T07:39:48.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b5b9ea24aab6d5f-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.945 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:48.946 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:48.947 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:48.947 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.948 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.948 DEBUG:boto:Method: GET 2022-09-13T07:39:48.949 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:48.949 DEBUG:boto:Data: 2022-09-13T07:39:48.950 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.950 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.952 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.952 DEBUG:boto:Params: {} 2022-09-13T07:39:48.953 DEBUG:boto:Token: None 2022-09-13T07:39:48.953 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/ 2022-09-13T07:39:48.954 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4= 2022-09-13T07:39:48.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FZbS15vxVt4qUnG2mVcRgRlgi/4=', 'Content-Length': '0'} 2022-09-13T07:39:48.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000765a4dc56be84d3b-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.959 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:48.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:48.961 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.962 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.963 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:48.964 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:48.965 DEBUG:boto:Method: GET 2022-09-13T07:39:48.966 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:48.967 DEBUG:boto:Data: 2022-09-13T07:39:48.968 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.969 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.970 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.971 DEBUG:boto:Params: {} 2022-09-13T07:39:48.972 DEBUG:boto:Token: None 2022-09-13T07:39:48.973 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:48.974 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:L5X6DLiPXdoL1Fv5TNWmkaYvKaY= 2022-09-13T07:39:48.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:L5X6DLiPXdoL1Fv5TNWmkaYvKaY=', 'Content-Length': '0'} 2022-09-13T07:39:48.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070004d250791f6d2-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.980 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.981 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.981 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:48.982 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:48.983 DEBUG:boto:Method: GET 2022-09-13T07:39:48.984 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:48.985 DEBUG:boto:Data: 2022-09-13T07:39:48.986 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.987 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:48.987 DEBUG:boto:Port: 8000 2022-09-13T07:39:48.988 DEBUG:boto:Params: {} 2022-09-13T07:39:48.989 DEBUG:boto:Token: None 2022-09-13T07:39:48.989 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:48.990 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:L5X6DLiPXdoL1Fv5TNWmkaYvKaY= 2022-09-13T07:39:48.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:L5X6DLiPXdoL1Fv5TNWmkaYvKaY=', 'Content-Length': '0'} 2022-09-13T07:39:48.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064b0a3cb7ee22ad9-00632033c4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:48.993 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:48.994 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:48.994 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:48.995 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:48.995 DEBUG:boto:Method: HEAD 2022-09-13T07:39:48.996 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:48.996 DEBUG:boto:Data: 2022-09-13T07:39:48.997 DEBUG:boto:Headers: {} 2022-09-13T07:39:48.997 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:48.998 DEBUG:boto:Port: 8001 2022-09-13T07:39:48.998 DEBUG:boto:Params: {} 2022-09-13T07:39:48.999 DEBUG:boto:Token: None 2022-09-13T07:39:48.999 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:48 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:49.000 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qmZuvMxOFusiz8v2/sZrh+AiO8I= 2022-09-13T07:39:49.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qmZuvMxOFusiz8v2/sZrh+AiO8I=', 'Content-Length': '0'} 2022-09-13T07:39:49.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1e085d3cf65f585-00632033c4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.004 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.005 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.006 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:49.006 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:49.007 DEBUG:boto:Method: HEAD 2022-09-13T07:39:49.008 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:49.009 DEBUG:boto:Data: 2022-09-13T07:39:49.009 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.010 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.010 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.011 DEBUG:boto:Params: {} 2022-09-13T07:39:49.011 DEBUG:boto:Token: None 2022-09-13T07:39:49.012 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:49.012 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KED8C6kc+Fyhlenjqrg2kwd/rH8= 2022-09-13T07:39:49.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KED8C6kc+Fyhlenjqrg2kwd/rH8=', 'Content-Length': '0'} 2022-09-13T07:39:49.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b21d8b1c1abc4dfb-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:49.018 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.019 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.019 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:49.020 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:49.020 DEBUG:boto:Method: GET 2022-09-13T07:39:49.021 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:49.021 DEBUG:boto:Data: 2022-09-13T07:39:49.022 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.022 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.023 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.024 DEBUG:boto:Params: {} 2022-09-13T07:39:49.024 DEBUG:boto:Token: None 2022-09-13T07:39:49.025 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:49.025 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OkXjQcFJOx3gYh5FPjwDZpB3dQw= 2022-09-13T07:39:49.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OkXjQcFJOx3gYh5FPjwDZpB3dQw=', 'Content-Length': '0'} 2022-09-13T07:39:49.029 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb452edb28fcfd4b-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.070 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.072 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.073 DEBUG:boto:path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:49.074 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:49.074 DEBUG:boto:Method: GET 2022-09-13T07:39:49.075 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:49.076 DEBUG:boto:Data: 2022-09-13T07:39:49.076 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.077 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.077 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.078 DEBUG:boto:Params: {} 2022-09-13T07:39:49.078 DEBUG:boto:Token: None 2022-09-13T07:39:49.079 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o 2022-09-13T07:39:49.080 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OkXjQcFJOx3gYh5FPjwDZpB3dQw= 2022-09-13T07:39:49.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OkXjQcFJOx3gYh5FPjwDZpB3dQw=', 'Content-Length': '0'} 2022-09-13T07:39:49.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'N1ijg0L2Sa1qSFcwgB7aYlf3fysHL.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081bab17808ca054c-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.083 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:49.084 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:49.084 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.085 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.085 DEBUG:boto:Method: GET 2022-09-13T07:39:49.086 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.086 DEBUG:boto:Data: 2022-09-13T07:39:49.087 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.087 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.088 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.088 DEBUG:boto:Params: {} 2022-09-13T07:39:49.089 DEBUG:boto:Token: None 2022-09-13T07:39:49.089 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/ 2022-09-13T07:39:49.090 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ= 2022-09-13T07:39:49.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ=', 'Content-Length': '0'} 2022-09-13T07:39:49.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2b4c8ef0b181cc3-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.094 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:49.095 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:49.096 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:49.096 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.097 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.097 DEBUG:boto:Method: GET 2022-09-13T07:39:49.098 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.098 DEBUG:boto:Data: 2022-09-13T07:39:49.099 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.099 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.100 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.100 DEBUG:boto:Params: {} 2022-09-13T07:39:49.101 DEBUG:boto:Token: None 2022-09-13T07:39:49.101 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/ 2022-09-13T07:39:49.102 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ= 2022-09-13T07:39:49.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ=', 'Content-Length': '0'} 2022-09-13T07:39:49.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f48bd3163d8bf33-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.108 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:49.110 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.111 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.112 DEBUG:boto:Method: HEAD 2022-09-13T07:39:49.113 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:49.114 DEBUG:boto:Data: 2022-09-13T07:39:49.115 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.116 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.117 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.118 DEBUG:boto:Params: {} 2022-09-13T07:39:49.119 DEBUG:boto:Token: None 2022-09-13T07:39:49.120 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj 2022-09-13T07:39:49.121 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2yvcD4R9+XP4JJg7ZB53d5dJtnE= 2022-09-13T07:39:49.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2yvcD4R9+XP4JJg7ZB53d5dJtnE=', 'Content-Length': '0'} 2022-09-13T07:39:49.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfd6a5304a96637c-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.128 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.129 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.130 DEBUG:boto:Method: HEAD 2022-09-13T07:39:49.131 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:49.132 DEBUG:boto:Data: 2022-09-13T07:39:49.133 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.134 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.135 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.136 DEBUG:boto:Params: {} 2022-09-13T07:39:49.137 DEBUG:boto:Token: None 2022-09-13T07:39:49.138 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj 2022-09-13T07:39:49.139 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2yvcD4R9+XP4JJg7ZB53d5dJtnE= 2022-09-13T07:39:49.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2yvcD4R9+XP4JJg7ZB53d5dJtnE=', 'Content-Length': '0'} 2022-09-13T07:39:49.144 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a2372fe3218dc1f-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:49.146 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.148 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.148 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.149 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.150 DEBUG:boto:Method: GET 2022-09-13T07:39:49.151 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.152 DEBUG:boto:Data: 2022-09-13T07:39:49.153 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.154 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.155 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.156 DEBUG:boto:Params: {} 2022-09-13T07:39:49.158 DEBUG:boto:Token: None 2022-09-13T07:39:49.159 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.160 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RUATXn5SkJyXrvYFkUV5l+p4qo= 2022-09-13T07:39:49.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RUATXn5SkJyXrvYFkUV5l+p4qo=', 'Content-Length': '0'} 2022-09-13T07:39:49.164 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d1741d54f6aae4c-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.165 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.166 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.167 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.168 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.169 DEBUG:boto:Method: GET 2022-09-13T07:39:49.170 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.171 DEBUG:boto:Data: 2022-09-13T07:39:49.172 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.173 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.174 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.175 DEBUG:boto:Params: {} 2022-09-13T07:39:49.176 DEBUG:boto:Token: None 2022-09-13T07:39:49.177 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.178 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RUATXn5SkJyXrvYFkUV5l+p4qo= 2022-09-13T07:39:49.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RUATXn5SkJyXrvYFkUV5l+p4qo=', 'Content-Length': '0'} 2022-09-13T07:39:49.182 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000debcdd3d87d222e2-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.226 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:49.227 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:49.228 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.229 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.230 DEBUG:boto:Method: GET 2022-09-13T07:39:49.231 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.232 DEBUG:boto:Data: 2022-09-13T07:39:49.233 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.233 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.234 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.234 DEBUG:boto:Params: {} 2022-09-13T07:39:49.235 DEBUG:boto:Token: None 2022-09-13T07:39:49.236 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/ 2022-09-13T07:39:49.236 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ= 2022-09-13T07:39:49.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ=', 'Content-Length': '0'} 2022-09-13T07:39:49.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034d3ede3b7ff2964-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.241 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:49.242 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:49.242 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:49.243 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.243 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.244 DEBUG:boto:Method: GET 2022-09-13T07:39:49.244 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.245 DEBUG:boto:Data: 2022-09-13T07:39:49.245 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.246 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.246 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.247 DEBUG:boto:Params: {} 2022-09-13T07:39:49.247 DEBUG:boto:Token: None 2022-09-13T07:39:49.248 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/ 2022-09-13T07:39:49.249 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ= 2022-09-13T07:39:49.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ=', 'Content-Length': '0'} 2022-09-13T07:39:49.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f5b8b7be8d82e40-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.253 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:49.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:49.254 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.255 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.255 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.256 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.256 DEBUG:boto:Method: GET 2022-09-13T07:39:49.257 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.257 DEBUG:boto:Data: 2022-09-13T07:39:49.258 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.258 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.259 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.259 DEBUG:boto:Params: {} 2022-09-13T07:39:49.260 DEBUG:boto:Token: None 2022-09-13T07:39:49.261 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.261 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pq4ZYF0Pdih2AIhwTjYpsuKwuJs= 2022-09-13T07:39:49.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pq4ZYF0Pdih2AIhwTjYpsuKwuJs=', 'Content-Length': '0'} 2022-09-13T07:39:49.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b89f94528709e1e9-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.310 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.311 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.312 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.313 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.315 DEBUG:boto:Method: GET 2022-09-13T07:39:49.315 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.316 DEBUG:boto:Data: 2022-09-13T07:39:49.316 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.317 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.318 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.319 DEBUG:boto:Params: {} 2022-09-13T07:39:49.320 DEBUG:boto:Token: None 2022-09-13T07:39:49.321 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.322 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pq4ZYF0Pdih2AIhwTjYpsuKwuJs= 2022-09-13T07:39:49.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pq4ZYF0Pdih2AIhwTjYpsuKwuJs=', 'Content-Length': '0'} 2022-09-13T07:39:49.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000833e51f7d3f7aa34-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.328 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.329 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.329 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.330 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.331 DEBUG:boto:Method: HEAD 2022-09-13T07:39:49.332 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.333 DEBUG:boto:Data: 2022-09-13T07:39:49.334 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.335 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.336 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.337 DEBUG:boto:Params: {} 2022-09-13T07:39:49.338 DEBUG:boto:Token: None 2022-09-13T07:39:49.339 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.340 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+QVb7MIjY7C4OQDC4fsK53l7+d4= 2022-09-13T07:39:49.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+QVb7MIjY7C4OQDC4fsK53l7+d4=', 'Content-Length': '0'} 2022-09-13T07:39:49.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015842455d6cada94-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.345 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.346 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.347 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.348 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.348 DEBUG:boto:Method: HEAD 2022-09-13T07:39:49.349 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.349 DEBUG:boto:Data: 2022-09-13T07:39:49.350 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.350 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.351 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.351 DEBUG:boto:Params: {} 2022-09-13T07:39:49.352 DEBUG:boto:Token: None 2022-09-13T07:39:49.352 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.353 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+QVb7MIjY7C4OQDC4fsK53l7+d4= 2022-09-13T07:39:49.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+QVb7MIjY7C4OQDC4fsK53l7+d4=', 'Content-Length': '0'} 2022-09-13T07:39:49.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5d007aba2125577-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:49.357 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.357 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.358 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.359 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.359 DEBUG:boto:Method: GET 2022-09-13T07:39:49.360 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.360 DEBUG:boto:Data: 2022-09-13T07:39:49.361 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.361 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.362 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.363 DEBUG:boto:Params: {} 2022-09-13T07:39:49.363 DEBUG:boto:Token: None 2022-09-13T07:39:49.365 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.366 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pq4ZYF0Pdih2AIhwTjYpsuKwuJs= 2022-09-13T07:39:49.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pq4ZYF0Pdih2AIhwTjYpsuKwuJs=', 'Content-Length': '0'} 2022-09-13T07:39:49.369 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084d73d520a9b0381-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.370 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.371 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.371 DEBUG:boto:path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.372 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.372 DEBUG:boto:Method: GET 2022-09-13T07:39:49.373 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.373 DEBUG:boto:Data: 2022-09-13T07:39:49.374 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.374 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.375 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.376 DEBUG:boto:Params: {} 2022-09-13T07:39:49.377 DEBUG:boto:Token: None 2022-09-13T07:39:49.377 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p 2022-09-13T07:39:49.378 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pq4ZYF0Pdih2AIhwTjYpsuKwuJs= 2022-09-13T07:39:49.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pq4ZYF0Pdih2AIhwTjYpsuKwuJs=', 'Content-Length': '0'} 2022-09-13T07:39:49.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'j0V8x9PrXoTJ-Iwmsxni9-Gv0bOLV6p'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd3e6580c1fd7ba8-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.382 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:49.383 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:49.383 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.383 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.384 DEBUG:boto:Method: GET 2022-09-13T07:39:49.385 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.385 DEBUG:boto:Data: 2022-09-13T07:39:49.386 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.386 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.387 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.388 DEBUG:boto:Params: {} 2022-09-13T07:39:49.388 DEBUG:boto:Token: None 2022-09-13T07:39:49.389 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/ 2022-09-13T07:39:49.389 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ= 2022-09-13T07:39:49.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ=', 'Content-Length': '0'} 2022-09-13T07:39:49.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005365709403efa2eb-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.395 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:49.396 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:49.397 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:49.398 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.398 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.399 DEBUG:boto:Method: GET 2022-09-13T07:39:49.400 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.401 DEBUG:boto:Data: 2022-09-13T07:39:49.401 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.402 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.402 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.403 DEBUG:boto:Params: {} 2022-09-13T07:39:49.404 DEBUG:boto:Token: None 2022-09-13T07:39:49.404 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/ 2022-09-13T07:39:49.405 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ= 2022-09-13T07:39:49.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ=', 'Content-Length': '0'} 2022-09-13T07:39:49.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebde8b5c0fa820ab-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.410 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:49.411 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.412 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.412 DEBUG:boto:Method: HEAD 2022-09-13T07:39:49.413 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:49.413 DEBUG:boto:Data: 2022-09-13T07:39:49.414 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.415 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.415 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.416 DEBUG:boto:Params: {} 2022-09-13T07:39:49.416 DEBUG:boto:Token: None 2022-09-13T07:39:49.417 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj 2022-09-13T07:39:49.418 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2yvcD4R9+XP4JJg7ZB53d5dJtnE= 2022-09-13T07:39:49.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2yvcD4R9+XP4JJg7ZB53d5dJtnE=', 'Content-Length': '0'} 2022-09-13T07:39:49.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000110323f0864cc81e-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.423 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.424 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.425 DEBUG:boto:Method: HEAD 2022-09-13T07:39:49.425 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:49.426 DEBUG:boto:Data: 2022-09-13T07:39:49.427 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.427 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.428 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.428 DEBUG:boto:Params: {} 2022-09-13T07:39:49.429 DEBUG:boto:Token: None 2022-09-13T07:39:49.429 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj 2022-09-13T07:39:49.430 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2yvcD4R9+XP4JJg7ZB53d5dJtnE= 2022-09-13T07:39:49.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2yvcD4R9+XP4JJg7ZB53d5dJtnE=', 'Content-Length': '0'} 2022-09-13T07:39:49.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021160c5ad8b286e4-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:49.436 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.436 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.437 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.437 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.438 DEBUG:boto:Method: GET 2022-09-13T07:39:49.438 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.439 DEBUG:boto:Data: 2022-09-13T07:39:49.439 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.440 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.440 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.441 DEBUG:boto:Params: {} 2022-09-13T07:39:49.442 DEBUG:boto:Token: None 2022-09-13T07:39:49.442 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.443 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RUATXn5SkJyXrvYFkUV5l+p4qo= 2022-09-13T07:39:49.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RUATXn5SkJyXrvYFkUV5l+p4qo=', 'Content-Length': '0'} 2022-09-13T07:39:49.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc77cbcb47ede06d-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.486 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.487 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.488 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.491 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.492 DEBUG:boto:Method: GET 2022-09-13T07:39:49.492 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.493 DEBUG:boto:Data: 2022-09-13T07:39:49.493 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.494 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.494 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.495 DEBUG:boto:Params: {} 2022-09-13T07:39:49.496 DEBUG:boto:Token: None 2022-09-13T07:39:49.497 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.497 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RUATXn5SkJyXrvYFkUV5l+p4qo= 2022-09-13T07:39:49.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RUATXn5SkJyXrvYFkUV5l+p4qo=', 'Content-Length': '0'} 2022-09-13T07:39:49.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081c6af28af12abdd-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.501 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:49.502 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:49.502 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.503 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.504 DEBUG:boto:Method: GET 2022-09-13T07:39:49.504 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.505 DEBUG:boto:Data: 2022-09-13T07:39:49.506 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.506 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.507 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.507 DEBUG:boto:Params: {} 2022-09-13T07:39:49.508 DEBUG:boto:Token: None 2022-09-13T07:39:49.509 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/ 2022-09-13T07:39:49.509 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ= 2022-09-13T07:39:49.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ=', 'Content-Length': '0'} 2022-09-13T07:39:49.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b761afb7ff435ab-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.514 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:49.515 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:49.516 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:49.517 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.517 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.519 DEBUG:boto:Method: GET 2022-09-13T07:39:49.520 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.521 DEBUG:boto:Data: 2022-09-13T07:39:49.521 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.522 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.523 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.523 DEBUG:boto:Params: {} 2022-09-13T07:39:49.524 DEBUG:boto:Token: None 2022-09-13T07:39:49.524 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/ 2022-09-13T07:39:49.525 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ= 2022-09-13T07:39:49.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ=', 'Content-Length': '0'} 2022-09-13T07:39:49.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be32fef9933a728d-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.530 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:49.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:49.532 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.533 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.533 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.534 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.534 DEBUG:boto:Method: GET 2022-09-13T07:39:49.535 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.535 DEBUG:boto:Data: 2022-09-13T07:39:49.536 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.536 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.537 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.537 DEBUG:boto:Params: {} 2022-09-13T07:39:49.538 DEBUG:boto:Token: None 2022-09-13T07:39:49.539 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.540 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Vs0Hi84I7oiEBa3y7pTqQtu8yoo= 2022-09-13T07:39:49.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Vs0Hi84I7oiEBa3y7pTqQtu8yoo=', 'Content-Length': '0'} 2022-09-13T07:39:49.543 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0f94b9e39c57efb-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.544 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.544 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.545 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.545 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.546 DEBUG:boto:Method: GET 2022-09-13T07:39:49.546 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.547 DEBUG:boto:Data: 2022-09-13T07:39:49.547 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.548 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.548 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.549 DEBUG:boto:Params: {} 2022-09-13T07:39:49.550 DEBUG:boto:Token: None 2022-09-13T07:39:49.551 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.552 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Vs0Hi84I7oiEBa3y7pTqQtu8yoo= 2022-09-13T07:39:49.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Vs0Hi84I7oiEBa3y7pTqQtu8yoo=', 'Content-Length': '0'} 2022-09-13T07:39:49.556 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002ff57e387c916fc-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.558 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.559 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.560 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.561 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.563 DEBUG:boto:Method: HEAD 2022-09-13T07:39:49.564 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.564 DEBUG:boto:Data: 2022-09-13T07:39:49.565 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.565 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.566 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.566 DEBUG:boto:Params: {} 2022-09-13T07:39:49.567 DEBUG:boto:Token: None 2022-09-13T07:39:49.567 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.568 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:g6PNt1sNAmetZpltM+Xks7jq/pY= 2022-09-13T07:39:49.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:g6PNt1sNAmetZpltM+Xks7jq/pY=', 'Content-Length': '0'} 2022-09-13T07:39:49.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c64c23eb4866fed1-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.571 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.572 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.573 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.573 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.574 DEBUG:boto:Method: HEAD 2022-09-13T07:39:49.574 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.575 DEBUG:boto:Data: 2022-09-13T07:39:49.575 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.576 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.577 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.577 DEBUG:boto:Params: {} 2022-09-13T07:39:49.578 DEBUG:boto:Token: None 2022-09-13T07:39:49.578 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.579 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:g6PNt1sNAmetZpltM+Xks7jq/pY= 2022-09-13T07:39:49.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:g6PNt1sNAmetZpltM+Xks7jq/pY=', 'Content-Length': '0'} 2022-09-13T07:39:49.581 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b1cc1326d2f0a55-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:49.583 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.583 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.584 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.584 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.585 DEBUG:boto:Method: GET 2022-09-13T07:39:49.585 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.586 DEBUG:boto:Data: 2022-09-13T07:39:49.586 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.587 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.588 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.588 DEBUG:boto:Params: {} 2022-09-13T07:39:49.589 DEBUG:boto:Token: None 2022-09-13T07:39:49.590 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.590 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Vs0Hi84I7oiEBa3y7pTqQtu8yoo= 2022-09-13T07:39:49.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Vs0Hi84I7oiEBa3y7pTqQtu8yoo=', 'Content-Length': '0'} 2022-09-13T07:39:49.593 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfa8fe0aa0dc8616-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.636 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.637 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.637 DEBUG:boto:path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.638 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.638 DEBUG:boto:Method: GET 2022-09-13T07:39:49.639 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.639 DEBUG:boto:Data: 2022-09-13T07:39:49.640 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.641 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.641 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.642 DEBUG:boto:Params: {} 2022-09-13T07:39:49.642 DEBUG:boto:Token: None 2022-09-13T07:39:49.643 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM 2022-09-13T07:39:49.644 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Vs0Hi84I7oiEBa3y7pTqQtu8yoo= 2022-09-13T07:39:49.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Vs0Hi84I7oiEBa3y7pTqQtu8yoo=', 'Content-Length': '0'} 2022-09-13T07:39:49.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'r3j0LS1bjhh8xQBCJCWMQPOkDqpqjuM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6a0c9a491ad4629-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.649 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:49.649 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:49.650 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.651 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.652 DEBUG:boto:Method: GET 2022-09-13T07:39:49.653 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.653 DEBUG:boto:Data: 2022-09-13T07:39:49.654 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.655 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.655 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.656 DEBUG:boto:Params: {} 2022-09-13T07:39:49.657 DEBUG:boto:Token: None 2022-09-13T07:39:49.657 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/ 2022-09-13T07:39:49.658 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ= 2022-09-13T07:39:49.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ=', 'Content-Length': '0'} 2022-09-13T07:39:49.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083e17324fd198a83-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.663 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:49.664 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:49.665 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:49.665 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.666 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.666 DEBUG:boto:Method: GET 2022-09-13T07:39:49.667 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.667 DEBUG:boto:Data: 2022-09-13T07:39:49.668 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.668 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.669 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.670 DEBUG:boto:Params: {} 2022-09-13T07:39:49.671 DEBUG:boto:Token: None 2022-09-13T07:39:49.672 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/ 2022-09-13T07:39:49.672 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ= 2022-09-13T07:39:49.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ=', 'Content-Length': '0'} 2022-09-13T07:39:49.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8052dc937f0d73a-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.677 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:49.677 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.678 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.678 DEBUG:boto:Method: HEAD 2022-09-13T07:39:49.679 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:49.680 DEBUG:boto:Data: 2022-09-13T07:39:49.680 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.681 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.681 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.682 DEBUG:boto:Params: {} 2022-09-13T07:39:49.683 DEBUG:boto:Token: None 2022-09-13T07:39:49.683 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj 2022-09-13T07:39:49.684 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2yvcD4R9+XP4JJg7ZB53d5dJtnE= 2022-09-13T07:39:49.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2yvcD4R9+XP4JJg7ZB53d5dJtnE=', 'Content-Length': '0'} 2022-09-13T07:39:49.688 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6a2ab3008a1d466-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.689 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.690 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.690 DEBUG:boto:Method: HEAD 2022-09-13T07:39:49.691 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:49.691 DEBUG:boto:Data: 2022-09-13T07:39:49.692 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.693 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.693 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.694 DEBUG:boto:Params: {} 2022-09-13T07:39:49.695 DEBUG:boto:Token: None 2022-09-13T07:39:49.695 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj 2022-09-13T07:39:49.696 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2yvcD4R9+XP4JJg7ZB53d5dJtnE= 2022-09-13T07:39:49.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2yvcD4R9+XP4JJg7ZB53d5dJtnE=', 'Content-Length': '0'} 2022-09-13T07:39:49.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6f18c94dcc65f72-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:49.702 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.702 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.703 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.703 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.704 DEBUG:boto:Method: GET 2022-09-13T07:39:49.705 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.705 DEBUG:boto:Data: 2022-09-13T07:39:49.706 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.706 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.707 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.708 DEBUG:boto:Params: {} 2022-09-13T07:39:49.708 DEBUG:boto:Token: None 2022-09-13T07:39:49.709 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.710 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RUATXn5SkJyXrvYFkUV5l+p4qo= 2022-09-13T07:39:49.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RUATXn5SkJyXrvYFkUV5l+p4qo=', 'Content-Length': '0'} 2022-09-13T07:39:49.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e3370106d3933ac-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.758 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.759 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.760 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.761 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.762 DEBUG:boto:Method: GET 2022-09-13T07:39:49.763 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.764 DEBUG:boto:Data: 2022-09-13T07:39:49.765 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.766 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.767 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.768 DEBUG:boto:Params: {} 2022-09-13T07:39:49.769 DEBUG:boto:Token: None 2022-09-13T07:39:49.770 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:49.771 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6RUATXn5SkJyXrvYFkUV5l+p4qo= 2022-09-13T07:39:49.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6RUATXn5SkJyXrvYFkUV5l+p4qo=', 'Content-Length': '0'} 2022-09-13T07:39:49.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecdd8f05d83f2ed5-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.776 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:49.777 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:49.778 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.779 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.780 DEBUG:boto:Method: GET 2022-09-13T07:39:49.781 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.782 DEBUG:boto:Data: 2022-09-13T07:39:49.783 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.784 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.785 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.786 DEBUG:boto:Params: {} 2022-09-13T07:39:49.787 DEBUG:boto:Token: None 2022-09-13T07:39:49.788 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/ 2022-09-13T07:39:49.789 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ= 2022-09-13T07:39:49.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ=', 'Content-Length': '0'} 2022-09-13T07:39:49.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be99e68d5979de19-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.795 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:49.797 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:49.798 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:49.799 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.800 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.800 DEBUG:boto:Method: GET 2022-09-13T07:39:49.801 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.801 DEBUG:boto:Data: 2022-09-13T07:39:49.802 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.803 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.803 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.804 DEBUG:boto:Params: {} 2022-09-13T07:39:49.805 DEBUG:boto:Token: None 2022-09-13T07:39:49.805 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/ 2022-09-13T07:39:49.807 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ= 2022-09-13T07:39:49.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ=', 'Content-Length': '0'} 2022-09-13T07:39:49.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000832e58f422ece4a1-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.812 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:49.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:49.815 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.816 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.816 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.817 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.818 DEBUG:boto:Method: GET 2022-09-13T07:39:49.819 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.820 DEBUG:boto:Data: 2022-09-13T07:39:49.821 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.822 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.823 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.824 DEBUG:boto:Params: {} 2022-09-13T07:39:49.826 DEBUG:boto:Token: None 2022-09-13T07:39:49.827 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.828 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:J59yvJVJeD9Yss8I0cAjchSN53c= 2022-09-13T07:39:49.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:J59yvJVJeD9Yss8I0cAjchSN53c=', 'Content-Length': '0'} 2022-09-13T07:39:49.832 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002cd56e5aad2ae6f1-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.834 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.835 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.835 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.836 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.837 DEBUG:boto:Method: GET 2022-09-13T07:39:49.838 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.839 DEBUG:boto:Data: 2022-09-13T07:39:49.840 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.841 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.841 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.842 DEBUG:boto:Params: {} 2022-09-13T07:39:49.843 DEBUG:boto:Token: None 2022-09-13T07:39:49.844 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.844 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:J59yvJVJeD9Yss8I0cAjchSN53c= 2022-09-13T07:39:49.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:J59yvJVJeD9Yss8I0cAjchSN53c=', 'Content-Length': '0'} 2022-09-13T07:39:49.847 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029710063cdddb510-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.848 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.848 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.849 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.849 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.850 DEBUG:boto:Method: HEAD 2022-09-13T07:39:49.850 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.851 DEBUG:boto:Data: 2022-09-13T07:39:49.852 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.852 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.853 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.853 DEBUG:boto:Params: {} 2022-09-13T07:39:49.854 DEBUG:boto:Token: None 2022-09-13T07:39:49.855 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.855 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:s9kQCOVVWg2BZgSTX21yIf8Z++s= 2022-09-13T07:39:49.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:s9kQCOVVWg2BZgSTX21yIf8Z++s=', 'Content-Length': '0'} 2022-09-13T07:39:49.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036a0473e4eca4ee5-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.861 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.862 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.862 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.863 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.864 DEBUG:boto:Method: HEAD 2022-09-13T07:39:49.864 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.864 DEBUG:boto:Data: 2022-09-13T07:39:49.865 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.865 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.866 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.867 DEBUG:boto:Params: {} 2022-09-13T07:39:49.867 DEBUG:boto:Token: None 2022-09-13T07:39:49.868 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.868 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:s9kQCOVVWg2BZgSTX21yIf8Z++s= 2022-09-13T07:39:49.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:s9kQCOVVWg2BZgSTX21yIf8Z++s=', 'Content-Length': '0'} 2022-09-13T07:39:49.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d0cd416363d93ec-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:49.880 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.880 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.881 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.882 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.883 DEBUG:boto:Method: GET 2022-09-13T07:39:49.884 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.884 DEBUG:boto:Data: 2022-09-13T07:39:49.885 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.886 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.887 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.888 DEBUG:boto:Params: {} 2022-09-13T07:39:49.889 DEBUG:boto:Token: None 2022-09-13T07:39:49.889 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.890 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:J59yvJVJeD9Yss8I0cAjchSN53c= 2022-09-13T07:39:49.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:J59yvJVJeD9Yss8I0cAjchSN53c=', 'Content-Length': '0'} 2022-09-13T07:39:49.894 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cca79d7a6e0c9752-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.938 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.939 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.940 DEBUG:boto:path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.941 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.942 DEBUG:boto:Method: GET 2022-09-13T07:39:49.943 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.944 DEBUG:boto:Data: 2022-09-13T07:39:49.946 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.947 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.948 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.949 DEBUG:boto:Params: {} 2022-09-13T07:39:49.950 DEBUG:boto:Token: None 2022-09-13T07:39:49.951 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/obj?versionId=UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG 2022-09-13T07:39:49.952 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:J59yvJVJeD9Yss8I0cAjchSN53c= 2022-09-13T07:39:49.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:J59yvJVJeD9Yss8I0cAjchSN53c=', 'Content-Length': '0'} 2022-09-13T07:39:49.956 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'UFIYwFqhiCTkSFbmDIDHjWDiTsQToDG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000697c49da97b9dd1a-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.958 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:49.959 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:49.959 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.960 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.961 DEBUG:boto:Method: GET 2022-09-13T07:39:49.961 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.962 DEBUG:boto:Data: 2022-09-13T07:39:49.963 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.964 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.964 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.965 DEBUG:boto:Params: {} 2022-09-13T07:39:49.965 DEBUG:boto:Token: None 2022-09-13T07:39:49.966 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/ 2022-09-13T07:39:49.967 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ= 2022-09-13T07:39:49.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ=', 'Content-Length': '0'} 2022-09-13T07:39:49.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032e8627821909633-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.972 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:49.974 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:49.975 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:49.975 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.976 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.977 DEBUG:boto:Method: GET 2022-09-13T07:39:49.978 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:49.979 DEBUG:boto:Data: 2022-09-13T07:39:49.980 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.980 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:49.981 DEBUG:boto:Port: 8000 2022-09-13T07:39:49.982 DEBUG:boto:Params: {} 2022-09-13T07:39:49.983 DEBUG:boto:Token: None 2022-09-13T07:39:49.984 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:49 GMT /yxvvuh-19/ 2022-09-13T07:39:49.985 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ= 2022-09-13T07:39:49.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:xUEkN+00MGw4sQ8cnVWcSVgoVCQ=', 'Content-Length': '0'} 2022-09-13T07:39:49.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcbe4946ee8458e8-00632033c5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:49.990 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:49.992 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:49.993 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:49.994 DEBUG:boto:Method: HEAD 2022-09-13T07:39:49.995 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:49.995 DEBUG:boto:Data: 2022-09-13T07:39:49.996 DEBUG:boto:Headers: {} 2022-09-13T07:39:49.997 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:49.998 DEBUG:boto:Port: 8001 2022-09-13T07:39:49.998 DEBUG:boto:Params: {} 2022-09-13T07:39:49.999 DEBUG:boto:Token: None 2022-09-13T07:39:50.000 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj 2022-09-13T07:39:50.001 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg= 2022-09-13T07:39:50.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg=', 'Content-Length': '0'} 2022-09-13T07:39:50.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f28200629fe0dfe-00632033c5-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.006 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.006 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.007 DEBUG:boto:Method: HEAD 2022-09-13T07:39:50.007 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:50.008 DEBUG:boto:Data: 2022-09-13T07:39:50.008 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.009 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.009 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.010 DEBUG:boto:Params: {} 2022-09-13T07:39:50.010 DEBUG:boto:Token: None 2022-09-13T07:39:50.011 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj 2022-09-13T07:39:50.012 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg= 2022-09-13T07:39:50.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg=', 'Content-Length': '0'} 2022-09-13T07:39:50.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4b584889f4df619-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:50.017 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.018 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.018 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.019 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.019 DEBUG:boto:Method: GET 2022-09-13T07:39:50.020 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.020 DEBUG:boto:Data: 2022-09-13T07:39:50.021 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.021 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.022 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.022 DEBUG:boto:Params: {} 2022-09-13T07:39:50.023 DEBUG:boto:Token: None 2022-09-13T07:39:50.024 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.024 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q= 2022-09-13T07:39:50.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q=', 'Content-Length': '0'} 2022-09-13T07:39:50.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a8a9f75fbabc692-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.029 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.029 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.030 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.030 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.031 DEBUG:boto:Method: GET 2022-09-13T07:39:50.031 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.032 DEBUG:boto:Data: 2022-09-13T07:39:50.032 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.033 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.033 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.034 DEBUG:boto:Params: {} 2022-09-13T07:39:50.035 DEBUG:boto:Token: None 2022-09-13T07:39:50.035 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.036 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q= 2022-09-13T07:39:50.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q=', 'Content-Length': '0'} 2022-09-13T07:39:50.038 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa33af0ec5891a1d-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.039 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:50.040 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:50.041 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.041 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.042 DEBUG:boto:Method: GET 2022-09-13T07:39:50.042 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.043 DEBUG:boto:Data: 2022-09-13T07:39:50.043 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.044 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.044 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.045 DEBUG:boto:Params: {} 2022-09-13T07:39:50.045 DEBUG:boto:Token: None 2022-09-13T07:39:50.046 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/ 2022-09-13T07:39:50.047 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ= 2022-09-13T07:39:50.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ=', 'Content-Length': '0'} 2022-09-13T07:39:50.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048531025e9476dc4-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.051 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:50.052 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:50.052 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:50.053 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.053 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.054 DEBUG:boto:Method: GET 2022-09-13T07:39:50.055 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.055 DEBUG:boto:Data: 2022-09-13T07:39:50.056 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.056 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.057 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.058 DEBUG:boto:Params: {} 2022-09-13T07:39:50.059 DEBUG:boto:Token: None 2022-09-13T07:39:50.060 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/ 2022-09-13T07:39:50.061 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ= 2022-09-13T07:39:50.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ=', 'Content-Length': '0'} 2022-09-13T07:39:50.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df1365d6e8bbea74-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.066 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:50.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:50.068 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.068 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.069 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.069 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.070 DEBUG:boto:Method: GET 2022-09-13T07:39:50.070 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.071 DEBUG:boto:Data: 2022-09-13T07:39:50.072 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.072 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.073 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.073 DEBUG:boto:Params: {} 2022-09-13T07:39:50.074 DEBUG:boto:Token: None 2022-09-13T07:39:50.075 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.076 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LsUoj7IScVnP3cVSP10TqB4NsGI= 2022-09-13T07:39:50.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LsUoj7IScVnP3cVSP10TqB4NsGI=', 'Content-Length': '0'} 2022-09-13T07:39:50.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000689d00e314d81e2a-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.122 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.123 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.124 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.125 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.126 DEBUG:boto:Method: GET 2022-09-13T07:39:50.127 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.128 DEBUG:boto:Data: 2022-09-13T07:39:50.129 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.130 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.131 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.132 DEBUG:boto:Params: {} 2022-09-13T07:39:50.133 DEBUG:boto:Token: None 2022-09-13T07:39:50.134 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.135 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LsUoj7IScVnP3cVSP10TqB4NsGI= 2022-09-13T07:39:50.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LsUoj7IScVnP3cVSP10TqB4NsGI=', 'Content-Length': '0'} 2022-09-13T07:39:50.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087752f919f398b0b-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.140 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.141 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.142 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.143 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.144 DEBUG:boto:Method: HEAD 2022-09-13T07:39:50.145 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.146 DEBUG:boto:Data: 2022-09-13T07:39:50.147 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.148 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.148 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.149 DEBUG:boto:Params: {} 2022-09-13T07:39:50.150 DEBUG:boto:Token: None 2022-09-13T07:39:50.151 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.152 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SVH8zZCJS0JAuBRE9hAT8t/bvJc= 2022-09-13T07:39:50.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SVH8zZCJS0JAuBRE9hAT8t/bvJc=', 'Content-Length': '0'} 2022-09-13T07:39:50.155 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097b889da49e50fd4-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.157 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.158 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.159 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.159 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.160 DEBUG:boto:Method: HEAD 2022-09-13T07:39:50.161 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.162 DEBUG:boto:Data: 2022-09-13T07:39:50.163 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.164 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.165 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.166 DEBUG:boto:Params: {} 2022-09-13T07:39:50.167 DEBUG:boto:Token: None 2022-09-13T07:39:50.168 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.169 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SVH8zZCJS0JAuBRE9hAT8t/bvJc= 2022-09-13T07:39:50.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SVH8zZCJS0JAuBRE9hAT8t/bvJc=', 'Content-Length': '0'} 2022-09-13T07:39:50.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000753df13a965a6ae9-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:50.182 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.183 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.184 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.185 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.186 DEBUG:boto:Method: GET 2022-09-13T07:39:50.187 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.188 DEBUG:boto:Data: 2022-09-13T07:39:50.189 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.190 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.191 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.191 DEBUG:boto:Params: {} 2022-09-13T07:39:50.192 DEBUG:boto:Token: None 2022-09-13T07:39:50.193 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.194 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LsUoj7IScVnP3cVSP10TqB4NsGI= 2022-09-13T07:39:50.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LsUoj7IScVnP3cVSP10TqB4NsGI=', 'Content-Length': '0'} 2022-09-13T07:39:50.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005ee54572c0d2de1-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.198 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.199 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.199 DEBUG:boto:path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.200 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.201 DEBUG:boto:Method: GET 2022-09-13T07:39:50.201 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.202 DEBUG:boto:Data: 2022-09-13T07:39:50.202 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.203 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.203 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.204 DEBUG:boto:Params: {} 2022-09-13T07:39:50.205 DEBUG:boto:Token: None 2022-09-13T07:39:50.206 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=ei7viyB1s5vEJjb-DgvRL3K6yeA94ng 2022-09-13T07:39:50.207 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LsUoj7IScVnP3cVSP10TqB4NsGI= 2022-09-13T07:39:50.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LsUoj7IScVnP3cVSP10TqB4NsGI=', 'Content-Length': '0'} 2022-09-13T07:39:50.211 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'ei7viyB1s5vEJjb-DgvRL3K6yeA94ng'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004006bd51f4d4b653-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.258 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:50.259 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:50.260 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.261 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.262 DEBUG:boto:Method: GET 2022-09-13T07:39:50.263 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.264 DEBUG:boto:Data: 2022-09-13T07:39:50.265 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.265 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.266 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.267 DEBUG:boto:Params: {} 2022-09-13T07:39:50.267 DEBUG:boto:Token: None 2022-09-13T07:39:50.268 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/ 2022-09-13T07:39:50.270 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ= 2022-09-13T07:39:50.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ=', 'Content-Length': '0'} 2022-09-13T07:39:50.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f04f6f979d1490ec-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.276 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:50.276 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:50.277 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:50.277 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.278 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.279 DEBUG:boto:Method: GET 2022-09-13T07:39:50.279 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.280 DEBUG:boto:Data: 2022-09-13T07:39:50.280 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.281 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.281 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.282 DEBUG:boto:Params: {} 2022-09-13T07:39:50.282 DEBUG:boto:Token: None 2022-09-13T07:39:50.283 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/ 2022-09-13T07:39:50.283 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ= 2022-09-13T07:39:50.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ=', 'Content-Length': '0'} 2022-09-13T07:39:50.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a31ec29de488121-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.289 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:50.290 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.291 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.291 DEBUG:boto:Method: HEAD 2022-09-13T07:39:50.292 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:50.292 DEBUG:boto:Data: 2022-09-13T07:39:50.293 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.294 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.295 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.297 DEBUG:boto:Params: {} 2022-09-13T07:39:50.298 DEBUG:boto:Token: None 2022-09-13T07:39:50.299 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj 2022-09-13T07:39:50.300 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg= 2022-09-13T07:39:50.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg=', 'Content-Length': '0'} 2022-09-13T07:39:50.306 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bcbb55fe6b33df8-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.308 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.309 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.310 DEBUG:boto:Method: HEAD 2022-09-13T07:39:50.311 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:50.312 DEBUG:boto:Data: 2022-09-13T07:39:50.312 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.314 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.314 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.315 DEBUG:boto:Params: {} 2022-09-13T07:39:50.315 DEBUG:boto:Token: None 2022-09-13T07:39:50.316 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj 2022-09-13T07:39:50.317 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg= 2022-09-13T07:39:50.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg=', 'Content-Length': '0'} 2022-09-13T07:39:50.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ffddd151f83f504a-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:50.323 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.323 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.324 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.325 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.326 DEBUG:boto:Method: GET 2022-09-13T07:39:50.326 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.327 DEBUG:boto:Data: 2022-09-13T07:39:50.328 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.329 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.330 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.331 DEBUG:boto:Params: {} 2022-09-13T07:39:50.332 DEBUG:boto:Token: None 2022-09-13T07:39:50.333 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.335 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q= 2022-09-13T07:39:50.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q=', 'Content-Length': '0'} 2022-09-13T07:39:50.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e997a754209a2a1-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.386 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.388 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.388 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.389 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.389 DEBUG:boto:Method: GET 2022-09-13T07:39:50.390 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.391 DEBUG:boto:Data: 2022-09-13T07:39:50.391 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.392 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.392 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.393 DEBUG:boto:Params: {} 2022-09-13T07:39:50.393 DEBUG:boto:Token: None 2022-09-13T07:39:50.394 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.395 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q= 2022-09-13T07:39:50.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q=', 'Content-Length': '0'} 2022-09-13T07:39:50.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000678ad09076dbedb8-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.399 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:50.399 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:50.400 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.400 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.401 DEBUG:boto:Method: GET 2022-09-13T07:39:50.402 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.403 DEBUG:boto:Data: 2022-09-13T07:39:50.403 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.404 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.404 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.405 DEBUG:boto:Params: {} 2022-09-13T07:39:50.406 DEBUG:boto:Token: None 2022-09-13T07:39:50.407 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/ 2022-09-13T07:39:50.407 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ= 2022-09-13T07:39:50.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ=', 'Content-Length': '0'} 2022-09-13T07:39:50.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007050587d14834694-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.412 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:50.413 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:50.414 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:50.414 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.415 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.415 DEBUG:boto:Method: GET 2022-09-13T07:39:50.416 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.416 DEBUG:boto:Data: 2022-09-13T07:39:50.417 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.417 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.418 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.418 DEBUG:boto:Params: {} 2022-09-13T07:39:50.419 DEBUG:boto:Token: None 2022-09-13T07:39:50.420 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/ 2022-09-13T07:39:50.421 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ= 2022-09-13T07:39:50.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ=', 'Content-Length': '0'} 2022-09-13T07:39:50.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000494e6228cce46afe-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.425 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:50.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:50.427 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.427 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.428 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.428 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.429 DEBUG:boto:Method: GET 2022-09-13T07:39:50.430 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.431 DEBUG:boto:Data: 2022-09-13T07:39:50.432 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.432 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.433 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.434 DEBUG:boto:Params: {} 2022-09-13T07:39:50.435 DEBUG:boto:Token: None 2022-09-13T07:39:50.437 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.437 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:THOGdnVBafmpZguPvK4nJiGJ+ow= 2022-09-13T07:39:50.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:THOGdnVBafmpZguPvK4nJiGJ+ow=', 'Content-Length': '0'} 2022-09-13T07:39:50.440 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005597b6e060711cc9-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.441 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.442 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.443 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.443 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.444 DEBUG:boto:Method: GET 2022-09-13T07:39:50.444 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.445 DEBUG:boto:Data: 2022-09-13T07:39:50.445 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.446 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.446 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.447 DEBUG:boto:Params: {} 2022-09-13T07:39:50.448 DEBUG:boto:Token: None 2022-09-13T07:39:50.448 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.449 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:THOGdnVBafmpZguPvK4nJiGJ+ow= 2022-09-13T07:39:50.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:THOGdnVBafmpZguPvK4nJiGJ+ow=', 'Content-Length': '0'} 2022-09-13T07:39:50.452 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af6b2a2a8b90b0e8-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.453 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.453 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.454 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.454 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.455 DEBUG:boto:Method: HEAD 2022-09-13T07:39:50.455 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.456 DEBUG:boto:Data: 2022-09-13T07:39:50.456 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.457 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.457 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.458 DEBUG:boto:Params: {} 2022-09-13T07:39:50.458 DEBUG:boto:Token: None 2022-09-13T07:39:50.459 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.460 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:miUT5GFeFN6ju87gZcjiZrdX360= 2022-09-13T07:39:50.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:miUT5GFeFN6ju87gZcjiZrdX360=', 'Content-Length': '0'} 2022-09-13T07:39:50.463 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000493007ad3bd110ac-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.464 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.466 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.467 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.468 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.468 DEBUG:boto:Method: HEAD 2022-09-13T07:39:50.470 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.471 DEBUG:boto:Data: 2022-09-13T07:39:50.472 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.472 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.473 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.474 DEBUG:boto:Params: {} 2022-09-13T07:39:50.475 DEBUG:boto:Token: None 2022-09-13T07:39:50.476 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.476 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:miUT5GFeFN6ju87gZcjiZrdX360= 2022-09-13T07:39:50.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:miUT5GFeFN6ju87gZcjiZrdX360=', 'Content-Length': '0'} 2022-09-13T07:39:50.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092eaf8b0353096c3-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:50.481 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.482 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.483 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.484 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.485 DEBUG:boto:Method: GET 2022-09-13T07:39:50.486 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.487 DEBUG:boto:Data: 2022-09-13T07:39:50.488 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.489 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.489 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.490 DEBUG:boto:Params: {} 2022-09-13T07:39:50.491 DEBUG:boto:Token: None 2022-09-13T07:39:50.492 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.493 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:THOGdnVBafmpZguPvK4nJiGJ+ow= 2022-09-13T07:39:50.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:THOGdnVBafmpZguPvK4nJiGJ+ow=', 'Content-Length': '0'} 2022-09-13T07:39:50.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d3703e178e3d2e3-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.542 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.543 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.544 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.545 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.546 DEBUG:boto:Method: GET 2022-09-13T07:39:50.547 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.548 DEBUG:boto:Data: 2022-09-13T07:39:50.549 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.550 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.550 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.551 DEBUG:boto:Params: {} 2022-09-13T07:39:50.552 DEBUG:boto:Token: None 2022-09-13T07:39:50.553 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=pHnhTbZbus4hq.TBh5.hecwSTeKVCaR 2022-09-13T07:39:50.554 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:THOGdnVBafmpZguPvK4nJiGJ+ow= 2022-09-13T07:39:50.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:THOGdnVBafmpZguPvK4nJiGJ+ow=', 'Content-Length': '0'} 2022-09-13T07:39:50.559 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'pHnhTbZbus4hq.TBh5.hecwSTeKVCaR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd1c118cc71bad0c-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.560 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:50.561 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:50.562 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.562 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.563 DEBUG:boto:Method: GET 2022-09-13T07:39:50.564 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.564 DEBUG:boto:Data: 2022-09-13T07:39:50.565 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.565 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.566 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.566 DEBUG:boto:Params: {} 2022-09-13T07:39:50.567 DEBUG:boto:Token: None 2022-09-13T07:39:50.567 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/ 2022-09-13T07:39:50.568 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ= 2022-09-13T07:39:50.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ=', 'Content-Length': '0'} 2022-09-13T07:39:50.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043365f224067315d-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.573 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:50.574 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:50.574 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:50.575 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.575 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.576 DEBUG:boto:Method: GET 2022-09-13T07:39:50.576 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.577 DEBUG:boto:Data: 2022-09-13T07:39:50.577 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.578 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.578 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.579 DEBUG:boto:Params: {} 2022-09-13T07:39:50.580 DEBUG:boto:Token: None 2022-09-13T07:39:50.580 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/ 2022-09-13T07:39:50.581 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ= 2022-09-13T07:39:50.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ=', 'Content-Length': '0'} 2022-09-13T07:39:50.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f10cd6203f4c5049-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.586 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:50.587 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.588 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.589 DEBUG:boto:Method: HEAD 2022-09-13T07:39:50.590 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:50.591 DEBUG:boto:Data: 2022-09-13T07:39:50.592 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.593 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.594 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.594 DEBUG:boto:Params: {} 2022-09-13T07:39:50.595 DEBUG:boto:Token: None 2022-09-13T07:39:50.596 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj 2022-09-13T07:39:50.596 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg= 2022-09-13T07:39:50.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg=', 'Content-Length': '0'} 2022-09-13T07:39:50.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054ddbb567ef6c826-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.601 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.602 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.603 DEBUG:boto:Method: HEAD 2022-09-13T07:39:50.603 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:50.604 DEBUG:boto:Data: 2022-09-13T07:39:50.604 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.605 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.605 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.606 DEBUG:boto:Params: {} 2022-09-13T07:39:50.607 DEBUG:boto:Token: None 2022-09-13T07:39:50.607 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj 2022-09-13T07:39:50.608 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg= 2022-09-13T07:39:50.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg=', 'Content-Length': '0'} 2022-09-13T07:39:50.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3c893031b38d40f-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:50.613 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.614 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.614 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.615 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.616 DEBUG:boto:Method: GET 2022-09-13T07:39:50.616 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.617 DEBUG:boto:Data: 2022-09-13T07:39:50.617 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.618 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.618 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.619 DEBUG:boto:Params: {} 2022-09-13T07:39:50.620 DEBUG:boto:Token: None 2022-09-13T07:39:50.620 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.621 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q= 2022-09-13T07:39:50.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q=', 'Content-Length': '0'} 2022-09-13T07:39:50.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5d88fbda22af11d-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.625 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.626 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.627 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.627 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.628 DEBUG:boto:Method: GET 2022-09-13T07:39:50.628 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.629 DEBUG:boto:Data: 2022-09-13T07:39:50.629 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.630 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.631 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.631 DEBUG:boto:Params: {} 2022-09-13T07:39:50.632 DEBUG:boto:Token: None 2022-09-13T07:39:50.632 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.633 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q= 2022-09-13T07:39:50.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q=', 'Content-Length': '0'} 2022-09-13T07:39:50.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a73de25b02c683d-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.678 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:50.679 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:50.680 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.681 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.682 DEBUG:boto:Method: GET 2022-09-13T07:39:50.683 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.683 DEBUG:boto:Data: 2022-09-13T07:39:50.684 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.685 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.685 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.686 DEBUG:boto:Params: {} 2022-09-13T07:39:50.687 DEBUG:boto:Token: None 2022-09-13T07:39:50.688 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/ 2022-09-13T07:39:50.689 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ= 2022-09-13T07:39:50.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ=', 'Content-Length': '0'} 2022-09-13T07:39:50.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b88c8d4f450710e-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.695 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:50.697 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:50.698 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:50.699 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.700 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.702 DEBUG:boto:Method: GET 2022-09-13T07:39:50.703 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.703 DEBUG:boto:Data: 2022-09-13T07:39:50.704 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.704 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.705 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.705 DEBUG:boto:Params: {} 2022-09-13T07:39:50.706 DEBUG:boto:Token: None 2022-09-13T07:39:50.707 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/ 2022-09-13T07:39:50.707 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ= 2022-09-13T07:39:50.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ=', 'Content-Length': '0'} 2022-09-13T07:39:50.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000927733d355fc976a-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.712 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:50.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:50.713 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.714 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.715 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.715 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.716 DEBUG:boto:Method: GET 2022-09-13T07:39:50.716 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.717 DEBUG:boto:Data: 2022-09-13T07:39:50.718 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.718 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.719 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.719 DEBUG:boto:Params: {} 2022-09-13T07:39:50.720 DEBUG:boto:Token: None 2022-09-13T07:39:50.721 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.722 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nJtGnpC6q1M/J6DK219SY6E+dB0= 2022-09-13T07:39:50.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nJtGnpC6q1M/J6DK219SY6E+dB0=', 'Content-Length': '0'} 2022-09-13T07:39:50.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015d551b078774690-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.726 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.727 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.727 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.728 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.728 DEBUG:boto:Method: GET 2022-09-13T07:39:50.729 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.729 DEBUG:boto:Data: 2022-09-13T07:39:50.730 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.731 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.732 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.732 DEBUG:boto:Params: {} 2022-09-13T07:39:50.733 DEBUG:boto:Token: None 2022-09-13T07:39:50.733 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.734 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nJtGnpC6q1M/J6DK219SY6E+dB0= 2022-09-13T07:39:50.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nJtGnpC6q1M/J6DK219SY6E+dB0=', 'Content-Length': '0'} 2022-09-13T07:39:50.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006870278d14aa4920-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.740 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.741 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.742 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.742 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.743 DEBUG:boto:Method: HEAD 2022-09-13T07:39:50.743 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.744 DEBUG:boto:Data: 2022-09-13T07:39:50.745 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.745 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.746 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.746 DEBUG:boto:Params: {} 2022-09-13T07:39:50.747 DEBUG:boto:Token: None 2022-09-13T07:39:50.747 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.748 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uBFeq67o9GuEwK9tnAl1k7pwAfo= 2022-09-13T07:39:50.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uBFeq67o9GuEwK9tnAl1k7pwAfo=', 'Content-Length': '0'} 2022-09-13T07:39:50.751 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000acd25316155304d1-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.752 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.752 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.753 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.753 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.754 DEBUG:boto:Method: HEAD 2022-09-13T07:39:50.754 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.755 DEBUG:boto:Data: 2022-09-13T07:39:50.756 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.756 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.757 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.757 DEBUG:boto:Params: {} 2022-09-13T07:39:50.758 DEBUG:boto:Token: None 2022-09-13T07:39:50.759 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.759 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uBFeq67o9GuEwK9tnAl1k7pwAfo= 2022-09-13T07:39:50.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uBFeq67o9GuEwK9tnAl1k7pwAfo=', 'Content-Length': '0'} 2022-09-13T07:39:50.762 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000908c7c1d2c211731-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:50.764 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.765 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.766 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.767 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.767 DEBUG:boto:Method: GET 2022-09-13T07:39:50.768 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.769 DEBUG:boto:Data: 2022-09-13T07:39:50.770 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.771 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.772 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.773 DEBUG:boto:Params: {} 2022-09-13T07:39:50.774 DEBUG:boto:Token: None 2022-09-13T07:39:50.775 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.775 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nJtGnpC6q1M/J6DK219SY6E+dB0= 2022-09-13T07:39:50.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nJtGnpC6q1M/J6DK219SY6E+dB0=', 'Content-Length': '0'} 2022-09-13T07:39:50.778 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038db127709c6f903-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.822 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.823 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.824 DEBUG:boto:path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.824 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.825 DEBUG:boto:Method: GET 2022-09-13T07:39:50.825 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.826 DEBUG:boto:Data: 2022-09-13T07:39:50.826 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.827 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.827 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.828 DEBUG:boto:Params: {} 2022-09-13T07:39:50.829 DEBUG:boto:Token: None 2022-09-13T07:39:50.830 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl 2022-09-13T07:39:50.831 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nJtGnpC6q1M/J6DK219SY6E+dB0= 2022-09-13T07:39:50.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nJtGnpC6q1M/J6DK219SY6E+dB0=', 'Content-Length': '0'} 2022-09-13T07:39:50.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '8bP-bkYz6JpHUV25JyCvc0fDB8aZwsl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006aa705a0cc2a7788-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.835 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:50.835 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:50.836 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.836 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.837 DEBUG:boto:Method: GET 2022-09-13T07:39:50.838 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.838 DEBUG:boto:Data: 2022-09-13T07:39:50.839 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.839 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.840 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.840 DEBUG:boto:Params: {} 2022-09-13T07:39:50.841 DEBUG:boto:Token: None 2022-09-13T07:39:50.842 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/ 2022-09-13T07:39:50.842 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ= 2022-09-13T07:39:50.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ=', 'Content-Length': '0'} 2022-09-13T07:39:50.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5a98e1126b9819f-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.847 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:50.848 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:50.849 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:50.849 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.850 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.850 DEBUG:boto:Method: GET 2022-09-13T07:39:50.851 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.851 DEBUG:boto:Data: 2022-09-13T07:39:50.852 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.852 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.853 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.853 DEBUG:boto:Params: {} 2022-09-13T07:39:50.854 DEBUG:boto:Token: None 2022-09-13T07:39:50.855 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/ 2022-09-13T07:39:50.855 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ= 2022-09-13T07:39:50.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ=', 'Content-Length': '0'} 2022-09-13T07:39:50.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014eda8fe2a2d2a89-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.860 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:50.860 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.861 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.861 DEBUG:boto:Method: HEAD 2022-09-13T07:39:50.862 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:50.862 DEBUG:boto:Data: 2022-09-13T07:39:50.863 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.863 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.864 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.865 DEBUG:boto:Params: {} 2022-09-13T07:39:50.865 DEBUG:boto:Token: None 2022-09-13T07:39:50.866 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj 2022-09-13T07:39:50.866 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg= 2022-09-13T07:39:50.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg=', 'Content-Length': '0'} 2022-09-13T07:39:50.871 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000daa4a276f2bd4c28-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.872 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.872 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.873 DEBUG:boto:Method: HEAD 2022-09-13T07:39:50.873 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:50.874 DEBUG:boto:Data: 2022-09-13T07:39:50.874 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.875 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.875 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.876 DEBUG:boto:Params: {} 2022-09-13T07:39:50.876 DEBUG:boto:Token: None 2022-09-13T07:39:50.877 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj 2022-09-13T07:39:50.877 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg= 2022-09-13T07:39:50.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cN1y4YUBx5fFNsHD4ffXM5Yl3kg=', 'Content-Length': '0'} 2022-09-13T07:39:50.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a29dc245bc1a2b8-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:50.883 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.883 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.884 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.884 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.885 DEBUG:boto:Method: GET 2022-09-13T07:39:50.885 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.886 DEBUG:boto:Data: 2022-09-13T07:39:50.886 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.887 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.887 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.888 DEBUG:boto:Params: {} 2022-09-13T07:39:50.888 DEBUG:boto:Token: None 2022-09-13T07:39:50.889 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.890 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q= 2022-09-13T07:39:50.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q=', 'Content-Length': '0'} 2022-09-13T07:39:50.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c819aadb79dd13b0-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.934 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.935 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.935 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.936 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.936 DEBUG:boto:Method: GET 2022-09-13T07:39:50.937 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.937 DEBUG:boto:Data: 2022-09-13T07:39:50.938 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.938 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.939 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.939 DEBUG:boto:Params: {} 2022-09-13T07:39:50.940 DEBUG:boto:Token: None 2022-09-13T07:39:50.941 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:50.942 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q= 2022-09-13T07:39:50.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:w7r9PE2U9aCDOggiiGF+QQZDZ2Q=', 'Content-Length': '0'} 2022-09-13T07:39:50.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f343c9388193df7e-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.946 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:50.946 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:50.947 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.947 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.948 DEBUG:boto:Method: GET 2022-09-13T07:39:50.948 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.949 DEBUG:boto:Data: 2022-09-13T07:39:50.949 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.950 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.951 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.951 DEBUG:boto:Params: {} 2022-09-13T07:39:50.952 DEBUG:boto:Token: None 2022-09-13T07:39:50.952 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/ 2022-09-13T07:39:50.953 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ= 2022-09-13T07:39:50.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ=', 'Content-Length': '0'} 2022-09-13T07:39:50.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f61efc65002072c9-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.958 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:50.958 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:50.959 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:50.960 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.960 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.961 DEBUG:boto:Method: GET 2022-09-13T07:39:50.961 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:50.962 DEBUG:boto:Data: 2022-09-13T07:39:50.963 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.963 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.964 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.965 DEBUG:boto:Params: {} 2022-09-13T07:39:50.965 DEBUG:boto:Token: None 2022-09-13T07:39:50.966 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/ 2022-09-13T07:39:50.967 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ= 2022-09-13T07:39:50.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rIkTVyLXO+jdpXsWgK+Pl5F2lxQ=', 'Content-Length': '0'} 2022-09-13T07:39:50.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006195c58dd9642fbd-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.975 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:50.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:50.977 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.978 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.979 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:50.980 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:50.981 DEBUG:boto:Method: GET 2022-09-13T07:39:50.981 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:50.982 DEBUG:boto:Data: 2022-09-13T07:39:50.983 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.984 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:50.984 DEBUG:boto:Port: 8001 2022-09-13T07:39:50.985 DEBUG:boto:Params: {} 2022-09-13T07:39:50.986 DEBUG:boto:Token: None 2022-09-13T07:39:50.987 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:50.988 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0oImDJJ5Fj6nEeAYaMMR2SIMrjQ= 2022-09-13T07:39:50.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0oImDJJ5Fj6nEeAYaMMR2SIMrjQ=', 'Content-Length': '0'} 2022-09-13T07:39:50.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003de93dd6c011a7f3-00632033c6-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:50.992 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:50.993 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:50.993 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:50.994 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:50.994 DEBUG:boto:Method: GET 2022-09-13T07:39:50.995 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:50.995 DEBUG:boto:Data: 2022-09-13T07:39:50.996 DEBUG:boto:Headers: {} 2022-09-13T07:39:50.996 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:50.997 DEBUG:boto:Port: 8000 2022-09-13T07:39:50.997 DEBUG:boto:Params: {} 2022-09-13T07:39:50.998 DEBUG:boto:Token: None 2022-09-13T07:39:50.998 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:50 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:50.999 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0oImDJJ5Fj6nEeAYaMMR2SIMrjQ= 2022-09-13T07:39:51.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:50 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0oImDJJ5Fj6nEeAYaMMR2SIMrjQ=', 'Content-Length': '0'} 2022-09-13T07:39:51.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7adf2f91ba07092-00632033c6-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.003 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.003 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.004 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.004 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.005 DEBUG:boto:Method: HEAD 2022-09-13T07:39:51.005 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.006 DEBUG:boto:Data: 2022-09-13T07:39:51.006 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.007 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.007 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.008 DEBUG:boto:Params: {} 2022-09-13T07:39:51.008 DEBUG:boto:Token: None 2022-09-13T07:39:51.009 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.009 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:doGdNofeTy4umwqsC2HUmwNx3K0= 2022-09-13T07:39:51.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:doGdNofeTy4umwqsC2HUmwNx3K0=', 'Content-Length': '0'} 2022-09-13T07:39:51.012 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008778f98a10ac1664-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.013 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.013 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.014 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.014 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.015 DEBUG:boto:Method: HEAD 2022-09-13T07:39:51.015 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.016 DEBUG:boto:Data: 2022-09-13T07:39:51.016 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.017 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.017 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.018 DEBUG:boto:Params: {} 2022-09-13T07:39:51.018 DEBUG:boto:Token: None 2022-09-13T07:39:51.019 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.019 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:doGdNofeTy4umwqsC2HUmwNx3K0= 2022-09-13T07:39:51.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:doGdNofeTy4umwqsC2HUmwNx3K0=', 'Content-Length': '0'} 2022-09-13T07:39:51.022 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6ef8cb216411dff-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:51.023 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.024 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.024 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.025 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.026 DEBUG:boto:Method: GET 2022-09-13T07:39:51.026 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.027 DEBUG:boto:Data: 2022-09-13T07:39:51.027 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.028 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.028 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.029 DEBUG:boto:Params: {} 2022-09-13T07:39:51.029 DEBUG:boto:Token: None 2022-09-13T07:39:51.030 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.030 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LtXyk2XVAv3in3sM2HY/ysL13PU= 2022-09-13T07:39:51.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LtXyk2XVAv3in3sM2HY/ysL13PU=', 'Content-Length': '0'} 2022-09-13T07:39:51.033 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae52f5c057bf4b25-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.034 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.034 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.035 DEBUG:boto:path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.035 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.036 DEBUG:boto:Method: GET 2022-09-13T07:39:51.036 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.037 DEBUG:boto:Data: 2022-09-13T07:39:51.037 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.038 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.038 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.039 DEBUG:boto:Params: {} 2022-09-13T07:39:51.040 DEBUG:boto:Token: None 2022-09-13T07:39:51.040 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC 2022-09-13T07:39:51.041 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LtXyk2XVAv3in3sM2HY/ysL13PU= 2022-09-13T07:39:51.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LtXyk2XVAv3in3sM2HY/ysL13PU=', 'Content-Length': '0'} 2022-09-13T07:39:51.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'cPIo6H3tnk3IHLM4mBy4H8IXyB8CUPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de1bd6b272ecdb69-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.086 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:51.087 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:51.088 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.088 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.089 DEBUG:boto:Method: GET 2022-09-13T07:39:51.089 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.090 DEBUG:boto:Data: 2022-09-13T07:39:51.090 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.091 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.091 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.092 DEBUG:boto:Params: {} 2022-09-13T07:39:51.093 DEBUG:boto:Token: None 2022-09-13T07:39:51.093 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/ 2022-09-13T07:39:51.094 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk= 2022-09-13T07:39:51.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk=', 'Content-Length': '0'} 2022-09-13T07:39:51.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068fd88f91662b6e9-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.099 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:51.099 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:51.100 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:51.101 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.101 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.102 DEBUG:boto:Method: GET 2022-09-13T07:39:51.102 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.103 DEBUG:boto:Data: 2022-09-13T07:39:51.103 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.104 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.104 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.105 DEBUG:boto:Params: {} 2022-09-13T07:39:51.105 DEBUG:boto:Token: None 2022-09-13T07:39:51.106 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/ 2022-09-13T07:39:51.106 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk= 2022-09-13T07:39:51.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk=', 'Content-Length': '0'} 2022-09-13T07:39:51.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082b8a58146132bea-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.111 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:51.112 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.112 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.113 DEBUG:boto:Method: HEAD 2022-09-13T07:39:51.113 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:51.114 DEBUG:boto:Data: 2022-09-13T07:39:51.114 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.115 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.115 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.116 DEBUG:boto:Params: {} 2022-09-13T07:39:51.116 DEBUG:boto:Token: None 2022-09-13T07:39:51.117 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj 2022-09-13T07:39:51.117 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8pXOOFsYFjIl/KOyk5O91j9XtD4= 2022-09-13T07:39:51.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8pXOOFsYFjIl/KOyk5O91j9XtD4=', 'Content-Length': '0'} 2022-09-13T07:39:51.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d89f2af811251f4-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.122 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.123 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.123 DEBUG:boto:Method: HEAD 2022-09-13T07:39:51.124 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:51.124 DEBUG:boto:Data: 2022-09-13T07:39:51.125 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.125 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.126 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.126 DEBUG:boto:Params: {} 2022-09-13T07:39:51.127 DEBUG:boto:Token: None 2022-09-13T07:39:51.127 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj 2022-09-13T07:39:51.128 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8pXOOFsYFjIl/KOyk5O91j9XtD4= 2022-09-13T07:39:51.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8pXOOFsYFjIl/KOyk5O91j9XtD4=', 'Content-Length': '0'} 2022-09-13T07:39:51.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa911a2bae59e7df-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:51.133 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.134 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.134 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.135 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.135 DEBUG:boto:Method: GET 2022-09-13T07:39:51.136 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.136 DEBUG:boto:Data: 2022-09-13T07:39:51.137 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.137 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.138 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.138 DEBUG:boto:Params: {} 2022-09-13T07:39:51.139 DEBUG:boto:Token: None 2022-09-13T07:39:51.140 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.140 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5kOfndPvmRa/amy4C2I9afIl2no= 2022-09-13T07:39:51.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5kOfndPvmRa/amy4C2I9afIl2no=', 'Content-Length': '0'} 2022-09-13T07:39:51.143 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0d87cace6645a30-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.144 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.145 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.145 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.146 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.146 DEBUG:boto:Method: GET 2022-09-13T07:39:51.147 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.147 DEBUG:boto:Data: 2022-09-13T07:39:51.148 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.148 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.149 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.149 DEBUG:boto:Params: {} 2022-09-13T07:39:51.150 DEBUG:boto:Token: None 2022-09-13T07:39:51.151 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.151 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5kOfndPvmRa/amy4C2I9afIl2no= 2022-09-13T07:39:51.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5kOfndPvmRa/amy4C2I9afIl2no=', 'Content-Length': '0'} 2022-09-13T07:39:51.154 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c669b2bdf10840ee-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.202 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:51.203 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:51.203 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.204 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.204 DEBUG:boto:Method: GET 2022-09-13T07:39:51.205 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.205 DEBUG:boto:Data: 2022-09-13T07:39:51.206 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.206 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.207 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.207 DEBUG:boto:Params: {} 2022-09-13T07:39:51.208 DEBUG:boto:Token: None 2022-09-13T07:39:51.208 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/ 2022-09-13T07:39:51.209 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk= 2022-09-13T07:39:51.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk=', 'Content-Length': '0'} 2022-09-13T07:39:51.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c21823aa753e280-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.214 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:51.215 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:51.215 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:51.216 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.216 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.217 DEBUG:boto:Method: GET 2022-09-13T07:39:51.218 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.219 DEBUG:boto:Data: 2022-09-13T07:39:51.219 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.220 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.220 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.221 DEBUG:boto:Params: {} 2022-09-13T07:39:51.221 DEBUG:boto:Token: None 2022-09-13T07:39:51.222 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/ 2022-09-13T07:39:51.223 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk= 2022-09-13T07:39:51.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk=', 'Content-Length': '0'} 2022-09-13T07:39:51.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002190eba25c78a88d-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.228 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:51.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:51.230 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.231 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.232 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.232 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.233 DEBUG:boto:Method: GET 2022-09-13T07:39:51.233 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.234 DEBUG:boto:Data: 2022-09-13T07:39:51.234 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.235 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.235 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.236 DEBUG:boto:Params: {} 2022-09-13T07:39:51.237 DEBUG:boto:Token: None 2022-09-13T07:39:51.237 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.238 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tboKkfyELz5rU5u6G0t42F7ArM8= 2022-09-13T07:39:51.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tboKkfyELz5rU5u6G0t42F7ArM8=', 'Content-Length': '0'} 2022-09-13T07:39:51.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e45eb6fa7b1d615-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.241 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.242 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.242 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.243 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.243 DEBUG:boto:Method: GET 2022-09-13T07:39:51.244 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.244 DEBUG:boto:Data: 2022-09-13T07:39:51.245 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.245 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.246 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.246 DEBUG:boto:Params: {} 2022-09-13T07:39:51.247 DEBUG:boto:Token: None 2022-09-13T07:39:51.248 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.248 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tboKkfyELz5rU5u6G0t42F7ArM8= 2022-09-13T07:39:51.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tboKkfyELz5rU5u6G0t42F7ArM8=', 'Content-Length': '0'} 2022-09-13T07:39:51.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da3586d7777cb11b-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.255 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.256 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.257 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.258 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.259 DEBUG:boto:Method: HEAD 2022-09-13T07:39:51.260 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.261 DEBUG:boto:Data: 2022-09-13T07:39:51.261 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.262 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.262 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.263 DEBUG:boto:Params: {} 2022-09-13T07:39:51.263 DEBUG:boto:Token: None 2022-09-13T07:39:51.264 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.265 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6rNLkNfNa3IjTZUTEpAkc5Y9EwM= 2022-09-13T07:39:51.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6rNLkNfNa3IjTZUTEpAkc5Y9EwM=', 'Content-Length': '0'} 2022-09-13T07:39:51.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000496290ae26882e2a-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.268 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.269 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.269 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.270 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.270 DEBUG:boto:Method: HEAD 2022-09-13T07:39:51.271 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.272 DEBUG:boto:Data: 2022-09-13T07:39:51.272 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.273 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.275 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.275 DEBUG:boto:Params: {} 2022-09-13T07:39:51.277 DEBUG:boto:Token: None 2022-09-13T07:39:51.278 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.279 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6rNLkNfNa3IjTZUTEpAkc5Y9EwM= 2022-09-13T07:39:51.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6rNLkNfNa3IjTZUTEpAkc5Y9EwM=', 'Content-Length': '0'} 2022-09-13T07:39:51.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ae3e564044430ca-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:51.283 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.283 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.284 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.284 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.285 DEBUG:boto:Method: GET 2022-09-13T07:39:51.285 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.286 DEBUG:boto:Data: 2022-09-13T07:39:51.287 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.288 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.289 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.290 DEBUG:boto:Params: {} 2022-09-13T07:39:51.291 DEBUG:boto:Token: None 2022-09-13T07:39:51.292 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.293 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tboKkfyELz5rU5u6G0t42F7ArM8= 2022-09-13T07:39:51.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tboKkfyELz5rU5u6G0t42F7ArM8=', 'Content-Length': '0'} 2022-09-13T07:39:51.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000787265bf5b242ba5-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.298 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.299 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.300 DEBUG:boto:path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.301 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.302 DEBUG:boto:Method: GET 2022-09-13T07:39:51.303 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.304 DEBUG:boto:Data: 2022-09-13T07:39:51.305 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.306 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.307 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.308 DEBUG:boto:Params: {} 2022-09-13T07:39:51.309 DEBUG:boto:Token: None 2022-09-13T07:39:51.310 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl- 2022-09-13T07:39:51.311 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tboKkfyELz5rU5u6G0t42F7ArM8= 2022-09-13T07:39:51.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tboKkfyELz5rU5u6G0t42F7ArM8=', 'Content-Length': '0'} 2022-09-13T07:39:51.315 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'hbDI.zGtg2Z2CdX7v99Tk7qGzYDwzl-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033bb4631490e572b-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.358 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:51.359 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:51.359 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.360 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.361 DEBUG:boto:Method: GET 2022-09-13T07:39:51.362 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.362 DEBUG:boto:Data: 2022-09-13T07:39:51.363 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.364 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.365 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.367 DEBUG:boto:Params: {} 2022-09-13T07:39:51.368 DEBUG:boto:Token: None 2022-09-13T07:39:51.369 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/ 2022-09-13T07:39:51.370 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk= 2022-09-13T07:39:51.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk=', 'Content-Length': '0'} 2022-09-13T07:39:51.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089bb15180be1b9c1-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.376 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:51.377 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:51.378 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:51.382 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.382 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.383 DEBUG:boto:Method: GET 2022-09-13T07:39:51.383 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.384 DEBUG:boto:Data: 2022-09-13T07:39:51.384 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.385 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.385 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.386 DEBUG:boto:Params: {} 2022-09-13T07:39:51.387 DEBUG:boto:Token: None 2022-09-13T07:39:51.387 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/ 2022-09-13T07:39:51.388 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk= 2022-09-13T07:39:51.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk=', 'Content-Length': '0'} 2022-09-13T07:39:51.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000079af22f7cb899ae-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.392 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:51.393 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.394 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.394 DEBUG:boto:Method: HEAD 2022-09-13T07:39:51.395 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:51.395 DEBUG:boto:Data: 2022-09-13T07:39:51.396 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.397 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.397 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.398 DEBUG:boto:Params: {} 2022-09-13T07:39:51.398 DEBUG:boto:Token: None 2022-09-13T07:39:51.399 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj 2022-09-13T07:39:51.399 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8pXOOFsYFjIl/KOyk5O91j9XtD4= 2022-09-13T07:39:51.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8pXOOFsYFjIl/KOyk5O91j9XtD4=', 'Content-Length': '0'} 2022-09-13T07:39:51.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079aa7752e657bc08-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.405 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.406 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.406 DEBUG:boto:Method: HEAD 2022-09-13T07:39:51.407 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:51.408 DEBUG:boto:Data: 2022-09-13T07:39:51.409 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.409 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.410 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.411 DEBUG:boto:Params: {} 2022-09-13T07:39:51.412 DEBUG:boto:Token: None 2022-09-13T07:39:51.412 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj 2022-09-13T07:39:51.413 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8pXOOFsYFjIl/KOyk5O91j9XtD4= 2022-09-13T07:39:51.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8pXOOFsYFjIl/KOyk5O91j9XtD4=', 'Content-Length': '0'} 2022-09-13T07:39:51.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bfb6d405f96468e-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:51.419 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.419 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.420 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.421 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.422 DEBUG:boto:Method: GET 2022-09-13T07:39:51.423 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.423 DEBUG:boto:Data: 2022-09-13T07:39:51.424 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.424 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.425 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.426 DEBUG:boto:Params: {} 2022-09-13T07:39:51.427 DEBUG:boto:Token: None 2022-09-13T07:39:51.427 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.428 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5kOfndPvmRa/amy4C2I9afIl2no= 2022-09-13T07:39:51.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5kOfndPvmRa/amy4C2I9afIl2no=', 'Content-Length': '0'} 2022-09-13T07:39:51.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000259ceb922be87814-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.474 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.475 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.475 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.476 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.476 DEBUG:boto:Method: GET 2022-09-13T07:39:51.477 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.477 DEBUG:boto:Data: 2022-09-13T07:39:51.478 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.479 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.482 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.483 DEBUG:boto:Params: {} 2022-09-13T07:39:51.484 DEBUG:boto:Token: None 2022-09-13T07:39:51.485 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.486 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5kOfndPvmRa/amy4C2I9afIl2no= 2022-09-13T07:39:51.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5kOfndPvmRa/amy4C2I9afIl2no=', 'Content-Length': '0'} 2022-09-13T07:39:51.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b858fd00d3336f92-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.492 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:51.493 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:51.494 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.495 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.496 DEBUG:boto:Method: GET 2022-09-13T07:39:51.497 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.498 DEBUG:boto:Data: 2022-09-13T07:39:51.499 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.500 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.501 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.502 DEBUG:boto:Params: {} 2022-09-13T07:39:51.503 DEBUG:boto:Token: None 2022-09-13T07:39:51.504 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/ 2022-09-13T07:39:51.505 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk= 2022-09-13T07:39:51.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk=', 'Content-Length': '0'} 2022-09-13T07:39:51.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007525cd743511c195-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.511 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:51.512 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:51.513 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:51.515 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.516 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.517 DEBUG:boto:Method: GET 2022-09-13T07:39:51.518 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.519 DEBUG:boto:Data: 2022-09-13T07:39:51.520 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.521 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.522 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.523 DEBUG:boto:Params: {} 2022-09-13T07:39:51.524 DEBUG:boto:Token: None 2022-09-13T07:39:51.525 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/ 2022-09-13T07:39:51.526 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk= 2022-09-13T07:39:51.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk=', 'Content-Length': '0'} 2022-09-13T07:39:51.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030e233d661ea74f1-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.574 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:51.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:51.577 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.578 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.579 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.580 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.581 DEBUG:boto:Method: GET 2022-09-13T07:39:51.581 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.582 DEBUG:boto:Data: 2022-09-13T07:39:51.583 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.584 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.585 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.587 DEBUG:boto:Params: {} 2022-09-13T07:39:51.588 DEBUG:boto:Token: None 2022-09-13T07:39:51.589 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.590 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bTGE4VqyFspfrqsIz7Etyx4GK7k= 2022-09-13T07:39:51.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bTGE4VqyFspfrqsIz7Etyx4GK7k=', 'Content-Length': '0'} 2022-09-13T07:39:51.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003de02a84188eda91-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.599 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.600 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.601 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.602 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.603 DEBUG:boto:Method: GET 2022-09-13T07:39:51.603 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.604 DEBUG:boto:Data: 2022-09-13T07:39:51.605 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.606 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.608 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.609 DEBUG:boto:Params: {} 2022-09-13T07:39:51.610 DEBUG:boto:Token: None 2022-09-13T07:39:51.611 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.612 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bTGE4VqyFspfrqsIz7Etyx4GK7k= 2022-09-13T07:39:51.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bTGE4VqyFspfrqsIz7Etyx4GK7k=', 'Content-Length': '0'} 2022-09-13T07:39:51.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c76cc70acf4f4594-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.618 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.619 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.620 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.621 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.622 DEBUG:boto:Method: HEAD 2022-09-13T07:39:51.624 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.624 DEBUG:boto:Data: 2022-09-13T07:39:51.625 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.625 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.626 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.626 DEBUG:boto:Params: {} 2022-09-13T07:39:51.627 DEBUG:boto:Token: None 2022-09-13T07:39:51.628 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.628 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tDQAdO/oiuf9EyKt4rQI1w0hAkU= 2022-09-13T07:39:51.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tDQAdO/oiuf9EyKt4rQI1w0hAkU=', 'Content-Length': '0'} 2022-09-13T07:39:51.631 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000062248e55137d16-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.632 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.632 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.633 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.633 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.634 DEBUG:boto:Method: HEAD 2022-09-13T07:39:51.634 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.635 DEBUG:boto:Data: 2022-09-13T07:39:51.635 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.636 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.636 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.637 DEBUG:boto:Params: {} 2022-09-13T07:39:51.638 DEBUG:boto:Token: None 2022-09-13T07:39:51.638 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.639 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tDQAdO/oiuf9EyKt4rQI1w0hAkU= 2022-09-13T07:39:51.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tDQAdO/oiuf9EyKt4rQI1w0hAkU=', 'Content-Length': '0'} 2022-09-13T07:39:51.642 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007947a209b6a1b989-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:51.644 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.644 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.645 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.645 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.646 DEBUG:boto:Method: GET 2022-09-13T07:39:51.646 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.647 DEBUG:boto:Data: 2022-09-13T07:39:51.648 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.649 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.650 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.651 DEBUG:boto:Params: {} 2022-09-13T07:39:51.652 DEBUG:boto:Token: None 2022-09-13T07:39:51.653 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.654 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bTGE4VqyFspfrqsIz7Etyx4GK7k= 2022-09-13T07:39:51.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bTGE4VqyFspfrqsIz7Etyx4GK7k=', 'Content-Length': '0'} 2022-09-13T07:39:51.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000368938127b5c6815-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.698 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.699 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.700 DEBUG:boto:path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.700 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.701 DEBUG:boto:Method: GET 2022-09-13T07:39:51.702 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.702 DEBUG:boto:Data: 2022-09-13T07:39:51.703 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.703 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.704 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.704 DEBUG:boto:Params: {} 2022-09-13T07:39:51.705 DEBUG:boto:Token: None 2022-09-13T07:39:51.706 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy 2022-09-13T07:39:51.706 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bTGE4VqyFspfrqsIz7Etyx4GK7k= 2022-09-13T07:39:51.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bTGE4VqyFspfrqsIz7Etyx4GK7k=', 'Content-Length': '0'} 2022-09-13T07:39:51.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'p3VqUFgxLYLsisMjs5Rrmm15ZQudhjy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fe91d80a37bfd15-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.710 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:51.711 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:51.712 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.712 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.713 DEBUG:boto:Method: GET 2022-09-13T07:39:51.713 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.714 DEBUG:boto:Data: 2022-09-13T07:39:51.715 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.715 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.716 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.716 DEBUG:boto:Params: {} 2022-09-13T07:39:51.717 DEBUG:boto:Token: None 2022-09-13T07:39:51.717 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/ 2022-09-13T07:39:51.718 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk= 2022-09-13T07:39:51.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk=', 'Content-Length': '0'} 2022-09-13T07:39:51.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a28b20cf2e041cc0-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.723 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:51.725 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:51.725 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:51.726 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.727 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.728 DEBUG:boto:Method: GET 2022-09-13T07:39:51.729 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.730 DEBUG:boto:Data: 2022-09-13T07:39:51.731 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.731 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.732 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.733 DEBUG:boto:Params: {} 2022-09-13T07:39:51.734 DEBUG:boto:Token: None 2022-09-13T07:39:51.735 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/ 2022-09-13T07:39:51.736 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk= 2022-09-13T07:39:51.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk=', 'Content-Length': '0'} 2022-09-13T07:39:51.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d3c4b408df0c5e6-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.753 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:51.754 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.755 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.755 DEBUG:boto:Method: HEAD 2022-09-13T07:39:51.756 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:51.756 DEBUG:boto:Data: 2022-09-13T07:39:51.757 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.757 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.758 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.759 DEBUG:boto:Params: {} 2022-09-13T07:39:51.759 DEBUG:boto:Token: None 2022-09-13T07:39:51.760 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj 2022-09-13T07:39:51.760 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8pXOOFsYFjIl/KOyk5O91j9XtD4= 2022-09-13T07:39:51.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8pXOOFsYFjIl/KOyk5O91j9XtD4=', 'Content-Length': '0'} 2022-09-13T07:39:51.765 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8698d3db1cd0cce-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.766 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.766 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.767 DEBUG:boto:Method: HEAD 2022-09-13T07:39:51.768 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:51.768 DEBUG:boto:Data: 2022-09-13T07:39:51.769 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.769 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.770 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.770 DEBUG:boto:Params: {} 2022-09-13T07:39:51.771 DEBUG:boto:Token: None 2022-09-13T07:39:51.771 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj 2022-09-13T07:39:51.772 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:8pXOOFsYFjIl/KOyk5O91j9XtD4= 2022-09-13T07:39:51.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:8pXOOFsYFjIl/KOyk5O91j9XtD4=', 'Content-Length': '0'} 2022-09-13T07:39:51.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053558619da596d97-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:51.778 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.778 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.779 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.780 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.780 DEBUG:boto:Method: GET 2022-09-13T07:39:51.781 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.781 DEBUG:boto:Data: 2022-09-13T07:39:51.782 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.782 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.783 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.783 DEBUG:boto:Params: {} 2022-09-13T07:39:51.784 DEBUG:boto:Token: None 2022-09-13T07:39:51.785 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.785 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5kOfndPvmRa/amy4C2I9afIl2no= 2022-09-13T07:39:51.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5kOfndPvmRa/amy4C2I9afIl2no=', 'Content-Length': '0'} 2022-09-13T07:39:51.789 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001855715441b15a89-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.790 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.790 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.791 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.791 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.792 DEBUG:boto:Method: GET 2022-09-13T07:39:51.793 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.793 DEBUG:boto:Data: 2022-09-13T07:39:51.794 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.794 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.795 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.795 DEBUG:boto:Params: {} 2022-09-13T07:39:51.796 DEBUG:boto:Token: None 2022-09-13T07:39:51.797 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:51.798 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5kOfndPvmRa/amy4C2I9afIl2no= 2022-09-13T07:39:51.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5kOfndPvmRa/amy4C2I9afIl2no=', 'Content-Length': '0'} 2022-09-13T07:39:51.801 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057ba3a69a7ba8139-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.802 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:51.802 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:51.803 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.804 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.805 DEBUG:boto:Method: GET 2022-09-13T07:39:51.806 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.807 DEBUG:boto:Data: 2022-09-13T07:39:51.808 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.809 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.809 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.810 DEBUG:boto:Params: {} 2022-09-13T07:39:51.811 DEBUG:boto:Token: None 2022-09-13T07:39:51.812 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/ 2022-09-13T07:39:51.813 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk= 2022-09-13T07:39:51.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk=', 'Content-Length': '0'} 2022-09-13T07:39:51.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004281c61ca1e5248d-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.823 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:51.824 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:51.825 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:51.826 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.826 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.827 DEBUG:boto:Method: GET 2022-09-13T07:39:51.828 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.829 DEBUG:boto:Data: 2022-09-13T07:39:51.830 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.831 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.834 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.835 DEBUG:boto:Params: {} 2022-09-13T07:39:51.835 DEBUG:boto:Token: None 2022-09-13T07:39:51.836 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/ 2022-09-13T07:39:51.837 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk= 2022-09-13T07:39:51.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk=', 'Content-Length': '0'} 2022-09-13T07:39:51.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000251288e7336a7118-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.843 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:51.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:51.845 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.845 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.846 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.847 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.847 DEBUG:boto:Method: GET 2022-09-13T07:39:51.848 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.848 DEBUG:boto:Data: 2022-09-13T07:39:51.849 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.849 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.850 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.850 DEBUG:boto:Params: {} 2022-09-13T07:39:51.851 DEBUG:boto:Token: None 2022-09-13T07:39:51.852 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.852 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jkN74vNoXPYGK6uWyOHgCu8Y5sk= 2022-09-13T07:39:51.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jkN74vNoXPYGK6uWyOHgCu8Y5sk=', 'Content-Length': '0'} 2022-09-13T07:39:51.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001006991e0242eec2-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.856 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.857 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.857 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.858 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.859 DEBUG:boto:Method: GET 2022-09-13T07:39:51.859 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.860 DEBUG:boto:Data: 2022-09-13T07:39:51.860 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.861 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.861 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.862 DEBUG:boto:Params: {} 2022-09-13T07:39:51.863 DEBUG:boto:Token: None 2022-09-13T07:39:51.863 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.864 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jkN74vNoXPYGK6uWyOHgCu8Y5sk= 2022-09-13T07:39:51.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jkN74vNoXPYGK6uWyOHgCu8Y5sk=', 'Content-Length': '0'} 2022-09-13T07:39:51.867 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b6de67b2ab73694-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.868 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.869 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.869 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.870 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.870 DEBUG:boto:Method: HEAD 2022-09-13T07:39:51.871 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.871 DEBUG:boto:Data: 2022-09-13T07:39:51.872 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.872 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.873 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.873 DEBUG:boto:Params: {} 2022-09-13T07:39:51.874 DEBUG:boto:Token: None 2022-09-13T07:39:51.876 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.876 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KWVvjgbNNP1VUDdyUknD/SFB1EA= 2022-09-13T07:39:51.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KWVvjgbNNP1VUDdyUknD/SFB1EA=', 'Content-Length': '0'} 2022-09-13T07:39:51.879 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006bc57e5536bdfc0-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.880 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.880 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.881 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.881 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.881 DEBUG:boto:Method: HEAD 2022-09-13T07:39:51.882 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.883 DEBUG:boto:Data: 2022-09-13T07:39:51.883 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.884 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.884 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.885 DEBUG:boto:Params: {} 2022-09-13T07:39:51.885 DEBUG:boto:Token: None 2022-09-13T07:39:51.886 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.886 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KWVvjgbNNP1VUDdyUknD/SFB1EA= 2022-09-13T07:39:51.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KWVvjgbNNP1VUDdyUknD/SFB1EA=', 'Content-Length': '0'} 2022-09-13T07:39:51.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc82c383a7dc82d5-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:51.891 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.891 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.892 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.893 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.893 DEBUG:boto:Method: GET 2022-09-13T07:39:51.894 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.894 DEBUG:boto:Data: 2022-09-13T07:39:51.895 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.895 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.896 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.896 DEBUG:boto:Params: {} 2022-09-13T07:39:51.897 DEBUG:boto:Token: None 2022-09-13T07:39:51.898 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.898 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jkN74vNoXPYGK6uWyOHgCu8Y5sk= 2022-09-13T07:39:51.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jkN74vNoXPYGK6uWyOHgCu8Y5sk=', 'Content-Length': '0'} 2022-09-13T07:39:51.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068ed1cac2a380196-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.944 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:51.945 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:51.945 DEBUG:boto:path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.946 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.947 DEBUG:boto:Method: GET 2022-09-13T07:39:51.948 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.948 DEBUG:boto:Data: 2022-09-13T07:39:51.949 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.949 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.950 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.950 DEBUG:boto:Params: {} 2022-09-13T07:39:51.951 DEBUG:boto:Token: None 2022-09-13T07:39:51.951 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/obj?versionId=pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4 2022-09-13T07:39:51.952 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jkN74vNoXPYGK6uWyOHgCu8Y5sk= 2022-09-13T07:39:51.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jkN74vNoXPYGK6uWyOHgCu8Y5sk=', 'Content-Length': '0'} 2022-09-13T07:39:51.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'pQEwpIzw7c8BEjD3dpaFOBxw.JBKZU4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006cbcc340867c29ec-00632033c7-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.957 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:51.958 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:51.959 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.960 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.961 DEBUG:boto:Method: GET 2022-09-13T07:39:51.962 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.963 DEBUG:boto:Data: 2022-09-13T07:39:51.964 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.965 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:51.966 DEBUG:boto:Port: 8001 2022-09-13T07:39:51.967 DEBUG:boto:Params: {} 2022-09-13T07:39:51.968 DEBUG:boto:Token: None 2022-09-13T07:39:51.970 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/ 2022-09-13T07:39:51.971 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk= 2022-09-13T07:39:51.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk=', 'Content-Length': '0'} 2022-09-13T07:39:51.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b65e6ee40a0d8228-00632033c7-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:51.983 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:51.984 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:51.985 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:51.986 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.987 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.989 DEBUG:boto:Method: GET 2022-09-13T07:39:51.990 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:51.991 DEBUG:boto:Data: 2022-09-13T07:39:51.993 DEBUG:boto:Headers: {} 2022-09-13T07:39:51.994 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:51.995 DEBUG:boto:Port: 8000 2022-09-13T07:39:51.996 DEBUG:boto:Params: {} 2022-09-13T07:39:51.998 DEBUG:boto:Token: None 2022-09-13T07:39:51.999 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:51 GMT /yxvvuh-19/ 2022-09-13T07:39:52.000 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk= 2022-09-13T07:39:52.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qq/CJ5dAg1Wi1ijZ7RrK0NrmGjk=', 'Content-Length': '0'} 2022-09-13T07:39:52.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000913a1624342132e9-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.006 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:52.008 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.009 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.010 DEBUG:boto:Method: HEAD 2022-09-13T07:39:52.010 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:52.012 DEBUG:boto:Data: 2022-09-13T07:39:52.013 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.013 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.014 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.015 DEBUG:boto:Params: {} 2022-09-13T07:39:52.016 DEBUG:boto:Token: None 2022-09-13T07:39:52.017 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj 2022-09-13T07:39:52.018 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA= 2022-09-13T07:39:52.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA=', 'Content-Length': '0'} 2022-09-13T07:39:52.024 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b69403250467a029-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.025 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.025 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.026 DEBUG:boto:Method: HEAD 2022-09-13T07:39:52.026 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:52.027 DEBUG:boto:Data: 2022-09-13T07:39:52.027 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.028 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.028 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.029 DEBUG:boto:Params: {} 2022-09-13T07:39:52.029 DEBUG:boto:Token: None 2022-09-13T07:39:52.030 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj 2022-09-13T07:39:52.031 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA= 2022-09-13T07:39:52.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA=', 'Content-Length': '0'} 2022-09-13T07:39:52.036 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8b3ced326d37268-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:52.039 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.039 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.040 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.040 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.041 DEBUG:boto:Method: GET 2022-09-13T07:39:52.041 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.042 DEBUG:boto:Data: 2022-09-13T07:39:52.042 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.043 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.043 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.044 DEBUG:boto:Params: {} 2022-09-13T07:39:52.045 DEBUG:boto:Token: None 2022-09-13T07:39:52.045 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.046 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss= 2022-09-13T07:39:52.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss=', 'Content-Length': '0'} 2022-09-13T07:39:52.049 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1b7d623f1bc1c7b-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.093 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.094 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.094 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.095 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.096 DEBUG:boto:Method: GET 2022-09-13T07:39:52.096 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.097 DEBUG:boto:Data: 2022-09-13T07:39:52.097 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.102 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.102 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.103 DEBUG:boto:Params: {} 2022-09-13T07:39:52.103 DEBUG:boto:Token: None 2022-09-13T07:39:52.104 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.105 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss= 2022-09-13T07:39:52.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss=', 'Content-Length': '0'} 2022-09-13T07:39:52.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069e48b2d0f49f6de-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.109 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:52.109 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:52.110 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.110 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.111 DEBUG:boto:Method: GET 2022-09-13T07:39:52.111 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.112 DEBUG:boto:Data: 2022-09-13T07:39:52.112 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.113 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.113 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.114 DEBUG:boto:Params: {} 2022-09-13T07:39:52.114 DEBUG:boto:Token: None 2022-09-13T07:39:52.115 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/ 2022-09-13T07:39:52.116 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA= 2022-09-13T07:39:52.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA=', 'Content-Length': '0'} 2022-09-13T07:39:52.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d291c7d1777eb58-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.126 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:52.127 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:52.128 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:52.129 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.129 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.130 DEBUG:boto:Method: GET 2022-09-13T07:39:52.131 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.132 DEBUG:boto:Data: 2022-09-13T07:39:52.133 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.134 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.135 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.136 DEBUG:boto:Params: {} 2022-09-13T07:39:52.137 DEBUG:boto:Token: None 2022-09-13T07:39:52.137 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/ 2022-09-13T07:39:52.138 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA= 2022-09-13T07:39:52.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA=', 'Content-Length': '0'} 2022-09-13T07:39:52.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086358afcb19bb7b1-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.144 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:52.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:52.146 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.147 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.147 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.148 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.149 DEBUG:boto:Method: GET 2022-09-13T07:39:52.149 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.150 DEBUG:boto:Data: 2022-09-13T07:39:52.150 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.151 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.151 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.152 DEBUG:boto:Params: {} 2022-09-13T07:39:52.152 DEBUG:boto:Token: None 2022-09-13T07:39:52.153 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.154 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0kxrDJWIxAHCpuh+3k9If893T64= 2022-09-13T07:39:52.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0kxrDJWIxAHCpuh+3k9If893T64=', 'Content-Length': '0'} 2022-09-13T07:39:52.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce48aceb5181e2a8-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.158 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.158 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.159 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.160 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.161 DEBUG:boto:Method: GET 2022-09-13T07:39:52.162 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.163 DEBUG:boto:Data: 2022-09-13T07:39:52.164 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.164 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.165 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.166 DEBUG:boto:Params: {} 2022-09-13T07:39:52.167 DEBUG:boto:Token: None 2022-09-13T07:39:52.168 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.168 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0kxrDJWIxAHCpuh+3k9If893T64= 2022-09-13T07:39:52.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0kxrDJWIxAHCpuh+3k9If893T64=', 'Content-Length': '0'} 2022-09-13T07:39:52.171 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070a2320a8e7f924f-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.172 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.173 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.173 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.174 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.174 DEBUG:boto:Method: HEAD 2022-09-13T07:39:52.175 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.175 DEBUG:boto:Data: 2022-09-13T07:39:52.176 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.177 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.177 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.178 DEBUG:boto:Params: {} 2022-09-13T07:39:52.178 DEBUG:boto:Token: None 2022-09-13T07:39:52.179 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.180 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AWL/Sq9+k+b50mZZ+QoUAtBCIms= 2022-09-13T07:39:52.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AWL/Sq9+k+b50mZZ+QoUAtBCIms=', 'Content-Length': '0'} 2022-09-13T07:39:52.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002eeba77a26d28ab0-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.185 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.186 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.187 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.188 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.189 DEBUG:boto:Method: HEAD 2022-09-13T07:39:52.189 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.190 DEBUG:boto:Data: 2022-09-13T07:39:52.191 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.191 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.192 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.193 DEBUG:boto:Params: {} 2022-09-13T07:39:52.194 DEBUG:boto:Token: None 2022-09-13T07:39:52.195 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.201 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AWL/Sq9+k+b50mZZ+QoUAtBCIms= 2022-09-13T07:39:52.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AWL/Sq9+k+b50mZZ+QoUAtBCIms=', 'Content-Length': '0'} 2022-09-13T07:39:52.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000129dc9549fa2c4ab-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:52.206 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.206 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.207 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.207 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.208 DEBUG:boto:Method: GET 2022-09-13T07:39:52.208 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.209 DEBUG:boto:Data: 2022-09-13T07:39:52.209 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.210 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.211 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.211 DEBUG:boto:Params: {} 2022-09-13T07:39:52.212 DEBUG:boto:Token: None 2022-09-13T07:39:52.212 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.213 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0kxrDJWIxAHCpuh+3k9If893T64= 2022-09-13T07:39:52.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0kxrDJWIxAHCpuh+3k9If893T64=', 'Content-Length': '0'} 2022-09-13T07:39:52.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080d4cf93a3becad3-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.217 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.217 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.218 DEBUG:boto:path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.218 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.219 DEBUG:boto:Method: GET 2022-09-13T07:39:52.219 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.220 DEBUG:boto:Data: 2022-09-13T07:39:52.220 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.221 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.221 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.223 DEBUG:boto:Params: {} 2022-09-13T07:39:52.223 DEBUG:boto:Token: None 2022-09-13T07:39:52.224 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=xshrAtab2xKYxtLaiLBmYWJxRHGcH2r 2022-09-13T07:39:52.225 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0kxrDJWIxAHCpuh+3k9If893T64= 2022-09-13T07:39:52.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0kxrDJWIxAHCpuh+3k9If893T64=', 'Content-Length': '0'} 2022-09-13T07:39:52.227 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'xshrAtab2xKYxtLaiLBmYWJxRHGcH2r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066fb7245f02a8c1c-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.229 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:52.229 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:52.230 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.231 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.231 DEBUG:boto:Method: GET 2022-09-13T07:39:52.232 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.232 DEBUG:boto:Data: 2022-09-13T07:39:52.233 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.233 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.234 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.234 DEBUG:boto:Params: {} 2022-09-13T07:39:52.235 DEBUG:boto:Token: None 2022-09-13T07:39:52.235 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/ 2022-09-13T07:39:52.236 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA= 2022-09-13T07:39:52.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA=', 'Content-Length': '0'} 2022-09-13T07:39:52.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006886604efa21f74b-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.241 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:52.242 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:52.243 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:52.244 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.245 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.245 DEBUG:boto:Method: GET 2022-09-13T07:39:52.246 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.247 DEBUG:boto:Data: 2022-09-13T07:39:52.248 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.249 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.250 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.251 DEBUG:boto:Params: {} 2022-09-13T07:39:52.251 DEBUG:boto:Token: None 2022-09-13T07:39:52.252 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/ 2022-09-13T07:39:52.253 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA= 2022-09-13T07:39:52.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA=', 'Content-Length': '0'} 2022-09-13T07:39:52.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff09529efdfd16b2-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.259 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:52.260 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.261 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.262 DEBUG:boto:Method: HEAD 2022-09-13T07:39:52.263 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:52.264 DEBUG:boto:Data: 2022-09-13T07:39:52.265 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.265 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.266 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.267 DEBUG:boto:Params: {} 2022-09-13T07:39:52.268 DEBUG:boto:Token: None 2022-09-13T07:39:52.269 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj 2022-09-13T07:39:52.270 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA= 2022-09-13T07:39:52.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA=', 'Content-Length': '0'} 2022-09-13T07:39:52.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033719da682847db1-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.275 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.275 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.276 DEBUG:boto:Method: HEAD 2022-09-13T07:39:52.276 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:52.277 DEBUG:boto:Data: 2022-09-13T07:39:52.278 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.278 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.279 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.280 DEBUG:boto:Params: {} 2022-09-13T07:39:52.281 DEBUG:boto:Token: None 2022-09-13T07:39:52.281 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj 2022-09-13T07:39:52.282 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA= 2022-09-13T07:39:52.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA=', 'Content-Length': '0'} 2022-09-13T07:39:52.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd52152f3a943d9d-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:52.288 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.288 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.289 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.290 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.290 DEBUG:boto:Method: GET 2022-09-13T07:39:52.291 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.291 DEBUG:boto:Data: 2022-09-13T07:39:52.292 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.292 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.293 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.293 DEBUG:boto:Params: {} 2022-09-13T07:39:52.294 DEBUG:boto:Token: None 2022-09-13T07:39:52.295 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.295 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss= 2022-09-13T07:39:52.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss=', 'Content-Length': '0'} 2022-09-13T07:39:52.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000405d7cac71d35655-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.342 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.343 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.343 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.344 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.345 DEBUG:boto:Method: GET 2022-09-13T07:39:52.345 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.346 DEBUG:boto:Data: 2022-09-13T07:39:52.346 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.347 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.347 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.348 DEBUG:boto:Params: {} 2022-09-13T07:39:52.349 DEBUG:boto:Token: None 2022-09-13T07:39:52.349 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.350 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss= 2022-09-13T07:39:52.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss=', 'Content-Length': '0'} 2022-09-13T07:39:52.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fab29027d77ed6b-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.354 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:52.354 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:52.355 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.356 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.356 DEBUG:boto:Method: GET 2022-09-13T07:39:52.357 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.358 DEBUG:boto:Data: 2022-09-13T07:39:52.358 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.359 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.359 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.360 DEBUG:boto:Params: {} 2022-09-13T07:39:52.360 DEBUG:boto:Token: None 2022-09-13T07:39:52.361 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/ 2022-09-13T07:39:52.361 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA= 2022-09-13T07:39:52.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA=', 'Content-Length': '0'} 2022-09-13T07:39:52.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f7184180d6ad59d-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.366 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:52.367 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:52.368 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:52.368 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.369 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.369 DEBUG:boto:Method: GET 2022-09-13T07:39:52.370 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.371 DEBUG:boto:Data: 2022-09-13T07:39:52.371 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.372 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.372 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.373 DEBUG:boto:Params: {} 2022-09-13T07:39:52.374 DEBUG:boto:Token: None 2022-09-13T07:39:52.375 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/ 2022-09-13T07:39:52.376 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA= 2022-09-13T07:39:52.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA=', 'Content-Length': '0'} 2022-09-13T07:39:52.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2576850a7a91312-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.380 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:52.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:52.382 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.382 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.383 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.384 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.384 DEBUG:boto:Method: GET 2022-09-13T07:39:52.385 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.385 DEBUG:boto:Data: 2022-09-13T07:39:52.386 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.387 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.387 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.388 DEBUG:boto:Params: {} 2022-09-13T07:39:52.388 DEBUG:boto:Token: None 2022-09-13T07:39:52.389 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.390 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Onz8hsadKuXjYq3nLQ3JcVHqPSY= 2022-09-13T07:39:52.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Onz8hsadKuXjYq3nLQ3JcVHqPSY=', 'Content-Length': '0'} 2022-09-13T07:39:52.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000777d17638d227011-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.396 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.397 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.398 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.399 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.400 DEBUG:boto:Method: GET 2022-09-13T07:39:52.401 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.402 DEBUG:boto:Data: 2022-09-13T07:39:52.403 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.404 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.404 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.405 DEBUG:boto:Params: {} 2022-09-13T07:39:52.405 DEBUG:boto:Token: None 2022-09-13T07:39:52.406 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.407 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Onz8hsadKuXjYq3nLQ3JcVHqPSY= 2022-09-13T07:39:52.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Onz8hsadKuXjYq3nLQ3JcVHqPSY=', 'Content-Length': '0'} 2022-09-13T07:39:52.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c01bc58abe204f13-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.450 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.451 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.451 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.452 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.452 DEBUG:boto:Method: HEAD 2022-09-13T07:39:52.453 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.453 DEBUG:boto:Data: 2022-09-13T07:39:52.454 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.454 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.455 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.455 DEBUG:boto:Params: {} 2022-09-13T07:39:52.456 DEBUG:boto:Token: None 2022-09-13T07:39:52.456 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.457 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:as78VhhXPVA97cj6AFIR1LaRk3M= 2022-09-13T07:39:52.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:as78VhhXPVA97cj6AFIR1LaRk3M=', 'Content-Length': '0'} 2022-09-13T07:39:52.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000326b58b4816cb8b0-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.462 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.463 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.464 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.465 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.466 DEBUG:boto:Method: HEAD 2022-09-13T07:39:52.467 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.467 DEBUG:boto:Data: 2022-09-13T07:39:52.468 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.469 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.469 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.470 DEBUG:boto:Params: {} 2022-09-13T07:39:52.471 DEBUG:boto:Token: None 2022-09-13T07:39:52.472 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.473 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:as78VhhXPVA97cj6AFIR1LaRk3M= 2022-09-13T07:39:52.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:as78VhhXPVA97cj6AFIR1LaRk3M=', 'Content-Length': '0'} 2022-09-13T07:39:52.478 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e00e05c5f10f322-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:52.481 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.482 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.483 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.484 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.485 DEBUG:boto:Method: GET 2022-09-13T07:39:52.486 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.487 DEBUG:boto:Data: 2022-09-13T07:39:52.488 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.489 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.490 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.491 DEBUG:boto:Params: {} 2022-09-13T07:39:52.492 DEBUG:boto:Token: None 2022-09-13T07:39:52.493 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.494 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Onz8hsadKuXjYq3nLQ3JcVHqPSY= 2022-09-13T07:39:52.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Onz8hsadKuXjYq3nLQ3JcVHqPSY=', 'Content-Length': '0'} 2022-09-13T07:39:52.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069252a7e697eb829-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.500 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.501 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.502 DEBUG:boto:path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.502 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.503 DEBUG:boto:Method: GET 2022-09-13T07:39:52.503 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.504 DEBUG:boto:Data: 2022-09-13T07:39:52.504 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.505 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.505 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.506 DEBUG:boto:Params: {} 2022-09-13T07:39:52.506 DEBUG:boto:Token: None 2022-09-13T07:39:52.507 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I 2022-09-13T07:39:52.508 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Onz8hsadKuXjYq3nLQ3JcVHqPSY= 2022-09-13T07:39:52.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Onz8hsadKuXjYq3nLQ3JcVHqPSY=', 'Content-Length': '0'} 2022-09-13T07:39:52.511 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', '.0dD93mWeKV1KNZhcc5D9Bpp-Nhnh.I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072f93d024645a866-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.512 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:52.512 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:52.513 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.513 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.514 DEBUG:boto:Method: GET 2022-09-13T07:39:52.515 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.515 DEBUG:boto:Data: 2022-09-13T07:39:52.516 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.516 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.517 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.518 DEBUG:boto:Params: {} 2022-09-13T07:39:52.518 DEBUG:boto:Token: None 2022-09-13T07:39:52.519 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/ 2022-09-13T07:39:52.520 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA= 2022-09-13T07:39:52.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA=', 'Content-Length': '0'} 2022-09-13T07:39:52.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000150024352491ed82-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.525 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:52.526 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:52.527 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:52.527 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.528 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.528 DEBUG:boto:Method: GET 2022-09-13T07:39:52.529 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.530 DEBUG:boto:Data: 2022-09-13T07:39:52.530 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.531 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.531 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.532 DEBUG:boto:Params: {} 2022-09-13T07:39:52.533 DEBUG:boto:Token: None 2022-09-13T07:39:52.533 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/ 2022-09-13T07:39:52.534 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA= 2022-09-13T07:39:52.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA=', 'Content-Length': '0'} 2022-09-13T07:39:52.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c83cde7407710769-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.539 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:52.540 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.541 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.542 DEBUG:boto:Method: HEAD 2022-09-13T07:39:52.542 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:52.543 DEBUG:boto:Data: 2022-09-13T07:39:52.543 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.544 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.545 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.546 DEBUG:boto:Params: {} 2022-09-13T07:39:52.546 DEBUG:boto:Token: None 2022-09-13T07:39:52.547 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj 2022-09-13T07:39:52.548 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA= 2022-09-13T07:39:52.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA=', 'Content-Length': '0'} 2022-09-13T07:39:52.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034bda20948bc6ca4-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.553 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.554 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.555 DEBUG:boto:Method: HEAD 2022-09-13T07:39:52.556 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:52.557 DEBUG:boto:Data: 2022-09-13T07:39:52.557 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.558 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.559 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.560 DEBUG:boto:Params: {} 2022-09-13T07:39:52.561 DEBUG:boto:Token: None 2022-09-13T07:39:52.562 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj 2022-09-13T07:39:52.563 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA= 2022-09-13T07:39:52.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA=', 'Content-Length': '0'} 2022-09-13T07:39:52.568 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d157505b315f3285-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:52.570 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.570 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.572 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.573 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.573 DEBUG:boto:Method: GET 2022-09-13T07:39:52.574 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.575 DEBUG:boto:Data: 2022-09-13T07:39:52.576 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.577 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.578 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.579 DEBUG:boto:Params: {} 2022-09-13T07:39:52.580 DEBUG:boto:Token: None 2022-09-13T07:39:52.581 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.582 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss= 2022-09-13T07:39:52.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss=', 'Content-Length': '0'} 2022-09-13T07:39:52.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ecc1b1ed8e32263-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.626 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.627 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.628 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.629 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.630 DEBUG:boto:Method: GET 2022-09-13T07:39:52.631 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.632 DEBUG:boto:Data: 2022-09-13T07:39:52.633 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.634 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.635 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.636 DEBUG:boto:Params: {} 2022-09-13T07:39:52.637 DEBUG:boto:Token: None 2022-09-13T07:39:52.638 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.639 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss= 2022-09-13T07:39:52.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss=', 'Content-Length': '0'} 2022-09-13T07:39:52.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f660b57ab4f6c6c-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.645 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:52.645 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:52.646 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.647 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.647 DEBUG:boto:Method: GET 2022-09-13T07:39:52.648 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.648 DEBUG:boto:Data: 2022-09-13T07:39:52.649 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.649 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.650 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.651 DEBUG:boto:Params: {} 2022-09-13T07:39:52.651 DEBUG:boto:Token: None 2022-09-13T07:39:52.652 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/ 2022-09-13T07:39:52.652 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA= 2022-09-13T07:39:52.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA=', 'Content-Length': '0'} 2022-09-13T07:39:52.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b8fece67dd4bba3-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.659 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:52.661 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:52.662 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:52.663 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.664 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.665 DEBUG:boto:Method: GET 2022-09-13T07:39:52.665 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.666 DEBUG:boto:Data: 2022-09-13T07:39:52.667 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.668 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.669 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.670 DEBUG:boto:Params: {} 2022-09-13T07:39:52.671 DEBUG:boto:Token: None 2022-09-13T07:39:52.672 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/ 2022-09-13T07:39:52.673 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA= 2022-09-13T07:39:52.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA=', 'Content-Length': '0'} 2022-09-13T07:39:52.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f9997fa1b2c3482-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.679 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:52.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:52.681 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.682 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.683 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.684 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.685 DEBUG:boto:Method: GET 2022-09-13T07:39:52.686 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.687 DEBUG:boto:Data: 2022-09-13T07:39:52.687 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.688 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.689 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.690 DEBUG:boto:Params: {} 2022-09-13T07:39:52.691 DEBUG:boto:Token: None 2022-09-13T07:39:52.692 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.692 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KoYipolnt8dF3OpdMPz1iw35+dU= 2022-09-13T07:39:52.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KoYipolnt8dF3OpdMPz1iw35+dU=', 'Content-Length': '0'} 2022-09-13T07:39:52.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000502712c829e91e9f-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.696 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.696 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.697 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.697 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.698 DEBUG:boto:Method: GET 2022-09-13T07:39:52.698 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.699 DEBUG:boto:Data: 2022-09-13T07:39:52.699 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.700 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.700 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.701 DEBUG:boto:Params: {} 2022-09-13T07:39:52.701 DEBUG:boto:Token: None 2022-09-13T07:39:52.702 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.703 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KoYipolnt8dF3OpdMPz1iw35+dU= 2022-09-13T07:39:52.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KoYipolnt8dF3OpdMPz1iw35+dU=', 'Content-Length': '0'} 2022-09-13T07:39:52.706 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000706dc1e6a9f4bbb1-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.707 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.707 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.708 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.708 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.709 DEBUG:boto:Method: HEAD 2022-09-13T07:39:52.709 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.710 DEBUG:boto:Data: 2022-09-13T07:39:52.710 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.711 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.711 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.712 DEBUG:boto:Params: {} 2022-09-13T07:39:52.712 DEBUG:boto:Token: None 2022-09-13T07:39:52.713 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.713 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NcrV0akJz9GbYnSi2AlOxj+9SGU= 2022-09-13T07:39:52.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NcrV0akJz9GbYnSi2AlOxj+9SGU=', 'Content-Length': '0'} 2022-09-13T07:39:52.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfec66e292ff0abc-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.717 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.718 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.718 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.719 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.719 DEBUG:boto:Method: HEAD 2022-09-13T07:39:52.720 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.720 DEBUG:boto:Data: 2022-09-13T07:39:52.721 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.721 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.722 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.722 DEBUG:boto:Params: {} 2022-09-13T07:39:52.723 DEBUG:boto:Token: None 2022-09-13T07:39:52.723 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.724 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NcrV0akJz9GbYnSi2AlOxj+9SGU= 2022-09-13T07:39:52.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NcrV0akJz9GbYnSi2AlOxj+9SGU=', 'Content-Length': '0'} 2022-09-13T07:39:52.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000414cb6f2a1cc432c-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:52.728 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.728 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.729 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.730 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.730 DEBUG:boto:Method: GET 2022-09-13T07:39:52.731 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.731 DEBUG:boto:Data: 2022-09-13T07:39:52.732 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.732 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.733 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.734 DEBUG:boto:Params: {} 2022-09-13T07:39:52.734 DEBUG:boto:Token: None 2022-09-13T07:39:52.735 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.735 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KoYipolnt8dF3OpdMPz1iw35+dU= 2022-09-13T07:39:52.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KoYipolnt8dF3OpdMPz1iw35+dU=', 'Content-Length': '0'} 2022-09-13T07:39:52.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065f7479d8321fcdc-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.739 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.739 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.740 DEBUG:boto:path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.740 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.741 DEBUG:boto:Method: GET 2022-09-13T07:39:52.742 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.742 DEBUG:boto:Data: 2022-09-13T07:39:52.743 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.743 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.744 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.744 DEBUG:boto:Params: {} 2022-09-13T07:39:52.745 DEBUG:boto:Token: None 2022-09-13T07:39:52.745 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=C-XByO2K9qUOxsnzB35XnbS7CZIHKxH 2022-09-13T07:39:52.746 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:KoYipolnt8dF3OpdMPz1iw35+dU= 2022-09-13T07:39:52.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:KoYipolnt8dF3OpdMPz1iw35+dU=', 'Content-Length': '0'} 2022-09-13T07:39:52.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'C-XByO2K9qUOxsnzB35XnbS7CZIHKxH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009024fd3418452c6d-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.750 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:52.751 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:52.751 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.752 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.752 DEBUG:boto:Method: GET 2022-09-13T07:39:52.753 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.753 DEBUG:boto:Data: 2022-09-13T07:39:52.754 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.754 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.755 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.755 DEBUG:boto:Params: {} 2022-09-13T07:39:52.756 DEBUG:boto:Token: None 2022-09-13T07:39:52.757 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/ 2022-09-13T07:39:52.757 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA= 2022-09-13T07:39:52.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA=', 'Content-Length': '0'} 2022-09-13T07:39:52.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000063709d275efd65f-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.802 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:52.804 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:52.805 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:52.806 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.807 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.808 DEBUG:boto:Method: GET 2022-09-13T07:39:52.809 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.810 DEBUG:boto:Data: 2022-09-13T07:39:52.810 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.811 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.812 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.813 DEBUG:boto:Params: {} 2022-09-13T07:39:52.814 DEBUG:boto:Token: None 2022-09-13T07:39:52.814 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/ 2022-09-13T07:39:52.815 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA= 2022-09-13T07:39:52.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA=', 'Content-Length': '0'} 2022-09-13T07:39:52.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6d25279d902bbec-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.819 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:52.820 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.821 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.821 DEBUG:boto:Method: HEAD 2022-09-13T07:39:52.822 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:52.822 DEBUG:boto:Data: 2022-09-13T07:39:52.823 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.823 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.824 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.824 DEBUG:boto:Params: {} 2022-09-13T07:39:52.825 DEBUG:boto:Token: None 2022-09-13T07:39:52.826 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj 2022-09-13T07:39:52.826 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA= 2022-09-13T07:39:52.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA=', 'Content-Length': '0'} 2022-09-13T07:39:52.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e924396eedf98b8-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.831 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.832 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.833 DEBUG:boto:Method: HEAD 2022-09-13T07:39:52.833 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:52.834 DEBUG:boto:Data: 2022-09-13T07:39:52.834 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.835 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.835 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.836 DEBUG:boto:Params: {} 2022-09-13T07:39:52.836 DEBUG:boto:Token: None 2022-09-13T07:39:52.837 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj 2022-09-13T07:39:52.838 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA= 2022-09-13T07:39:52.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:O2o5xC9f9Ecpc6HDPeZPk6erVmA=', 'Content-Length': '0'} 2022-09-13T07:39:52.842 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ba13229f94562b9-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:52.843 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.844 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.845 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.845 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.846 DEBUG:boto:Method: GET 2022-09-13T07:39:52.846 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.847 DEBUG:boto:Data: 2022-09-13T07:39:52.847 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.848 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.849 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.849 DEBUG:boto:Params: {} 2022-09-13T07:39:52.850 DEBUG:boto:Token: None 2022-09-13T07:39:52.851 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.851 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss= 2022-09-13T07:39:52.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss=', 'Content-Length': '0'} 2022-09-13T07:39:52.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca85715ad39773e1-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.855 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.855 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.856 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.856 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.857 DEBUG:boto:Method: GET 2022-09-13T07:39:52.857 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.858 DEBUG:boto:Data: 2022-09-13T07:39:52.859 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.859 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.860 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.860 DEBUG:boto:Params: {} 2022-09-13T07:39:52.861 DEBUG:boto:Token: None 2022-09-13T07:39:52.861 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:52.862 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss= 2022-09-13T07:39:52.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:I1tnd/50uY7ibkxRv+80neuV+ss=', 'Content-Length': '0'} 2022-09-13T07:39:52.866 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b37fba5d3f4c7cd6-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.910 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:52.911 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:52.911 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.912 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.912 DEBUG:boto:Method: GET 2022-09-13T07:39:52.913 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.913 DEBUG:boto:Data: 2022-09-13T07:39:52.914 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.915 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.916 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.917 DEBUG:boto:Params: {} 2022-09-13T07:39:52.918 DEBUG:boto:Token: None 2022-09-13T07:39:52.919 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/ 2022-09-13T07:39:52.920 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA= 2022-09-13T07:39:52.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA=', 'Content-Length': '0'} 2022-09-13T07:39:52.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003197300033449629-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.924 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:52.925 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:52.926 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:52.926 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.927 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.927 DEBUG:boto:Method: GET 2022-09-13T07:39:52.928 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:52.928 DEBUG:boto:Data: 2022-09-13T07:39:52.929 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.930 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.930 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.931 DEBUG:boto:Params: {} 2022-09-13T07:39:52.931 DEBUG:boto:Token: None 2022-09-13T07:39:52.932 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/ 2022-09-13T07:39:52.932 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA= 2022-09-13T07:39:52.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MVGRtWpYYtlEkD612qlHqnaIOdA=', 'Content-Length': '0'} 2022-09-13T07:39:52.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038f664398c8322d8-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.936 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:52.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:52.938 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.938 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.939 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:52.939 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:52.940 DEBUG:boto:Method: GET 2022-09-13T07:39:52.940 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:52.941 DEBUG:boto:Data: 2022-09-13T07:39:52.941 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.942 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:52.942 DEBUG:boto:Port: 8001 2022-09-13T07:39:52.943 DEBUG:boto:Params: {} 2022-09-13T07:39:52.943 DEBUG:boto:Token: None 2022-09-13T07:39:52.944 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:52.944 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/2rEGWdKT3icuIBH2xNUGJtdZMY= 2022-09-13T07:39:52.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/2rEGWdKT3icuIBH2xNUGJtdZMY=', 'Content-Length': '0'} 2022-09-13T07:39:52.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd65ab5e4617b447-00632033c8-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:52.990 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:52.991 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:52.992 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:52.992 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:52.993 DEBUG:boto:Method: GET 2022-09-13T07:39:52.993 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:52.994 DEBUG:boto:Data: 2022-09-13T07:39:52.995 DEBUG:boto:Headers: {} 2022-09-13T07:39:52.995 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:52.996 DEBUG:boto:Port: 8000 2022-09-13T07:39:52.997 DEBUG:boto:Params: {} 2022-09-13T07:39:52.997 DEBUG:boto:Token: None 2022-09-13T07:39:52.998 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:52 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:52.999 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/2rEGWdKT3icuIBH2xNUGJtdZMY= 2022-09-13T07:39:53.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/2rEGWdKT3icuIBH2xNUGJtdZMY=', 'Content-Length': '0'} 2022-09-13T07:39:53.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006ba2596330ae7e8-00632033c8-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:53.003 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:53.003 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:53.004 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.004 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.005 DEBUG:boto:Method: HEAD 2022-09-13T07:39:53.006 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.006 DEBUG:boto:Data: 2022-09-13T07:39:53.007 DEBUG:boto:Headers: {} 2022-09-13T07:39:53.008 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:53.009 DEBUG:boto:Port: 8001 2022-09-13T07:39:53.010 DEBUG:boto:Params: {} 2022-09-13T07:39:53.011 DEBUG:boto:Token: None 2022-09-13T07:39:53.012 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:53 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.013 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z0m0d2BCJEGR87ITkzuZPi9XmBg= 2022-09-13T07:39:53.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z0m0d2BCJEGR87ITkzuZPi9XmBg=', 'Content-Length': '0'} 2022-09-13T07:39:53.017 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f951d0aeef97569-00632033c9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:53.018 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:53.019 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:53.019 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.019 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.020 DEBUG:boto:Method: HEAD 2022-09-13T07:39:53.020 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.021 DEBUG:boto:Data: 2022-09-13T07:39:53.021 DEBUG:boto:Headers: {} 2022-09-13T07:39:53.022 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:53.022 DEBUG:boto:Port: 8000 2022-09-13T07:39:53.023 DEBUG:boto:Params: {} 2022-09-13T07:39:53.023 DEBUG:boto:Token: None 2022-09-13T07:39:53.024 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:53 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.025 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:z0m0d2BCJEGR87ITkzuZPi9XmBg= 2022-09-13T07:39:53.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:z0m0d2BCJEGR87ITkzuZPi9XmBg=', 'Content-Length': '0'} 2022-09-13T07:39:53.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059ed78f4838b7703-00632033c9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:53.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:53.029 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:53.030 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:53.030 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.031 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.031 DEBUG:boto:Method: GET 2022-09-13T07:39:53.032 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.033 DEBUG:boto:Data: 2022-09-13T07:39:53.033 DEBUG:boto:Headers: {} 2022-09-13T07:39:53.034 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:53.043 DEBUG:boto:Port: 8001 2022-09-13T07:39:53.044 DEBUG:boto:Params: {} 2022-09-13T07:39:53.044 DEBUG:boto:Token: None 2022-09-13T07:39:53.045 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:53 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.046 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RRDtsUJkubDYDwMhUc5g+3+oCBc= 2022-09-13T07:39:53.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RRDtsUJkubDYDwMhUc5g+3+oCBc=', 'Content-Length': '0'} 2022-09-13T07:39:53.049 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f47b46b1c0e3fd8-00632033c9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:53.050 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:53.052 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:53.052 DEBUG:boto:path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.055 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.056 DEBUG:boto:Method: GET 2022-09-13T07:39:53.057 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.057 DEBUG:boto:Data: 2022-09-13T07:39:53.058 DEBUG:boto:Headers: {} 2022-09-13T07:39:53.059 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:53.059 DEBUG:boto:Port: 8000 2022-09-13T07:39:53.060 DEBUG:boto:Params: {} 2022-09-13T07:39:53.060 DEBUG:boto:Token: None 2022-09-13T07:39:53.061 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:53 GMT /yxvvuh-19/obj?versionId=aErb35kohBMBuUIVZDyjKh5q.cT.bAJ 2022-09-13T07:39:53.061 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RRDtsUJkubDYDwMhUc5g+3+oCBc= 2022-09-13T07:39:53.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RRDtsUJkubDYDwMhUc5g+3+oCBc=', 'Content-Length': '0'} 2022-09-13T07:39:53.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:29 GMT'), ('x-amz-version-id', 'aErb35kohBMBuUIVZDyjKh5q.cT.bAJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000367691fe6afc6be-00632033c9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:53.065 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:53.066 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:53.067 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:53.067 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:53.068 DEBUG:boto:Method: GET 2022-09-13T07:39:53.068 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:53.069 DEBUG:boto:Data: 2022-09-13T07:39:53.069 DEBUG:boto:Headers: {} 2022-09-13T07:39:53.070 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:53.070 DEBUG:boto:Port: 8001 2022-09-13T07:39:53.071 DEBUG:boto:Params: {} 2022-09-13T07:39:53.071 DEBUG:boto:Token: None 2022-09-13T07:39:53.072 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:53 GMT /yxvvuh-19/ 2022-09-13T07:39:53.073 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SU2hAk/5idklaVrc7ORp1xn5Cm4= 2022-09-13T07:39:53.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SU2hAk/5idklaVrc7ORp1xn5Cm4=', 'Content-Length': '0'} 2022-09-13T07:39:53.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a76a5312cd4133c2-00632033c9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:53.077 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:53.079 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:53.080 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:53.081 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:53.081 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:53.082 DEBUG:boto:Method: GET 2022-09-13T07:39:53.083 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:53.084 DEBUG:boto:Data: 2022-09-13T07:39:53.085 DEBUG:boto:Headers: {} 2022-09-13T07:39:53.086 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:53.086 DEBUG:boto:Port: 8000 2022-09-13T07:39:53.087 DEBUG:boto:Params: {} 2022-09-13T07:39:53.088 DEBUG:boto:Token: None 2022-09-13T07:39:53.089 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:53 GMT /yxvvuh-19/ 2022-09-13T07:39:53.090 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SU2hAk/5idklaVrc7ORp1xn5Cm4= 2022-09-13T07:39:53.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SU2hAk/5idklaVrc7ORp1xn5Cm4=', 'Content-Length': '0'} 2022-09-13T07:39:53.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6344e33cf26b3a6-00632033c9-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:53.095 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:53.097 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:53.097 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:53.098 DEBUG:boto:Method: HEAD 2022-09-13T07:39:53.099 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:53.100 DEBUG:boto:Data: 2022-09-13T07:39:53.101 DEBUG:boto:Headers: {} 2022-09-13T07:39:53.101 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:53.102 DEBUG:boto:Port: 8001 2022-09-13T07:39:53.103 DEBUG:boto:Params: {} 2022-09-13T07:39:53.104 DEBUG:boto:Token: None 2022-09-13T07:39:53.105 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:53 GMT /yxvvuh-19/obj 2022-09-13T07:39:53.106 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uKEWHOldBBHEQr2WUeZy+BkTLCw= 2022-09-13T07:39:53.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uKEWHOldBBHEQr2WUeZy+BkTLCw=', 'Content-Length': '0'} 2022-09-13T07:39:53.110 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd8f3db0b7fc89db-00632033c9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:53.111 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:53.112 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:53.113 DEBUG:boto:Method: HEAD 2022-09-13T07:39:53.114 DEBUG:boto:Path: /yxvvuh-19/obj 2022-09-13T07:39:53.115 DEBUG:boto:Data: 2022-09-13T07:39:53.115 DEBUG:boto:Headers: {} 2022-09-13T07:39:53.116 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:53.117 DEBUG:boto:Port: 8000 2022-09-13T07:39:53.118 DEBUG:boto:Params: {} 2022-09-13T07:39:53.119 DEBUG:boto:Token: None 2022-09-13T07:39:53.119 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:39:53 GMT /yxvvuh-19/obj 2022-09-13T07:39:53.120 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uKEWHOldBBHEQr2WUeZy+BkTLCw= 2022-09-13T07:39:53.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uKEWHOldBBHEQr2WUeZy+BkTLCw=', 'Content-Length': '0'} 2022-09-13T07:39:53.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000351812592a59927b-00632033c9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:53.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:39:53.127 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:53.128 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:53.128 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:53.129 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:53.130 DEBUG:boto:Method: GET 2022-09-13T07:39:53.131 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:53.131 DEBUG:boto:Data: 2022-09-13T07:39:53.132 DEBUG:boto:Headers: {} 2022-09-13T07:39:53.132 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:53.133 DEBUG:boto:Port: 8001 2022-09-13T07:39:53.133 DEBUG:boto:Params: {} 2022-09-13T07:39:53.134 DEBUG:boto:Token: None 2022-09-13T07:39:53.134 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:53 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:53.135 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/XgOg6npPPbTcG2lH+aqbWSaluE= 2022-09-13T07:39:53.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/XgOg6npPPbTcG2lH+aqbWSaluE=', 'Content-Length': '0'} 2022-09-13T07:39:53.138 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000392d8cc8d6364b0c-00632033c9-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:53.139 DEBUG:boto:path=/yxvvuh-19/obj 2022-09-13T07:39:53.139 DEBUG:boto:auth_path=/yxvvuh-19/obj 2022-09-13T07:39:53.140 DEBUG:boto:path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:53.140 DEBUG:boto:auth_path=/yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:53.141 DEBUG:boto:Method: GET 2022-09-13T07:39:53.141 DEBUG:boto:Path: /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:53.142 DEBUG:boto:Data: 2022-09-13T07:39:53.142 DEBUG:boto:Headers: {} 2022-09-13T07:39:53.143 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:53.143 DEBUG:boto:Port: 8000 2022-09-13T07:39:53.144 DEBUG:boto:Params: {} 2022-09-13T07:39:53.145 DEBUG:boto:Token: None 2022-09-13T07:39:53.146 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:53 GMT /yxvvuh-19/obj?versionId=En7MftyiCP0pYMPcqn5sySJi4UkpaiZ 2022-09-13T07:39:53.146 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/XgOg6npPPbTcG2lH+aqbWSaluE= 2022-09-13T07:39:53.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/XgOg6npPPbTcG2lH+aqbWSaluE=', 'Content-Length': '0'} 2022-09-13T07:39:53.149 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:37:30 GMT'), ('x-amz-version-id', 'En7MftyiCP0pYMPcqn5sySJi4UkpaiZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df3beeb6b27e1b5e-00632033c9-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:39:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:53.190 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:53.191 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:53.191 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:53.192 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:53.193 DEBUG:boto:Method: GET 2022-09-13T07:39:53.193 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:53.194 DEBUG:boto:Data: 2022-09-13T07:39:53.195 DEBUG:boto:Headers: {} 2022-09-13T07:39:53.195 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:39:53.196 DEBUG:boto:Port: 8001 2022-09-13T07:39:53.196 DEBUG:boto:Params: {} 2022-09-13T07:39:53.197 DEBUG:boto:Token: None 2022-09-13T07:39:53.197 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:53 GMT /yxvvuh-19/ 2022-09-13T07:39:53.198 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SU2hAk/5idklaVrc7ORp1xn5Cm4= 2022-09-13T07:39:53.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SU2hAk/5idklaVrc7ORp1xn5Cm4=', 'Content-Length': '0'} 2022-09-13T07:39:53.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002503d876aae1df04-00632033c9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:53.202 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:53.203 DEBUG:boto:path=/yxvvuh-19/ 2022-09-13T07:39:53.204 DEBUG:boto:auth_path=/yxvvuh-19/ 2022-09-13T07:39:53.204 DEBUG:boto:path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:53.205 DEBUG:boto:auth_path=/yxvvuh-19/?prefix=obj 2022-09-13T07:39:53.205 DEBUG:boto:Method: GET 2022-09-13T07:39:53.206 DEBUG:boto:Path: /yxvvuh-19/?prefix=obj 2022-09-13T07:39:53.206 DEBUG:boto:Data: 2022-09-13T07:39:53.207 DEBUG:boto:Headers: {} 2022-09-13T07:39:53.207 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:39:53.208 DEBUG:boto:Port: 8000 2022-09-13T07:39:53.208 DEBUG:boto:Params: {} 2022-09-13T07:39:53.209 DEBUG:boto:Token: None 2022-09-13T07:39:53.210 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:39:53 GMT /yxvvuh-19/ 2022-09-13T07:39:53.210 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SU2hAk/5idklaVrc7ORp1xn5Cm4= 2022-09-13T07:39:53.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SU2hAk/5idklaVrc7ORp1xn5Cm4=', 'Content-Length': '0'} 2022-09-13T07:39:53.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d90215b2cbd920d-00632033c9-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:39:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:53.215 DEBUG:boto:b'yxvvuh-19obj1000falseobj2022-09-13T07:37:30.651Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:39:53.216 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-19 zones={test-zone3, test-zone2} 2022-09-13T07:39:53.216 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2022-09-13T07:39:53.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:53.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:53.218 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:39:53.275 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:53.276 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:53.327 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:39:53.327 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:53.327 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000003050", 2022-09-13T07:39:53.328 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:35:46.147466Z" 2022-09-13T07:39:53.328 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:53.329 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:53.329 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000002063", 2022-09-13T07:39:53.329 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:39:41.890430Z" 2022-09-13T07:39:53.329 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:53.330 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:53.330 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000006575", 2022-09-13T07:39:53.330 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:37:29.880837Z" 2022-09-13T07:39:53.331 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:53.331 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:53.331 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:39:53.331 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:39:53.332 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:53.332 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:39:53.333 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003050', 1: '00000000000000000000:00000000000000002063', 2: '00000000000000000000:00000000000000006575', 3: ''} 2022-09-13T07:39:53.334 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-13T07:39:53.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:53.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:53.336 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:39:53.394 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:53.394 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:53.450 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:39:53.450 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:39:53.451 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:39:53.451 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:39:53.451 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:39:53.452 INFO:teuthology.orchestra.run.smithi083.stdout: "instance_id": 12687285500305080346 2022-09-13T07:39:53.452 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:53.452 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:39:53.453 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:53.453 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:39:53.454 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:39:53.454 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "incremental-sync", 2022-09-13T07:39:53.455 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "00000000000000000000:00000000000000003050", 2022-09-13T07:39:53.455 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:39:53.456 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:39:53.456 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:39:53.457 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:35:46.147466Z" 2022-09-13T07:39:53.457 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:53.458 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:53.458 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:53.459 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:39:53.459 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:39:53.460 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "incremental-sync", 2022-09-13T07:39:53.460 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "00000000000000000000:00000000000000002063", 2022-09-13T07:39:53.461 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:39:53.461 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:39:53.462 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:39:53.462 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:39:41.890430Z" 2022-09-13T07:39:53.463 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:53.463 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:53.464 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:53.464 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:39:53.465 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:39:53.465 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "incremental-sync", 2022-09-13T07:39:53.466 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "00000000000000000000:00000000000000006575", 2022-09-13T07:39:53.466 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:39:53.467 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:39:53.467 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:39:53.468 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:37:29.880837Z" 2022-09-13T07:39:53.468 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:53.469 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:53.469 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:53.470 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:39:53.470 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:39:53.471 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "incremental-sync", 2022-09-13T07:39:53.471 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:39:53.472 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:39:53.472 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:39:53.472 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:39:53.473 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000" 2022-09-13T07:39:53.473 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:53.474 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:53.474 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:39:53.475 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:53.475 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:39:53.476 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 0, 2022-09-13T07:39:53.476 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 0 2022-09-13T07:39:53.477 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:53.477 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:39:53Z" 2022-09-13T07:39:53.478 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:39:53.479 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12687285500305080346 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003050", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:35:46.147466Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:39:41.890430Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006575", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:37:29.880837Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-13T07:39:53Z" } 2022-09-13T07:39:53.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003050', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:35:46.147466Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:39:41.890430Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006575', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:37:29.880837Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-13T07:39:53.481 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003050', 1: '00000000000000000000:00000000000000002063', 2: '00000000000000000000:00000000000000006575', 3: ''} 2022-09-13T07:39:53.481 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003050', 1: '00000000000000000000:00000000000000002063', 2: '00000000000000000000:00000000000000006575', 3: ''} 2022-09-13T07:39:53.482 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-13T07:39:53.483 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2022-09-13T07:39:53.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:53.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:53.486 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:39:53.541 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:53.542 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:53.607 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:39:53.607 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:53.608 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000003050", 2022-09-13T07:39:53.608 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:35:46.147466Z" 2022-09-13T07:39:53.609 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:53.609 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:53.610 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000002063", 2022-09-13T07:39:53.610 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:39:41.890430Z" 2022-09-13T07:39:53.611 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:53.611 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:53.612 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000006575", 2022-09-13T07:39:53.613 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:37:29.880837Z" 2022-09-13T07:39:53.613 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:53.614 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:53.614 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:39:53.615 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:39:53.615 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:53.616 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:39:53.618 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003050', 1: '00000000000000000000:00000000000000002063', 2: '00000000000000000000:00000000000000006575', 3: ''} 2022-09-13T07:39:53.619 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-13T07:39:53.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:53.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:53.622 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:39:53.677 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:53.677 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:53.746 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:39:53.747 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:39:53.747 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:39:53.747 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:39:53.747 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:39:53.748 INFO:teuthology.orchestra.run.smithi036.stdout: "instance_id": 11637822015646413467 2022-09-13T07:39:53.748 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:53.748 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:39:53.748 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:53.749 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:39:53.749 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:53.749 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:53.750 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000003050", 2022-09-13T07:39:53.750 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:53.751 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:53.751 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:53.751 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:35:46.147466Z" 2022-09-13T07:39:53.752 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:53.752 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:53.752 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:53.753 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:39:53.753 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:53.753 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:53.753 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000002063", 2022-09-13T07:39:53.754 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:53.754 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:53.754 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:53.755 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:39:41.890430Z" 2022-09-13T07:39:53.755 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:53.755 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:53.755 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:53.756 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:39:53.756 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:53.756 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:53.757 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000006575", 2022-09-13T07:39:53.757 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:53.757 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:53.758 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:53.758 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:37:29.880837Z" 2022-09-13T07:39:53.758 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:53.759 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:53.759 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:53.759 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:39:53.760 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:53.760 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:53.760 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:39:53.760 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:53.761 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:53.761 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:53.761 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000" 2022-09-13T07:39:53.761 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:53.762 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:53.762 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:39:53.762 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:53.762 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:39:53.763 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 0, 2022-09-13T07:39:53.763 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 0 2022-09-13T07:39:53.763 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:53.764 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:39:53Z" 2022-09-13T07:39:53.764 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:39:53.765 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11637822015646413467 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003050", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:35:46.147466Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:39:41.890430Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006575", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:37:29.880837Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-13T07:39:53Z" } 2022-09-13T07:39:53.766 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003050', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:35:46.147466Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:39:41.890430Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006575', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:37:29.880837Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-13T07:39:53.767 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003050', 1: '00000000000000000000:00000000000000002063', 2: '00000000000000000000:00000000000000006575', 3: ''} 2022-09-13T07:39:53.768 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003050', 1: '00000000000000000000:00000000000000002063', 2: '00000000000000000000:00000000000000006575', 3: ''} 2022-09-13T07:39:53.768 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-13T07:39:53.769 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2022-09-13T07:39:53.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:53.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:53.771 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:39:53.823 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:53.823 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:53.880 INFO:teuthology.orchestra.run.smithi083.stdout:[ 2022-09-13T07:39:53.881 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:53.881 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-13T07:39:53.881 INFO:teuthology.orchestra.run.smithi083.stdout: "last_update": "2022-09-13T07:35:14.720968Z" 2022-09-13T07:39:53.882 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:53.882 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:53.883 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "00000000000000000000:00000000000000001358", 2022-09-13T07:39:53.883 INFO:teuthology.orchestra.run.smithi083.stdout: "last_update": "2022-09-13T07:38:47.047775Z" 2022-09-13T07:39:53.883 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:53.883 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:53.884 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-13T07:39:53.884 INFO:teuthology.orchestra.run.smithi083.stdout: "last_update": "2022-09-13T07:37:29.739220Z" 2022-09-13T07:39:53.884 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:53.884 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:53.885 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:39:53.885 INFO:teuthology.orchestra.run.smithi083.stdout: "last_update": "0.000000" 2022-09-13T07:39:53.885 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:53.886 INFO:teuthology.orchestra.run.smithi083.stdout:] 2022-09-13T07:39:53.887 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000005870', 3: ''} 2022-09-13T07:39:53.887 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-13T07:39:53.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:53.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:53.889 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:39:53.943 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:53.943 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:54.006 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:39:54.006 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:39:54.006 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:39:54.007 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:39:54.007 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:39:54.007 INFO:teuthology.orchestra.run.smithi036.stdout: "instance_id": 5812143756076532791 2022-09-13T07:39:54.008 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.008 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:39:54.008 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.008 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:39:54.009 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:54.009 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.009 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-13T07:39:54.009 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:54.010 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:54.010 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:54.010 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:35:14.720968Z" 2022-09-13T07:39:54.010 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.011 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.011 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.012 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:39:54.012 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:54.012 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.013 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000001358", 2022-09-13T07:39:54.013 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:54.013 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:54.014 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:54.014 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:38:47.047775Z" 2022-09-13T07:39:54.014 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.014 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.015 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.015 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:39:54.015 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:54.016 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.016 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-13T07:39:54.016 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:54.016 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:54.017 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:54.017 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:37:29.739220Z" 2022-09-13T07:39:54.017 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.018 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.018 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.018 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:39:54.018 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:54.019 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.019 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:39:54.019 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:54.019 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:54.020 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:54.020 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000" 2022-09-13T07:39:54.020 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.020 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.021 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:39:54.021 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.021 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:39:54.021 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 0, 2022-09-13T07:39:54.022 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 0 2022-09-13T07:39:54.022 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.022 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:39:53Z" 2022-09-13T07:39:54.022 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:39:54.024 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5812143756076532791 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:35:14.720968Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001358", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:38:47.047775Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005870", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:37:29.739220Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-13T07:39:53Z" } 2022-09-13T07:39:54.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:35:14.720968Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001358', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:38:47.047775Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:37:29.739220Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-13T07:39:54.025 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000005870', 3: ''} 2022-09-13T07:39:54.025 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000005870', 3: ''} 2022-09-13T07:39:54.026 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-13T07:39:54.026 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2022-09-13T07:39:54.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.028 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:39:54.077 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:54.077 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:54.119 INFO:teuthology.orchestra.run.smithi083.stdout:[ 2022-09-13T07:39:54.119 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:54.120 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-13T07:39:54.120 INFO:teuthology.orchestra.run.smithi083.stdout: "last_update": "2022-09-13T07:35:14.720968Z" 2022-09-13T07:39:54.121 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:54.121 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:54.122 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "00000000000000000000:00000000000000001358", 2022-09-13T07:39:54.122 INFO:teuthology.orchestra.run.smithi083.stdout: "last_update": "2022-09-13T07:38:47.047775Z" 2022-09-13T07:39:54.123 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:54.123 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:54.124 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-13T07:39:54.124 INFO:teuthology.orchestra.run.smithi083.stdout: "last_update": "2022-09-13T07:37:29.739220Z" 2022-09-13T07:39:54.125 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:54.125 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:54.126 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:39:54.126 INFO:teuthology.orchestra.run.smithi083.stdout: "last_update": "0.000000" 2022-09-13T07:39:54.127 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:54.127 INFO:teuthology.orchestra.run.smithi083.stdout:] 2022-09-13T07:39:54.130 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000005870', 3: ''} 2022-09-13T07:39:54.131 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-13T07:39:54.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.134 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:39:54.189 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:54.190 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:54.240 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:39:54.240 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:39:54.240 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:39:54.241 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:39:54.241 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:39:54.241 INFO:teuthology.orchestra.run.smithi036.stdout: "instance_id": 16090692030378027563 2022-09-13T07:39:54.242 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.242 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:39:54.242 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.242 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:39:54.243 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:54.243 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.243 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-13T07:39:54.244 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:54.244 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:54.244 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:54.245 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:35:14.720968Z" 2022-09-13T07:39:54.245 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.245 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.246 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.246 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:39:54.246 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:54.246 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.247 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000001358", 2022-09-13T07:39:54.247 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:54.247 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:54.248 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:54.248 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:38:47.047775Z" 2022-09-13T07:39:54.248 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.248 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.249 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.249 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:39:54.249 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:54.250 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.250 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-13T07:39:54.250 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:54.251 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:54.251 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:54.251 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:37:29.739220Z" 2022-09-13T07:39:54.251 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.252 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.252 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.252 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:39:54.253 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:54.253 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.253 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:39:54.253 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:54.254 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:54.254 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:54.254 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000" 2022-09-13T07:39:54.255 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.255 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.255 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:39:54.255 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.256 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:39:54.256 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 0, 2022-09-13T07:39:54.256 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 0 2022-09-13T07:39:54.256 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.257 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:39:54Z" 2022-09-13T07:39:54.257 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:39:54.259 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16090692030378027563 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:35:14.720968Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001358", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:38:47.047775Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005870", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:37:29.739220Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-13T07:39:54Z" } 2022-09-13T07:39:54.260 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:35:14.720968Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001358', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:38:47.047775Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:37:29.739220Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-13T07:39:54.261 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000005870', 3: ''} 2022-09-13T07:39:54.262 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000005870', 3: ''} 2022-09-13T07:39:54.263 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-13T07:39:54.264 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2022-09-13T07:39:54.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.267 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:39:54.317 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:54.318 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:54.366 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:39:54.367 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.368 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-13T07:39:54.368 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:35:32.294217Z" 2022-09-13T07:39:54.369 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.369 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.370 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-13T07:39:54.370 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:39:12.270190Z" 2022-09-13T07:39:54.371 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.371 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.372 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-13T07:39:54.372 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:37:13.928968Z" 2022-09-13T07:39:54.373 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.373 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.374 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:39:54.374 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:39:54.375 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.375 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:39:54.376 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005870', 3: ''} 2022-09-13T07:39:54.377 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-13T07:39:54.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.380 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:39:54.433 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:54.434 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:54.483 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:39:54.484 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:39:54.484 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:39:54.485 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:39:54.485 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:39:54.486 INFO:teuthology.orchestra.run.smithi036.stdout: "instance_id": 10389248121584615780 2022-09-13T07:39:54.486 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.486 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:39:54.487 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.487 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:39:54.488 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:54.488 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.488 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-13T07:39:54.489 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:54.490 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:54.490 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:54.491 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:35:32.294217Z" 2022-09-13T07:39:54.491 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.491 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.491 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.492 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:39:54.492 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:54.492 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.492 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-13T07:39:54.493 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:54.493 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:54.493 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:54.493 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:39:12.270190Z" 2022-09-13T07:39:54.494 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.494 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.494 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.495 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:39:54.495 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:54.495 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.495 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-13T07:39:54.496 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:54.496 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:54.496 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:54.496 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:37:13.928968Z" 2022-09-13T07:39:54.497 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.497 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.497 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.497 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:39:54.498 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:39:54.498 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.499 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:39:54.499 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:39:54.500 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:39:54.500 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:39:54.500 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000" 2022-09-13T07:39:54.501 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.501 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.502 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:39:54.502 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.503 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:39:54.503 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 0, 2022-09-13T07:39:54.504 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 0 2022-09-13T07:39:54.504 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.505 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:39:54Z" 2022-09-13T07:39:54.505 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:39:54.507 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10389248121584615780 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:35:32.294217Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:39:12.270190Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005870", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:37:13.928968Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-13T07:39:54Z" } 2022-09-13T07:39:54.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:35:32.294217Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:39:12.270190Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:37:13.928968Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-13T07:39:54.509 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005870', 3: ''} 2022-09-13T07:39:54.510 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005870', 3: ''} 2022-09-13T07:39:54.511 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-13T07:39:54.511 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2022-09-13T07:39:54.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.514 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:39:54.564 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:54.564 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:54.620 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:39:54.621 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.622 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-13T07:39:54.622 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:35:32.294217Z" 2022-09-13T07:39:54.623 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.623 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.624 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-13T07:39:54.624 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:39:12.270190Z" 2022-09-13T07:39:54.625 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.625 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.626 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-13T07:39:54.626 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:37:13.928968Z" 2022-09-13T07:39:54.627 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.627 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.628 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:39:54.628 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:39:54.629 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.629 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:39:54.632 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005870', 3: ''} 2022-09-13T07:39:54.633 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-13T07:39:54.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.636 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:39:54.688 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:54.689 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:54.740 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:39:54.741 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:39:54.741 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:39:54.741 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:39:54.742 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:39:54.742 INFO:teuthology.orchestra.run.smithi083.stdout: "instance_id": 15082550319938433790 2022-09-13T07:39:54.742 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:54.743 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:39:54.743 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:54.743 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:39:54.744 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:39:54.744 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.744 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-13T07:39:54.745 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:39:54.745 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:39:54.746 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:39:54.746 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:35:32.294217Z" 2022-09-13T07:39:54.747 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:54.747 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:54.748 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:54.748 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:39:54.749 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:39:54.749 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.750 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-13T07:39:54.750 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:39:54.751 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:39:54.751 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:39:54.752 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:39:12.270190Z" 2022-09-13T07:39:54.752 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:54.753 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:54.753 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:54.754 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:39:54.754 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:39:54.755 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.755 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-13T07:39:54.756 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:39:54.756 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:39:54.757 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:39:54.757 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:37:13.928968Z" 2022-09-13T07:39:54.758 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:54.758 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:54.759 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:54.759 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:39:54.760 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:39:54.760 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "incremental-sync", 2022-09-13T07:39:54.761 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:39:54.761 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:39:54.762 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:39:54.763 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:39:54.763 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000" 2022-09-13T07:39:54.764 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:54.764 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:54.765 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:39:54.765 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:54.766 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:39:54.768 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 0, 2022-09-13T07:39:54.768 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 0 2022-09-13T07:39:54.769 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:54.769 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:39:54Z" 2022-09-13T07:39:54.770 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:39:54.771 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15082550319938433790 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:35:32.294217Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:39:12.270190Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005870", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:37:13.928968Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-13T07:39:54Z" } 2022-09-13T07:39:54.772 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:35:32.294217Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:39:12.270190Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:37:13.928968Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-13T07:39:54.773 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005870', 3: ''} 2022-09-13T07:39:54.774 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005870', 3: ''} 2022-09-13T07:39:54.775 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-13T07:39:54.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2022-09-13T07:39:54.779 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:39:54.779 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:39:54.780 DEBUG:boto:Method: PUT 2022-09-13T07:39:54.780 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:39:54.781 DEBUG:boto:Data: 2022-09-13T07:39:54.781 DEBUG:boto:Headers: {} 2022-09-13T07:39:54.782 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:39:54.782 DEBUG:boto:Port: 8000 2022-09-13T07:39:54.783 DEBUG:boto:Params: {} 2022-09-13T07:39:54.783 DEBUG:boto:Token: None 2022-09-13T07:39:54.784 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:39:54 GMT /yxvvuh-20/ 2022-09-13T07:39:54.785 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tvFUVCE7KmkdvadwImSgcVFD5Tc= 2022-09-13T07:39:54.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tvFUVCE7KmkdvadwImSgcVFD5Tc=', 'Content-Length': '0'} 2022-09-13T07:39:54.803 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005cb1bb79be5104f1-00632033ca-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:39:54.804 DEBUG:rgw_multi.tests:created bucket=yxvvuh-20 2022-09-13T07:39:54.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.806 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:39:54.856 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:54.857 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:54.893 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:39:54.905 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:39:54.905 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.906 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:39:54.906 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:39:54.907 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.907 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.908 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:39:54.908 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:39:54.908 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.909 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.910 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054794.798856_147.1", 2022-09-13T07:39:54.911 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:39:54.798856Z" 2022-09-13T07:39:54.911 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:39:54.911 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:39:54.912 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:39:54.912 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T07:39:54.913 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:39:54.913 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:39:54.916 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054794.798856_147.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:39:54.917 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:39:54.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:39:54.920 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:39:54.972 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:39:54.972 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:39:55.016 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:39:55.011+0000 7fa91f5e5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:39:55.017 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:39:55.012+0000 7fa91f5e5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:39:55.034 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:39:55.035 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:39:55.036 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:39:55.036 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:39:55.037 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:39:55.037 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:39:55.038 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:39:55.038 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:55.039 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:39:55.039 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:55.040 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:39:55.041 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:39:55.041 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:39:55.042 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:39:55.042 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:39:55.043 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:39:55.044 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:39:55.044 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:39:55.045 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:39:55.045 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:55.046 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:55.046 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:55.047 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:39:55.047 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:39:55.048 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:39:55.048 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:39:55.049 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:39:55.050 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:39:55.050 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:39:55.051 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:39:55.051 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:39:55.052 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:55.052 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:55.053 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:55.053 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:39:55.054 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:39:55.054 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:39:55.055 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054794.791007_142.1", 2022-09-13T07:39:55.055 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:39:55.056 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:39:55.056 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:39:55.057 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:39:54.791007Z", 2022-09-13T07:39:55.057 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:39:55.058 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:55.058 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:55.059 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:39:55.060 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:39:55.061 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:39:55.062 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:39:55.062 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:39:55.063 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:39:55.063 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:39:55.064 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:39:55.064 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:39:55.065 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:39:55.065 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:55.066 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:39:55.067 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:39:55.067 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:55.068 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:39:55.068 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:39:55.069 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:39:55.069 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:39:55.070 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:39:55Z" 2022-09-13T07:39:55.070 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:39:55.073 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054794.791007_142.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:39:54.791007Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:39:55Z" } 2022-09-13T07:39:55.074 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054794.791007_142.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:39:54.791007Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:39:55.075 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054794.798856_147.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:39:55.076 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054794.791007_142.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:39:55.077 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663054794.798856_147.1 target=1_1663054794.791007_142.1 2022-09-13T07:40:00.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:40:00.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:40:00.080 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:40:00.138 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:40:00.138 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:40:00.176 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:00.172+0000 7f59cdc93600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:40:00.178 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:00.174+0000 7f59cdc93600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:40:00.205 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:40:00.205 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:40:00.206 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:40:00.206 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:40:00.207 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:40:00.207 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:40:00.208 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:40:00.208 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:40:00.209 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:40:00.209 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:40:00.209 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:40:00.209 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:40:00.210 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:40:00.210 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:40:00.210 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:40:00.211 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:40:00.211 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:40:00.212 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:40:00.213 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:40:00.213 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:40:00.213 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:40:00.214 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:40:00.215 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:40:00.215 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:40:00.216 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:40:00.216 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:40:00.216 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:40:00.217 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:40:00.218 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:40:00.218 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:40:00.219 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:40:00.219 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:40:00.219 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:40:00.220 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:40:00.220 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:40:00.221 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:40:00.221 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:40:00.221 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054794.798856_147.1", 2022-09-13T07:40:00.222 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:40:00.222 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:40:00.223 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:40:00.223 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:39:54.798856Z", 2022-09-13T07:40:00.224 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:40:00.224 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:40:00.225 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:40:00.226 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:40:00.226 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:40:00.227 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:40:00.227 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:40:00.228 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:40:00.228 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:40:00.229 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:40:00.229 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:40:00.229 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:40:00.230 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:40:00.230 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:40:00.230 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:40:00.230 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:40:00.231 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:40:00.231 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:40:00.231 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:40:00.231 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:40:00.232 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:40:00.232 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:40:00Z" 2022-09-13T07:40:00.232 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:40:00.233 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054794.798856_147.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:39:54.798856Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:40:00Z" } 2022-09-13T07:40:00.233 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054794.798856_147.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:39:54.798856Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:40:00.234 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054794.798856_147.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:40:00.234 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054794.798856_147.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:40:00.235 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:40:00.235 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:40:00.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:40:00.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:40:00.237 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:40:00.297 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:40:00.298 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:40:00.338 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:00.335+0000 7fe067b39600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:40:00.340 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:00.336+0000 7fe067b39600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:40:00.352 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:40:00.353 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:40:00.353 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:40:00.353 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:40:00.354 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:40:00.354 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:40:00.354 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:40:00.354 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:40:00.355 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:40:00.355 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:40:00.356 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:40:00.356 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:40:00.357 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:40:00.358 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:40:00.359 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:40:00.359 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:40:00.359 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:40:00.360 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:40:00.360 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:40:00.360 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:40:00.360 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:40:00.361 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:40:00.361 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:40:00.361 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:40:00.361 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:40:00.362 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:40:00.362 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:40:00.363 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:40:00.363 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:40:00.364 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:40:00.364 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:40:00.365 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:40:00.365 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:40:00.366 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:40:00.366 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:40:00.367 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:40:00.367 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:40:00.368 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054794.798856_147.1", 2022-09-13T07:40:00.368 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:40:00.369 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:40:00.369 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:40:00.370 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:39:54.798856Z", 2022-09-13T07:40:00.370 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:40:00.370 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:40:00.371 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:40:00.374 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:40:00.374 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:40:00.375 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:40:00.375 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:40:00.378 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:40:00.378 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:40:00.378 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:40:00.379 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:40:00.379 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:40:00.379 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:40:00.380 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:40:00.380 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:40:00.380 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:40:00.381 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:40:00.381 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:40:00.381 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:40:00.381 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:40:00.382 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:40:00.382 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:40:00Z" 2022-09-13T07:40:00.382 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:40:00.384 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054794.798856_147.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:39:54.798856Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:40:00Z" } 2022-09-13T07:40:00.384 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054794.798856_147.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:39:54.798856Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:40:00.385 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054794.798856_147.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:40:00.385 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054794.798856_147.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:40:00.386 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:40:00.386 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:00.387 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:00.387 DEBUG:boto:Method: HEAD 2022-09-13T07:40:00.388 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:40:00.388 DEBUG:boto:Data: 2022-09-13T07:40:00.389 DEBUG:boto:Headers: {} 2022-09-13T07:40:00.389 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:00.390 DEBUG:boto:Port: 8000 2022-09-13T07:40:00.390 DEBUG:boto:Params: {} 2022-09-13T07:40:00.391 DEBUG:boto:Token: None 2022-09-13T07:40:00.391 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:00 GMT /yxvvuh-20/ 2022-09-13T07:40:00.392 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:g6KCc9dFIwzxZw1yXQ9N1H+WNpM= 2022-09-13T07:40:00.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:00 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:g6KCc9dFIwzxZw1yXQ9N1H+WNpM=', 'Content-Length': '0'} 2022-09-13T07:40:00.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf8d066aa90fbfaa-00632033d0-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:00.396 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:00.397 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:00.397 DEBUG:boto:Method: PUT 2022-09-13T07:40:00.398 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:40:00.398 DEBUG:boto:Data: 2022-09-13T07:40:00.399 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T07:40:00.399 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:00.400 DEBUG:boto:Port: 8000 2022-09-13T07:40:00.400 DEBUG:boto:Params: {} 2022-09-13T07:40:00.401 DEBUG:boto:Token: None 2022-09-13T07:40:00.401 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 13 Sep 2022 07:40:00 GMT /yxvvuh-20/obj 2022-09-13T07:40:00.402 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yZDtonOKDJwfZcjbjhPDYzFtG8A= 2022-09-13T07:40:00.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:40:00 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yZDtonOKDJwfZcjbjhPDYzFtG8A='} 2022-09-13T07:40:00.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f834321274da949-00632033d0-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:00.447 DEBUG:rgw_multi.tests:created initial version id=None 2022-09-13T07:40:00.449 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-20 2022-09-13T07:40:00.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:40:00.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:40:00.451 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:40:00.511 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:40:00.512 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:40:00.567 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:00.563+0000 7fdd41a49600 1 waiting to reach incremental sync.. 2022-09-13T07:40:05.569 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:05.564+0000 7fdd41a49600 1 waiting to reach incremental sync.. 2022-09-13T07:40:10.570 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:10.565+0000 7fdd41a49600 1 bucket sync caught up with source: 2022-09-13T07:40:10.570 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T07:40:10.570 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T07:40:10.571 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:10.565+0000 7fdd41a49600 0 bucket checkpoint complete 2022-09-13T07:40:10.587 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-20 2022-09-13T07:40:10.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:40:10.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:40:10.590 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:40:10.657 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:40:10.657 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:40:10.699 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:10.695+0000 7f102b7de600 1 waiting to reach incremental sync.. 2022-09-13T07:40:15.701 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:15.697+0000 7f102b7de600 1 bucket sync caught up with source: 2022-09-13T07:40:15.701 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T07:40:15.702 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T07:40:15.702 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:15.697+0000 7f102b7de600 0 bucket checkpoint complete 2022-09-13T07:40:15.722 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-20 2022-09-13T07:40:15.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:40:15.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:40:15.725 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:40:15.789 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:40:15.790 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:40:15.835 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:15.831+0000 7f74bdc9d600 1 waiting to reach incremental sync.. 2022-09-13T07:40:20.835 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:20.832+0000 7f74bdc9d600 1 waiting to reach incremental sync.. 2022-09-13T07:40:25.836 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:25.832+0000 7f74bdc9d600 1 waiting to reach incremental sync.. 2022-09-13T07:40:30.838 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:30.834+0000 7f74bdc9d600 1 bucket sync caught up with source: 2022-09-13T07:40:30.838 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T07:40:30.839 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T07:40:30.839 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:30.834+0000 7f74bdc9d600 0 bucket checkpoint complete 2022-09-13T07:40:30.856 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-20 2022-09-13T07:40:30.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:40:30.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:40:30.858 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:40:30.922 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:40:30.923 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:40:30.975 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:30.971+0000 7fcf9662a600 1 bucket sync caught up with source: 2022-09-13T07:40:30.976 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T07:40:30.976 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T07:40:30.976 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:30.971+0000 7fcf9662a600 0 bucket checkpoint complete 2022-09-13T07:40:30.989 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-20 2022-09-13T07:40:30.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:40:30.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:40:30.991 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:40:31.048 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:40:31.049 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:40:31.085 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:31.081+0000 7f90ff6bd600 1 bucket sync caught up with source: 2022-09-13T07:40:31.085 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000002.64.5, , ] 2022-09-13T07:40:31.086 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000002.64.5, , ] 2022-09-13T07:40:31.086 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:31.081+0000 7f90ff6bd600 0 bucket checkpoint complete 2022-09-13T07:40:31.098 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-20 2022-09-13T07:40:31.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:40:31.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:40:31.099 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:40:31.164 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:40:31.164 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:40:31.224 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:31.220+0000 7fea0246e600 1 bucket sync caught up with source: 2022-09-13T07:40:31.224 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000002.64.5, , ] 2022-09-13T07:40:31.224 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000002.64.5, , ] 2022-09-13T07:40:31.225 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:31.220+0000 7fea0246e600 0 bucket checkpoint complete 2022-09-13T07:40:31.240 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-20 zones={test-zone2, test-zone1} 2022-09-13T07:40:31.241 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.242 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.242 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.243 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:40:31.243 DEBUG:boto:Data: 2022-09-13T07:40:31.244 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.244 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.244 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.245 DEBUG:boto:Params: {} 2022-09-13T07:40:31.246 DEBUG:boto:Token: None 2022-09-13T07:40:31.246 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/ 2022-09-13T07:40:31.247 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U= 2022-09-13T07:40:31.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U=', 'Content-Length': '0'} 2022-09-13T07:40:31.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc1446275dc5dfbd-00632033ef-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.250 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.251 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.251 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.252 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:40:31.252 DEBUG:boto:Data: 2022-09-13T07:40:31.253 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.253 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.254 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.254 DEBUG:boto:Params: {} 2022-09-13T07:40:31.255 DEBUG:boto:Token: None 2022-09-13T07:40:31.255 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/ 2022-09-13T07:40:31.256 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U= 2022-09-13T07:40:31.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U=', 'Content-Length': '0'} 2022-09-13T07:40:31.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002271181554045bb1-00632033ef-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.261 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:40:31.261 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.262 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.262 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.263 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.263 DEBUG:boto:Method: GET 2022-09-13T07:40:31.264 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.264 DEBUG:boto:Data: 2022-09-13T07:40:31.265 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.265 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.266 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.266 DEBUG:boto:Params: {} 2022-09-13T07:40:31.267 DEBUG:boto:Token: None 2022-09-13T07:40:31.267 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/?versions 2022-09-13T07:40:31.268 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T07:40:31.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T07:40:31.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e987d43602f27a3-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.271 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:40:31.273 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:40:31.273 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.274 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.274 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.274 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.275 DEBUG:boto:Method: GET 2022-09-13T07:40:31.275 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.276 DEBUG:boto:Data: 2022-09-13T07:40:31.276 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.277 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.277 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.278 DEBUG:boto:Params: {} 2022-09-13T07:40:31.279 DEBUG:boto:Token: None 2022-09-13T07:40:31.279 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/?versions 2022-09-13T07:40:31.280 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T07:40:31.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T07:40:31.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000474bd447e98d848a-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.283 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:40:31.285 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.285 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.286 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.286 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.287 DEBUG:boto:Method: GET 2022-09-13T07:40:31.287 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.288 DEBUG:boto:Data: 2022-09-13T07:40:31.288 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.288 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.289 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.289 DEBUG:boto:Params: {} 2022-09-13T07:40:31.290 DEBUG:boto:Token: None 2022-09-13T07:40:31.291 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/?versions 2022-09-13T07:40:31.291 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T07:40:31.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T07:40:31.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac9687f56b18abaf-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.294 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.295 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.296 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.296 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.297 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.297 DEBUG:boto:Method: GET 2022-09-13T07:40:31.298 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.298 DEBUG:boto:Data: 2022-09-13T07:40:31.299 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.299 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.300 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.300 DEBUG:boto:Params: {} 2022-09-13T07:40:31.301 DEBUG:boto:Token: None 2022-09-13T07:40:31.301 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/?versions 2022-09-13T07:40:31.302 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T07:40:31.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T07:40:31.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095d0a18410f7a15e-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.306 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:40:31.307 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.308 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.308 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.309 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.309 DEBUG:boto:Method: GET 2022-09-13T07:40:31.310 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.310 DEBUG:boto:Data: 2022-09-13T07:40:31.311 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.311 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.312 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.312 DEBUG:boto:Params: {} 2022-09-13T07:40:31.313 DEBUG:boto:Token: None 2022-09-13T07:40:31.313 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.314 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T07:40:31.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T07:40:31.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000071bab7a36bf4834-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.317 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.318 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.318 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.319 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.319 DEBUG:boto:Method: GET 2022-09-13T07:40:31.320 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.320 DEBUG:boto:Data: 2022-09-13T07:40:31.321 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.321 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.322 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.322 DEBUG:boto:Params: {} 2022-09-13T07:40:31.323 DEBUG:boto:Token: None 2022-09-13T07:40:31.323 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.324 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T07:40:31.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T07:40:31.327 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040d38cddfe3d9b43-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.327 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.328 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.328 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.329 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.329 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.330 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.330 DEBUG:boto:Data: 2022-09-13T07:40:31.331 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.331 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.332 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.332 DEBUG:boto:Params: {} 2022-09-13T07:40:31.333 DEBUG:boto:Token: None 2022-09-13T07:40:31.333 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.334 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ= 2022-09-13T07:40:31.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ=', 'Content-Length': '0'} 2022-09-13T07:40:31.336 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e69f3db726a03a97-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.337 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.337 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.338 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.338 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.339 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.339 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.340 DEBUG:boto:Data: 2022-09-13T07:40:31.340 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.341 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.341 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.342 DEBUG:boto:Params: {} 2022-09-13T07:40:31.343 DEBUG:boto:Token: None 2022-09-13T07:40:31.343 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.344 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ= 2022-09-13T07:40:31.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ=', 'Content-Length': '0'} 2022-09-13T07:40:31.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000061a62398be3efffc-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:40:31.347 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.348 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.348 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.349 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.349 DEBUG:boto:Method: GET 2022-09-13T07:40:31.350 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.350 DEBUG:boto:Data: 2022-09-13T07:40:31.351 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.351 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.352 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.352 DEBUG:boto:Params: {} 2022-09-13T07:40:31.353 DEBUG:boto:Token: None 2022-09-13T07:40:31.354 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.354 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T07:40:31.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T07:40:31.357 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003b60881f0f8ff629-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.358 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.358 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.359 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.359 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.360 DEBUG:boto:Method: GET 2022-09-13T07:40:31.361 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.361 DEBUG:boto:Data: 2022-09-13T07:40:31.362 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.363 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.364 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.364 DEBUG:boto:Params: {} 2022-09-13T07:40:31.365 DEBUG:boto:Token: None 2022-09-13T07:40:31.365 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.366 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T07:40:31.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T07:40:31.368 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000921a22419b17ccd6-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.369 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.370 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.371 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.371 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.371 DEBUG:boto:Method: GET 2022-09-13T07:40:31.372 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.372 DEBUG:boto:Data: 2022-09-13T07:40:31.373 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.373 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.374 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.374 DEBUG:boto:Params: {} 2022-09-13T07:40:31.375 DEBUG:boto:Token: None 2022-09-13T07:40:31.376 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/ 2022-09-13T07:40:31.376 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T07:40:31.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T07:40:31.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009686484e39d37822-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.422 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.423 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.424 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.424 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.425 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.425 DEBUG:boto:Method: GET 2022-09-13T07:40:31.426 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.426 DEBUG:boto:Data: 2022-09-13T07:40:31.427 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.427 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.428 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.428 DEBUG:boto:Params: {} 2022-09-13T07:40:31.429 DEBUG:boto:Token: None 2022-09-13T07:40:31.429 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/ 2022-09-13T07:40:31.430 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T07:40:31.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T07:40:31.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b29448057fce256-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.433 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.434 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.435 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.435 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.436 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:40:31.436 DEBUG:boto:Data: 2022-09-13T07:40:31.437 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.437 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.438 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.438 DEBUG:boto:Params: {} 2022-09-13T07:40:31.439 DEBUG:boto:Token: None 2022-09-13T07:40:31.439 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj 2022-09-13T07:40:31.440 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA= 2022-09-13T07:40:31.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA=', 'Content-Length': '0'} 2022-09-13T07:40:31.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000024a44c312a7bc20b-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.443 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.444 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.444 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.445 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:40:31.445 DEBUG:boto:Data: 2022-09-13T07:40:31.445 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.446 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.446 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.447 DEBUG:boto:Params: {} 2022-09-13T07:40:31.448 DEBUG:boto:Token: None 2022-09-13T07:40:31.448 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj 2022-09-13T07:40:31.449 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA= 2022-09-13T07:40:31.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA=', 'Content-Length': '0'} 2022-09-13T07:40:31.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005ead47aa89c4a024-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:40:31.452 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.453 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.454 DEBUG:boto:Method: GET 2022-09-13T07:40:31.454 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:40:31.455 DEBUG:boto:Data: 2022-09-13T07:40:31.455 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.456 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.456 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.457 DEBUG:boto:Params: {} 2022-09-13T07:40:31.458 DEBUG:boto:Token: None 2022-09-13T07:40:31.458 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj 2022-09-13T07:40:31.459 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y= 2022-09-13T07:40:31.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y=', 'Content-Length': '0'} 2022-09-13T07:40:31.461 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000056e4fdd3cc163b02-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.462 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.463 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.463 DEBUG:boto:Method: GET 2022-09-13T07:40:31.464 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:40:31.464 DEBUG:boto:Data: 2022-09-13T07:40:31.465 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.465 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.466 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.466 DEBUG:boto:Params: {} 2022-09-13T07:40:31.467 DEBUG:boto:Token: None 2022-09-13T07:40:31.468 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj 2022-09-13T07:40:31.468 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y= 2022-09-13T07:40:31.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y=', 'Content-Length': '0'} 2022-09-13T07:40:31.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080c93ddb34d541f2-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.471 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.472 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.472 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.473 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.474 DEBUG:boto:Method: GET 2022-09-13T07:40:31.474 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.475 DEBUG:boto:Data: 2022-09-13T07:40:31.475 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.476 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.476 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.477 DEBUG:boto:Params: {} 2022-09-13T07:40:31.477 DEBUG:boto:Token: None 2022-09-13T07:40:31.478 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/ 2022-09-13T07:40:31.478 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T07:40:31.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T07:40:31.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f12184f1dadd221-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.482 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.483 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.483 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.484 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.485 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.485 DEBUG:boto:Method: GET 2022-09-13T07:40:31.486 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.486 DEBUG:boto:Data: 2022-09-13T07:40:31.487 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.488 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.488 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.489 DEBUG:boto:Params: {} 2022-09-13T07:40:31.489 DEBUG:boto:Token: None 2022-09-13T07:40:31.490 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/ 2022-09-13T07:40:31.490 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T07:40:31.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T07:40:31.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c350c2cc490a0bcc-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.494 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.495 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-20 zones={test-zone2, test-zone1} 2022-09-13T07:40:31.496 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-20 zones={test-zone3, test-zone1} 2022-09-13T07:40:31.496 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.497 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.497 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.498 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:40:31.498 DEBUG:boto:Data: 2022-09-13T07:40:31.499 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.499 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.500 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.500 DEBUG:boto:Params: {} 2022-09-13T07:40:31.501 DEBUG:boto:Token: None 2022-09-13T07:40:31.502 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/ 2022-09-13T07:40:31.502 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U= 2022-09-13T07:40:31.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U=', 'Content-Length': '0'} 2022-09-13T07:40:31.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d54da016e15f76ee-00632033ef-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.506 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.507 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.507 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.508 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:40:31.509 DEBUG:boto:Data: 2022-09-13T07:40:31.509 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.510 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.510 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.510 DEBUG:boto:Params: {} 2022-09-13T07:40:31.511 DEBUG:boto:Token: None 2022-09-13T07:40:31.512 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/ 2022-09-13T07:40:31.512 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U= 2022-09-13T07:40:31.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U=', 'Content-Length': '0'} 2022-09-13T07:40:31.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e2f2a917c6b753a-00632033ef-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.516 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:40:31.517 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.517 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.518 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.518 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.519 DEBUG:boto:Method: GET 2022-09-13T07:40:31.519 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.520 DEBUG:boto:Data: 2022-09-13T07:40:31.520 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.520 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.521 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.521 DEBUG:boto:Params: {} 2022-09-13T07:40:31.522 DEBUG:boto:Token: None 2022-09-13T07:40:31.523 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/?versions 2022-09-13T07:40:31.523 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T07:40:31.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T07:40:31.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082e6eb060106538c-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.527 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:40:31.529 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:40:31.530 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.530 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.531 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.532 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.532 DEBUG:boto:Method: GET 2022-09-13T07:40:31.533 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.533 DEBUG:boto:Data: 2022-09-13T07:40:31.533 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.534 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.535 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.535 DEBUG:boto:Params: {} 2022-09-13T07:40:31.536 DEBUG:boto:Token: None 2022-09-13T07:40:31.537 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/?versions 2022-09-13T07:40:31.537 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T07:40:31.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T07:40:31.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9e1fab51ea0c8f2-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.542 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:40:31.543 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.544 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.544 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.545 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.545 DEBUG:boto:Method: GET 2022-09-13T07:40:31.546 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.547 DEBUG:boto:Data: 2022-09-13T07:40:31.548 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.548 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.549 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.550 DEBUG:boto:Params: {} 2022-09-13T07:40:31.550 DEBUG:boto:Token: None 2022-09-13T07:40:31.551 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/?versions 2022-09-13T07:40:31.552 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T07:40:31.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T07:40:31.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acc03080c8628507-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.598 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.599 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.600 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.600 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.601 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.601 DEBUG:boto:Method: GET 2022-09-13T07:40:31.602 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.602 DEBUG:boto:Data: 2022-09-13T07:40:31.603 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.603 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.604 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.604 DEBUG:boto:Params: {} 2022-09-13T07:40:31.605 DEBUG:boto:Token: None 2022-09-13T07:40:31.606 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/?versions 2022-09-13T07:40:31.607 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T07:40:31.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T07:40:31.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e66b7f0e9d1ae42d-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.610 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:40:31.612 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.612 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.613 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.613 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.614 DEBUG:boto:Method: GET 2022-09-13T07:40:31.615 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.615 DEBUG:boto:Data: 2022-09-13T07:40:31.616 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.616 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.617 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.617 DEBUG:boto:Params: {} 2022-09-13T07:40:31.618 DEBUG:boto:Token: None 2022-09-13T07:40:31.618 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.619 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T07:40:31.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T07:40:31.621 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003e9393209af32e06-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.622 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.623 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.623 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.624 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.624 DEBUG:boto:Method: GET 2022-09-13T07:40:31.625 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.625 DEBUG:boto:Data: 2022-09-13T07:40:31.626 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.626 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.627 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.627 DEBUG:boto:Params: {} 2022-09-13T07:40:31.628 DEBUG:boto:Token: None 2022-09-13T07:40:31.628 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.629 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T07:40:31.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T07:40:31.632 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bdbfcbad7ead7879-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.632 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.633 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.633 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.634 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.634 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.635 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.635 DEBUG:boto:Data: 2022-09-13T07:40:31.636 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.636 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.637 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.637 DEBUG:boto:Params: {} 2022-09-13T07:40:31.638 DEBUG:boto:Token: None 2022-09-13T07:40:31.639 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.639 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ= 2022-09-13T07:40:31.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ=', 'Content-Length': '0'} 2022-09-13T07:40:31.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036ed0c35adc5e53f-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.642 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.643 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.643 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.644 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.644 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.645 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.646 DEBUG:boto:Data: 2022-09-13T07:40:31.646 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.647 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.647 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.648 DEBUG:boto:Params: {} 2022-09-13T07:40:31.648 DEBUG:boto:Token: None 2022-09-13T07:40:31.649 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.649 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ= 2022-09-13T07:40:31.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ=', 'Content-Length': '0'} 2022-09-13T07:40:31.652 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000806aac0fb4e3fa3e-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:40:31.653 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.654 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.655 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.655 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.656 DEBUG:boto:Method: GET 2022-09-13T07:40:31.656 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.657 DEBUG:boto:Data: 2022-09-13T07:40:31.657 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.658 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.658 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.659 DEBUG:boto:Params: {} 2022-09-13T07:40:31.659 DEBUG:boto:Token: None 2022-09-13T07:40:31.660 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.661 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T07:40:31.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T07:40:31.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d1445664761efcb-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.664 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.665 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.665 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.666 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.667 DEBUG:boto:Method: GET 2022-09-13T07:40:31.667 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.668 DEBUG:boto:Data: 2022-09-13T07:40:31.668 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.669 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.669 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.670 DEBUG:boto:Params: {} 2022-09-13T07:40:31.670 DEBUG:boto:Token: None 2022-09-13T07:40:31.671 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.671 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T07:40:31.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T07:40:31.674 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fe915d4ff8c8c061-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.675 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.676 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.676 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.677 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.677 DEBUG:boto:Method: GET 2022-09-13T07:40:31.678 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.678 DEBUG:boto:Data: 2022-09-13T07:40:31.679 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.679 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.680 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.680 DEBUG:boto:Params: {} 2022-09-13T07:40:31.681 DEBUG:boto:Token: None 2022-09-13T07:40:31.681 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/ 2022-09-13T07:40:31.682 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T07:40:31.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T07:40:31.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000659290c8dc7623e2-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.686 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.688 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.688 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.689 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.689 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.690 DEBUG:boto:Method: GET 2022-09-13T07:40:31.690 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.691 DEBUG:boto:Data: 2022-09-13T07:40:31.691 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.692 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.692 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.693 DEBUG:boto:Params: {} 2022-09-13T07:40:31.694 DEBUG:boto:Token: None 2022-09-13T07:40:31.694 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/ 2022-09-13T07:40:31.695 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T07:40:31.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T07:40:31.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f7279c1ab67e72c-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.742 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.743 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.744 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.744 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.745 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:40:31.745 DEBUG:boto:Data: 2022-09-13T07:40:31.746 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.746 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.747 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.747 DEBUG:boto:Params: {} 2022-09-13T07:40:31.748 DEBUG:boto:Token: None 2022-09-13T07:40:31.748 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj 2022-09-13T07:40:31.750 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA= 2022-09-13T07:40:31.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA=', 'Content-Length': '0'} 2022-09-13T07:40:31.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000007996de83597c681-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.754 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.754 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.755 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.755 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:40:31.756 DEBUG:boto:Data: 2022-09-13T07:40:31.756 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.757 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.757 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.757 DEBUG:boto:Params: {} 2022-09-13T07:40:31.758 DEBUG:boto:Token: None 2022-09-13T07:40:31.759 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj 2022-09-13T07:40:31.759 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA= 2022-09-13T07:40:31.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA=', 'Content-Length': '0'} 2022-09-13T07:40:31.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000063936968378e5e71-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:40:31.764 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.764 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.765 DEBUG:boto:Method: GET 2022-09-13T07:40:31.765 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:40:31.766 DEBUG:boto:Data: 2022-09-13T07:40:31.766 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.767 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.767 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.768 DEBUG:boto:Params: {} 2022-09-13T07:40:31.768 DEBUG:boto:Token: None 2022-09-13T07:40:31.769 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj 2022-09-13T07:40:31.769 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y= 2022-09-13T07:40:31.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y=', 'Content-Length': '0'} 2022-09-13T07:40:31.772 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa267b0a0019819d-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.772 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.773 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.774 DEBUG:boto:Method: GET 2022-09-13T07:40:31.774 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:40:31.775 DEBUG:boto:Data: 2022-09-13T07:40:31.775 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.776 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.776 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.777 DEBUG:boto:Params: {} 2022-09-13T07:40:31.777 DEBUG:boto:Token: None 2022-09-13T07:40:31.778 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj 2022-09-13T07:40:31.778 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y= 2022-09-13T07:40:31.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y=', 'Content-Length': '0'} 2022-09-13T07:40:31.781 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000015dee212e01d6a93-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.782 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.783 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.783 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.784 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.784 DEBUG:boto:Method: GET 2022-09-13T07:40:31.785 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.785 DEBUG:boto:Data: 2022-09-13T07:40:31.786 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.786 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.787 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.787 DEBUG:boto:Params: {} 2022-09-13T07:40:31.788 DEBUG:boto:Token: None 2022-09-13T07:40:31.789 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/ 2022-09-13T07:40:31.789 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T07:40:31.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T07:40:31.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080d92b896010004e-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.793 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.794 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.795 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.795 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.796 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.796 DEBUG:boto:Method: GET 2022-09-13T07:40:31.797 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.797 DEBUG:boto:Data: 2022-09-13T07:40:31.798 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.798 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:31.799 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.799 DEBUG:boto:Params: {} 2022-09-13T07:40:31.800 DEBUG:boto:Token: None 2022-09-13T07:40:31.801 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/ 2022-09-13T07:40:31.801 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T07:40:31.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T07:40:31.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030be43d9f3b8e04f-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.805 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.806 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-20 zones={test-zone3, test-zone1} 2022-09-13T07:40:31.807 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-20 zones={test-zone3, test-zone2} 2022-09-13T07:40:31.808 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.808 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.809 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.809 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:40:31.810 DEBUG:boto:Data: 2022-09-13T07:40:31.810 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.811 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.811 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.812 DEBUG:boto:Params: {} 2022-09-13T07:40:31.813 DEBUG:boto:Token: None 2022-09-13T07:40:31.814 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/ 2022-09-13T07:40:31.814 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U= 2022-09-13T07:40:31.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U=', 'Content-Length': '0'} 2022-09-13T07:40:31.817 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c500a12444a75af7-00632033ef-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.818 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.818 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.819 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.819 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:40:31.820 DEBUG:boto:Data: 2022-09-13T07:40:31.820 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.821 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.821 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.822 DEBUG:boto:Params: {} 2022-09-13T07:40:31.822 DEBUG:boto:Token: None 2022-09-13T07:40:31.823 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/ 2022-09-13T07:40:31.824 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U= 2022-09-13T07:40:31.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U=', 'Content-Length': '0'} 2022-09-13T07:40:31.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c62352cf5999388f-00632033ef-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.827 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:40:31.828 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.828 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.829 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.829 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.830 DEBUG:boto:Method: GET 2022-09-13T07:40:31.830 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.831 DEBUG:boto:Data: 2022-09-13T07:40:31.831 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.832 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.832 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.833 DEBUG:boto:Params: {} 2022-09-13T07:40:31.833 DEBUG:boto:Token: None 2022-09-13T07:40:31.835 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/?versions 2022-09-13T07:40:31.835 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T07:40:31.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T07:40:31.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e8d1852c83c4473-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.839 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:40:31.840 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:40:31.841 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.841 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.842 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.842 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.843 DEBUG:boto:Method: GET 2022-09-13T07:40:31.843 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.844 DEBUG:boto:Data: 2022-09-13T07:40:31.844 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.845 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.846 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.846 DEBUG:boto:Params: {} 2022-09-13T07:40:31.847 DEBUG:boto:Token: None 2022-09-13T07:40:31.847 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/?versions 2022-09-13T07:40:31.848 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T07:40:31.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T07:40:31.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d77de341b5ce19e-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.894 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:40:31.896 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.896 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.897 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.897 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.898 DEBUG:boto:Method: GET 2022-09-13T07:40:31.899 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.899 DEBUG:boto:Data: 2022-09-13T07:40:31.900 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.900 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.901 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.901 DEBUG:boto:Params: {} 2022-09-13T07:40:31.902 DEBUG:boto:Token: None 2022-09-13T07:40:31.902 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/?versions 2022-09-13T07:40:31.903 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T07:40:31.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T07:40:31.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca3e91577a7929bb-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.907 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.908 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.909 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.910 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.910 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.911 DEBUG:boto:Method: GET 2022-09-13T07:40:31.911 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:40:31.912 DEBUG:boto:Data: 2022-09-13T07:40:31.912 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.913 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.913 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.914 DEBUG:boto:Params: {} 2022-09-13T07:40:31.914 DEBUG:boto:Token: None 2022-09-13T07:40:31.915 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/?versions 2022-09-13T07:40:31.916 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T07:40:31.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T07:40:31.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddcf14121524b8db-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.920 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:31.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:40:31.921 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.922 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.922 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.923 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.924 DEBUG:boto:Method: GET 2022-09-13T07:40:31.925 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.925 DEBUG:boto:Data: 2022-09-13T07:40:31.926 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.927 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.928 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.929 DEBUG:boto:Params: {} 2022-09-13T07:40:31.930 DEBUG:boto:Token: None 2022-09-13T07:40:31.931 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.931 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T07:40:31.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T07:40:31.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002c53d97eb8894ad4-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.935 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.936 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.936 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.937 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.938 DEBUG:boto:Method: GET 2022-09-13T07:40:31.938 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.939 DEBUG:boto:Data: 2022-09-13T07:40:31.940 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.941 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.941 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.942 DEBUG:boto:Params: {} 2022-09-13T07:40:31.943 DEBUG:boto:Token: None 2022-09-13T07:40:31.944 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.945 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T07:40:31.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T07:40:31.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e39d90a4ec93b8a8-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.948 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.948 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.949 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.950 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.950 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.951 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.951 DEBUG:boto:Data: 2022-09-13T07:40:31.952 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.953 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.953 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.954 DEBUG:boto:Params: {} 2022-09-13T07:40:31.955 DEBUG:boto:Token: None 2022-09-13T07:40:31.955 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.956 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ= 2022-09-13T07:40:31.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ=', 'Content-Length': '0'} 2022-09-13T07:40:31.959 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f397e9f0a6a37d04-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.959 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.960 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.961 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.962 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.962 DEBUG:boto:Method: HEAD 2022-09-13T07:40:31.963 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.963 DEBUG:boto:Data: 2022-09-13T07:40:31.964 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.964 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.965 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.965 DEBUG:boto:Params: {} 2022-09-13T07:40:31.966 DEBUG:boto:Token: None 2022-09-13T07:40:31.966 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.967 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ= 2022-09-13T07:40:31.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ=', 'Content-Length': '0'} 2022-09-13T07:40:31.969 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c81f93f381d9bca6-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:40:31.971 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.971 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.972 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.972 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.973 DEBUG:boto:Method: GET 2022-09-13T07:40:31.973 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.974 DEBUG:boto:Data: 2022-09-13T07:40:31.974 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.975 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.975 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.976 DEBUG:boto:Params: {} 2022-09-13T07:40:31.977 DEBUG:boto:Token: None 2022-09-13T07:40:31.977 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.978 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T07:40:31.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T07:40:31.980 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a19f36e2311e364c-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.981 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:31.982 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:31.982 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.983 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.983 DEBUG:boto:Method: GET 2022-09-13T07:40:31.984 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.984 DEBUG:boto:Data: 2022-09-13T07:40:31.985 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.985 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:31.986 DEBUG:boto:Port: 8000 2022-09-13T07:40:31.986 DEBUG:boto:Params: {} 2022-09-13T07:40:31.987 DEBUG:boto:Token: None 2022-09-13T07:40:31.988 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:40:31.988 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T07:40:31.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T07:40:31.991 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd41594c6599690d-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:31.992 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:31.992 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:31.993 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.993 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.994 DEBUG:boto:Method: GET 2022-09-13T07:40:31.995 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:40:31.995 DEBUG:boto:Data: 2022-09-13T07:40:31.996 DEBUG:boto:Headers: {} 2022-09-13T07:40:31.996 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:31.997 DEBUG:boto:Port: 8001 2022-09-13T07:40:31.997 DEBUG:boto:Params: {} 2022-09-13T07:40:31.998 DEBUG:boto:Token: None 2022-09-13T07:40:31.998 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:31 GMT /yxvvuh-20/ 2022-09-13T07:40:31.999 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T07:40:31.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T07:40:32.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022ddb3c38713a812-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:32.046 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:32.047 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:32.049 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:32.049 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:32.050 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:32.051 DEBUG:boto:Method: GET 2022-09-13T07:40:32.052 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:40:32.052 DEBUG:boto:Data: 2022-09-13T07:40:32.053 DEBUG:boto:Headers: {} 2022-09-13T07:40:32.054 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:32.055 DEBUG:boto:Port: 8000 2022-09-13T07:40:32.055 DEBUG:boto:Params: {} 2022-09-13T07:40:32.056 DEBUG:boto:Token: None 2022-09-13T07:40:32.057 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:32 GMT /yxvvuh-20/ 2022-09-13T07:40:32.058 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EQR0ru4msaGkW4+U60wpbykFfiM= 2022-09-13T07:40:32.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EQR0ru4msaGkW4+U60wpbykFfiM=', 'Content-Length': '0'} 2022-09-13T07:40:32.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b58daa4279160623-00632033f0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:32.063 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:32.064 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:32.065 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:32.066 DEBUG:boto:Method: HEAD 2022-09-13T07:40:32.066 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:40:32.067 DEBUG:boto:Data: 2022-09-13T07:40:32.067 DEBUG:boto:Headers: {} 2022-09-13T07:40:32.068 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:32.068 DEBUG:boto:Port: 8001 2022-09-13T07:40:32.069 DEBUG:boto:Params: {} 2022-09-13T07:40:32.069 DEBUG:boto:Token: None 2022-09-13T07:40:32.070 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:32 GMT /yxvvuh-20/obj 2022-09-13T07:40:32.070 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Up1ZYsmylWYV4SF1qH9sxKuRXuY= 2022-09-13T07:40:32.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Up1ZYsmylWYV4SF1qH9sxKuRXuY=', 'Content-Length': '0'} 2022-09-13T07:40:32.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000996bee48142421b9-00632033f0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:32.074 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:32.074 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:32.075 DEBUG:boto:Method: HEAD 2022-09-13T07:40:32.075 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:40:32.076 DEBUG:boto:Data: 2022-09-13T07:40:32.076 DEBUG:boto:Headers: {} 2022-09-13T07:40:32.077 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:32.077 DEBUG:boto:Port: 8000 2022-09-13T07:40:32.078 DEBUG:boto:Params: {} 2022-09-13T07:40:32.078 DEBUG:boto:Token: None 2022-09-13T07:40:32.079 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:32 GMT /yxvvuh-20/obj 2022-09-13T07:40:32.081 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Up1ZYsmylWYV4SF1qH9sxKuRXuY= 2022-09-13T07:40:32.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Up1ZYsmylWYV4SF1qH9sxKuRXuY=', 'Content-Length': '0'} 2022-09-13T07:40:32.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000033b9cc9d076a34bc-00632033f0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:32.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:40:32.086 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:32.086 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:32.087 DEBUG:boto:Method: GET 2022-09-13T07:40:32.087 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:40:32.088 DEBUG:boto:Data: 2022-09-13T07:40:32.088 DEBUG:boto:Headers: {} 2022-09-13T07:40:32.089 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:32.089 DEBUG:boto:Port: 8001 2022-09-13T07:40:32.090 DEBUG:boto:Params: {} 2022-09-13T07:40:32.092 DEBUG:boto:Token: None 2022-09-13T07:40:32.094 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:32 GMT /yxvvuh-20/obj 2022-09-13T07:40:32.096 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VGKw4BZsknAtOX3kJAhcyMhTRhI= 2022-09-13T07:40:32.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VGKw4BZsknAtOX3kJAhcyMhTRhI=', 'Content-Length': '0'} 2022-09-13T07:40:32.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000ef96f4078087b27-00632033f0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:32.100 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:32.101 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:32.102 DEBUG:boto:Method: GET 2022-09-13T07:40:32.103 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:40:32.104 DEBUG:boto:Data: 2022-09-13T07:40:32.105 DEBUG:boto:Headers: {} 2022-09-13T07:40:32.106 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:32.107 DEBUG:boto:Port: 8000 2022-09-13T07:40:32.108 DEBUG:boto:Params: {} 2022-09-13T07:40:32.109 DEBUG:boto:Token: None 2022-09-13T07:40:32.110 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:32 GMT /yxvvuh-20/obj 2022-09-13T07:40:32.111 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:VGKw4BZsknAtOX3kJAhcyMhTRhI= 2022-09-13T07:40:32.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VGKw4BZsknAtOX3kJAhcyMhTRhI=', 'Content-Length': '0'} 2022-09-13T07:40:32.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d981d4444f29d8e5-00632033f0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:32.115 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:32.116 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:32.117 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:32.118 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:32.119 DEBUG:boto:Method: GET 2022-09-13T07:40:32.120 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:40:32.121 DEBUG:boto:Data: 2022-09-13T07:40:32.122 DEBUG:boto:Headers: {} 2022-09-13T07:40:32.123 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:40:32.124 DEBUG:boto:Port: 8001 2022-09-13T07:40:32.125 DEBUG:boto:Params: {} 2022-09-13T07:40:32.126 DEBUG:boto:Token: None 2022-09-13T07:40:32.127 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:32 GMT /yxvvuh-20/ 2022-09-13T07:40:32.128 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EQR0ru4msaGkW4+U60wpbykFfiM= 2022-09-13T07:40:32.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EQR0ru4msaGkW4+U60wpbykFfiM=', 'Content-Length': '0'} 2022-09-13T07:40:32.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064df19905293c355-00632033f0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:32.133 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:32.134 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:32.135 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:32.136 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:32.137 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:40:32.138 DEBUG:boto:Method: GET 2022-09-13T07:40:32.139 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:40:32.140 DEBUG:boto:Data: 2022-09-13T07:40:32.141 DEBUG:boto:Headers: {} 2022-09-13T07:40:32.142 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:40:32.142 DEBUG:boto:Port: 8000 2022-09-13T07:40:32.143 DEBUG:boto:Params: {} 2022-09-13T07:40:32.144 DEBUG:boto:Token: None 2022-09-13T07:40:32.145 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:40:32 GMT /yxvvuh-20/ 2022-09-13T07:40:32.146 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:EQR0ru4msaGkW4+U60wpbykFfiM= 2022-09-13T07:40:32.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EQR0ru4msaGkW4+U60wpbykFfiM=', 'Content-Length': '0'} 2022-09-13T07:40:32.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000557c70c2e3adbfa6-00632033f0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:32.194 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:40:32.195 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-20 zones={test-zone3, test-zone2} 2022-09-13T07:40:32.196 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:32.197 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:32.197 DEBUG:boto:path=/yxvvuh-20/?versioning 2022-09-13T07:40:32.198 DEBUG:boto:auth_path=/yxvvuh-20/?versioning 2022-09-13T07:40:32.198 DEBUG:boto:Method: PUT 2022-09-13T07:40:32.199 DEBUG:boto:Path: /yxvvuh-20/?versioning 2022-09-13T07:40:32.199 DEBUG:boto:Data: Enabled Disabled 2022-09-13T07:40:32.200 DEBUG:boto:Headers: {} 2022-09-13T07:40:32.200 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:32.201 DEBUG:boto:Port: 8000 2022-09-13T07:40:32.201 DEBUG:boto:Params: {} 2022-09-13T07:40:32.202 DEBUG:boto:Token: None 2022-09-13T07:40:32.202 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:40:32 GMT /yxvvuh-20/?versioning 2022-09-13T07:40:32.203 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eAomLi6Vz1w8+lC2LUkNRTe561s= 2022-09-13T07:40:32.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eAomLi6Vz1w8+lC2LUkNRTe561s=', 'Content-Length': '228'} 2022-09-13T07:40:32.211 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ecf68a4e280da9be-00632033f0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:32.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:40:32.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:40:32.214 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:40:32.272 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:40:32.273 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:40:32.306 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:40:32.323 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:40:32.323 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:40:32.324 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:40:32.324 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:40:32.325 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:40:32.325 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:40:32.326 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:40:32.326 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:40:32.327 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:40:32.327 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:40:32.328 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054832.207579_149.1", 2022-09-13T07:40:32.328 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:40:32.207579Z" 2022-09-13T07:40:32.329 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:40:32.329 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:40:32.329 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:40:32.330 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T07:40:32.330 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:40:32.331 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:40:32.332 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054832.207579_149.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:40:32.333 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:40:32.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:40:32.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:40:32.335 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:40:32.393 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:40:32.394 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:40:32.425 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:32.422+0000 7f37db3b2600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:40:32.426 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:32.423+0000 7f37db3b2600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:40:32.445 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:40:32.446 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:40:32.446 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:40:32.447 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:40:32.447 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:40:32.448 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:40:32.448 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:40:32.449 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:40:32.449 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:40:32.449 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:40:32.450 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:40:32.450 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:40:32.451 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:40:32.451 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:40:32.451 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:40:32.452 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:40:32.452 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:40:32.453 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:40:32.453 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:40:32.454 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:40:32.454 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:40:32.455 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:40:32.455 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:40:32.456 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:40:32.456 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:40:32.456 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:40:32.457 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:40:32.457 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:40:32.458 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:40:32.458 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:40:32.459 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:40:32.459 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:40:32.460 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:40:32.460 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:40:32.461 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:40:32.461 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:40:32.462 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:40:32.462 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054832.207579_149.1", 2022-09-13T07:40:32.463 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:40:32.463 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:40:32.464 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:40:32.464 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:40:32.207579Z", 2022-09-13T07:40:32.465 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:40:32.465 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:40:32.465 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:40:32.466 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:40:32.466 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:40:32.466 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:40:32.467 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:40:32.467 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:40:32.468 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:40:32.468 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:40:32.468 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:40:32.469 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:40:32.469 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:40:32.469 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:40:32.470 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:40:32.470 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:40:32.471 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:40:32.471 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:40:32.472 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:40:32.472 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:40:32.473 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:40:32.473 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:40:32Z" 2022-09-13T07:40:32.474 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:40:32.475 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054832.207579_149.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:40:32.207579Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:40:32Z" } 2022-09-13T07:40:32.476 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054832.207579_149.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:40:32.207579Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:40:32.477 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054832.207579_149.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:40:32.477 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054832.207579_149.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:40:32.478 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:40:32.478 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:40:32.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:40:32.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:40:32.480 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:40:32.532 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:40:32.533 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:40:32.566 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:32.562+0000 7f850d0d5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:40:32.566 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:40:32.563+0000 7f850d0d5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:40:32.582 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:40:32.582 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:40:32.583 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:40:32.583 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:40:32.583 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:40:32.583 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:40:32.584 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:40:32.584 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:40:32.584 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:40:32.584 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:40:32.585 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:40:32.585 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:40:32.585 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:40:32.585 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:40:32.586 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:40:32.586 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:40:32.586 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:40:32.587 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:40:32.587 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:40:32.587 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:40:32.588 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:40:32.588 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:40:32.588 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:40:32.588 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:40:32.589 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:40:32.589 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:40:32.589 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:40:32.589 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:40:32.590 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:40:32.590 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:40:32.590 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:40:32.590 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:40:32.591 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:40:32.591 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:40:32.591 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:40:32.591 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:40:32.592 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:40:32.592 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054832.207579_149.1", 2022-09-13T07:40:32.592 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:40:32.592 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:40:32.593 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:40:32.593 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:40:32.207579Z", 2022-09-13T07:40:32.593 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:40:32.593 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:40:32.594 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:40:32.594 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:40:32.594 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:40:32.594 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:40:32.595 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:40:32.595 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:40:32.595 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:40:32.595 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:40:32.596 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:40:32.596 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:40:32.596 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:40:32.596 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:40:32.597 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:40:32.597 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:40:32.597 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:40:32.597 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:40:32.598 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:40:32.598 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:40:32.598 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:40:32.598 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:40:32Z" 2022-09-13T07:40:32.599 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:40:32.600 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054832.207579_149.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:40:32.207579Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:40:32Z" } 2022-09-13T07:40:32.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054832.207579_149.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:40:32.207579Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:40:32.601 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054832.207579_149.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:40:32.602 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054832.207579_149.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:40:32.603 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:40:32.603 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:40:32.604 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:40:32.605 DEBUG:boto:Method: HEAD 2022-09-13T07:40:32.605 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:40:32.606 DEBUG:boto:Data: 2022-09-13T07:40:32.606 DEBUG:boto:Headers: {} 2022-09-13T07:40:32.607 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:32.607 DEBUG:boto:Port: 8000 2022-09-13T07:40:32.608 DEBUG:boto:Params: {} 2022-09-13T07:40:32.608 DEBUG:boto:Token: None 2022-09-13T07:40:32.609 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:40:32 GMT /yxvvuh-20/ 2022-09-13T07:40:32.609 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FORhZGRUQbOyQrIevPhZRq1QEpM= 2022-09-13T07:40:32.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FORhZGRUQbOyQrIevPhZRq1QEpM=', 'Content-Length': '0'} 2022-09-13T07:40:32.612 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060adcb476143c2ee-00632033f0-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:32.613 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:40:32.614 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:40:32.614 DEBUG:boto:Method: PUT 2022-09-13T07:40:32.614 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:40:32.615 DEBUG:boto:Data: 2022-09-13T07:40:32.615 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T07:40:32.616 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:40:32.616 DEBUG:boto:Port: 8000 2022-09-13T07:40:32.617 DEBUG:boto:Params: {} 2022-09-13T07:40:32.617 DEBUG:boto:Token: None 2022-09-13T07:40:32.618 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 13 Sep 2022 07:40:32 GMT /yxvvuh-20/obj 2022-09-13T07:40:32.619 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WPhnKbuuHsG1m4vG8qHMH2zl2gE= 2022-09-13T07:40:32.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WPhnKbuuHsG1m4vG8qHMH2zl2gE='} 2022-09-13T07:40:32.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-amz-request-id', 'tx000001792f86485979461-00632033f0-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:40:32.639 DEBUG:rgw_multi.tests:created new version id=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:40:32.640 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-20 2022-09-13T07:40:32.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:40:32.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:40:32.643 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:40:32.701 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:40:32.701 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:40:32.746 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:32.743+0000 7f4414aaa600 1 waiting for incremental sync to catch up: 2022-09-13T07:40:32.747 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T07:40:32.748 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T07:40:37.748 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:37.744+0000 7f4414aaa600 1 waiting for incremental sync to catch up: 2022-09-13T07:40:37.749 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T07:40:37.749 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T07:40:42.749 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:42.745+0000 7f4414aaa600 1 waiting for incremental sync to catch up: 2022-09-13T07:40:42.750 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T07:40:42.750 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T07:40:47.751 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:47.746+0000 7f4414aaa600 1 waiting for incremental sync to catch up: 2022-09-13T07:40:47.751 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T07:40:47.752 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T07:40:52.752 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:52.748+0000 7f4414aaa600 1 waiting for incremental sync to catch up: 2022-09-13T07:40:52.753 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T07:40:52.754 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T07:40:57.754 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:40:57.749+0000 7f4414aaa600 1 waiting for incremental sync to catch up: 2022-09-13T07:40:57.755 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T07:40:57.755 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T07:41:02.755 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:41:02.751+0000 7f4414aaa600 1 waiting for incremental sync to catch up: 2022-09-13T07:41:02.756 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T07:41:02.756 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T07:41:07.757 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:41:07.753+0000 7f4414aaa600 1 bucket sync caught up with source: 2022-09-13T07:41:07.757 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T07:41:07.758 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T07:41:07.758 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:41:07.753+0000 7f4414aaa600 0 bucket checkpoint complete 2022-09-13T07:41:07.774 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-20 2022-09-13T07:41:07.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:07.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:07.777 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:41:07.844 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:41:07.844 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:41:07.892 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:07.889+0000 7f0b138fa600 1 bucket sync caught up with source: 2022-09-13T07:41:07.892 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T07:41:07.893 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T07:41:07.893 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:07.889+0000 7f0b138fa600 0 bucket checkpoint complete 2022-09-13T07:41:07.905 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-20 2022-09-13T07:41:07.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:07.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:07.907 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:41:07.965 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:41:07.966 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:41:08.001 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:07.998+0000 7f275b5ab600 1 waiting for incremental sync to catch up: 2022-09-13T07:41:08.001 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T07:41:08.002 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T07:41:13.002 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:12.998+0000 7f275b5ab600 1 waiting for incremental sync to catch up: 2022-09-13T07:41:13.003 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T07:41:13.003 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T07:41:18.004 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:17.999+0000 7f275b5ab600 1 waiting for incremental sync to catch up: 2022-09-13T07:41:18.005 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T07:41:18.005 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T07:41:23.013 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:23.001+0000 7f275b5ab600 1 waiting for incremental sync to catch up: 2022-09-13T07:41:23.013 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T07:41:23.014 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T07:41:28.006 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:28.002+0000 7f275b5ab600 1 bucket sync caught up with source: 2022-09-13T07:41:28.007 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T07:41:28.007 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T07:41:28.008 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:28.002+0000 7f275b5ab600 0 bucket checkpoint complete 2022-09-13T07:41:28.028 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-20 2022-09-13T07:41:28.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:28.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:28.031 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:41:28.094 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:41:28.095 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:41:28.131 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:28.127+0000 7faa3518d600 1 bucket sync caught up with source: 2022-09-13T07:41:28.132 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T07:41:28.132 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T07:41:28.133 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:28.127+0000 7faa3518d600 0 bucket checkpoint complete 2022-09-13T07:41:28.144 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-20 2022-09-13T07:41:28.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:28.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:28.146 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:41:28.204 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:41:28.205 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:41:28.251 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:28.247+0000 7fe052b24600 1 bucket sync caught up with source: 2022-09-13T07:41:28.251 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000005.70.9, , ] 2022-09-13T07:41:28.252 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000005.70.9, , ] 2022-09-13T07:41:28.252 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:28.247+0000 7fe052b24600 0 bucket checkpoint complete 2022-09-13T07:41:28.263 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-20 2022-09-13T07:41:28.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:28.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:28.265 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:41:28.329 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:41:28.329 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:41:29.377 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:41:29.373+0000 7f11e424a600 1 bucket sync caught up with source: 2022-09-13T07:41:29.377 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000005.70.9, , ] 2022-09-13T07:41:29.378 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000005.70.9, , ] 2022-09-13T07:41:29.378 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:41:29.373+0000 7f11e424a600 0 bucket checkpoint complete 2022-09-13T07:41:29.396 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-20 zones={test-zone2, test-zone1} 2022-09-13T07:41:29.397 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.398 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.398 DEBUG:boto:Method: HEAD 2022-09-13T07:41:29.399 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:41:29.399 DEBUG:boto:Data: 2022-09-13T07:41:29.400 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.400 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.401 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.401 DEBUG:boto:Params: {} 2022-09-13T07:41:29.402 DEBUG:boto:Token: None 2022-09-13T07:41:29.403 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/ 2022-09-13T07:41:29.403 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo= 2022-09-13T07:41:29.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo=', 'Content-Length': '0'} 2022-09-13T07:41:29.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000beaa5cdecac7c639-0063203429-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.408 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.408 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.409 DEBUG:boto:Method: HEAD 2022-09-13T07:41:29.409 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:41:29.410 DEBUG:boto:Data: 2022-09-13T07:41:29.410 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.411 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.411 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.412 DEBUG:boto:Params: {} 2022-09-13T07:41:29.412 DEBUG:boto:Token: None 2022-09-13T07:41:29.413 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/ 2022-09-13T07:41:29.413 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo= 2022-09-13T07:41:29.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo=', 'Content-Length': '0'} 2022-09-13T07:41:29.417 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087126cd3f906475f-0063203429-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.418 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:41:29.419 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.419 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.420 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.420 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.421 DEBUG:boto:Method: GET 2022-09-13T07:41:29.421 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.422 DEBUG:boto:Data: 2022-09-13T07:41:29.422 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.423 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.423 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.424 DEBUG:boto:Params: {} 2022-09-13T07:41:29.424 DEBUG:boto:Token: None 2022-09-13T07:41:29.425 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/?versions 2022-09-13T07:41:29.425 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs= 2022-09-13T07:41:29.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs=', 'Content-Length': '0'} 2022-09-13T07:41:29.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edaf8c4f098ec1c6-0063203429-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.429 DEBUG:boto:b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:29.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:41:29.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:41:29.432 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:41:29.432 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.433 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.433 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.434 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.434 DEBUG:boto:Method: GET 2022-09-13T07:41:29.435 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.435 DEBUG:boto:Data: 2022-09-13T07:41:29.436 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.436 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.437 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.437 DEBUG:boto:Params: {} 2022-09-13T07:41:29.438 DEBUG:boto:Token: None 2022-09-13T07:41:29.438 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/?versions 2022-09-13T07:41:29.439 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs= 2022-09-13T07:41:29.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs=', 'Content-Length': '0'} 2022-09-13T07:41:29.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d92c87d410642e88-0063203429-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.443 DEBUG:boto:b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:29.444 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:41:29.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:41:29.445 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.446 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.446 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.447 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.447 DEBUG:boto:Method: GET 2022-09-13T07:41:29.448 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.448 DEBUG:boto:Data: 2022-09-13T07:41:29.449 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.449 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.449 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.450 DEBUG:boto:Params: {} 2022-09-13T07:41:29.451 DEBUG:boto:Token: None 2022-09-13T07:41:29.451 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/?versions 2022-09-13T07:41:29.452 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs= 2022-09-13T07:41:29.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs=', 'Content-Length': '0'} 2022-09-13T07:41:29.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fa264013e1f15a0-0063203429-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.455 DEBUG:boto:b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:29.456 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.457 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.457 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.458 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.458 DEBUG:boto:Method: GET 2022-09-13T07:41:29.459 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.459 DEBUG:boto:Data: 2022-09-13T07:41:29.460 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.460 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.461 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.461 DEBUG:boto:Params: {} 2022-09-13T07:41:29.462 DEBUG:boto:Token: None 2022-09-13T07:41:29.462 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/?versions 2022-09-13T07:41:29.463 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs= 2022-09-13T07:41:29.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs=', 'Content-Length': '0'} 2022-09-13T07:41:29.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcfeda408b5dedc9-0063203429-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.510 DEBUG:boto:b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:29.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:29.514 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.514 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.515 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.516 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.516 DEBUG:boto:Method: GET 2022-09-13T07:41:29.517 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.517 DEBUG:boto:Data: 2022-09-13T07:41:29.518 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.518 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.519 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.519 DEBUG:boto:Params: {} 2022-09-13T07:41:29.520 DEBUG:boto:Token: None 2022-09-13T07:41:29.521 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.521 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T07:41:29.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T07:41:29.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000048246dad7ea8cd3f-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.525 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.525 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.526 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.526 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.527 DEBUG:boto:Method: GET 2022-09-13T07:41:29.528 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.528 DEBUG:boto:Data: 2022-09-13T07:41:29.529 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.529 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.530 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.530 DEBUG:boto:Params: {} 2022-09-13T07:41:29.531 DEBUG:boto:Token: None 2022-09-13T07:41:29.532 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.532 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T07:41:29.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T07:41:29.535 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000006d4b220d0a5fea-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.536 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.536 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.537 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.537 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.538 DEBUG:boto:Method: HEAD 2022-09-13T07:41:29.539 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.539 DEBUG:boto:Data: 2022-09-13T07:41:29.540 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.540 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.541 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.541 DEBUG:boto:Params: {} 2022-09-13T07:41:29.542 DEBUG:boto:Token: None 2022-09-13T07:41:29.542 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.543 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5cGa9OcM8H994XOwy0zwsP56cW4= 2022-09-13T07:41:29.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5cGa9OcM8H994XOwy0zwsP56cW4=', 'Content-Length': '0'} 2022-09-13T07:41:29.545 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a8aeedbc4650310-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.546 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.547 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.547 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.548 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.548 DEBUG:boto:Method: HEAD 2022-09-13T07:41:29.549 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.549 DEBUG:boto:Data: 2022-09-13T07:41:29.550 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.550 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.551 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.551 DEBUG:boto:Params: {} 2022-09-13T07:41:29.552 DEBUG:boto:Token: None 2022-09-13T07:41:29.553 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.553 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5cGa9OcM8H994XOwy0zwsP56cW4= 2022-09-13T07:41:29.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5cGa9OcM8H994XOwy0zwsP56cW4=', 'Content-Length': '0'} 2022-09-13T07:41:29.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f78e302e51c58901-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:29.559 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.559 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.560 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.560 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.561 DEBUG:boto:Method: GET 2022-09-13T07:41:29.562 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.563 DEBUG:boto:Data: 2022-09-13T07:41:29.564 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.565 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.566 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.567 DEBUG:boto:Params: {} 2022-09-13T07:41:29.568 DEBUG:boto:Token: None 2022-09-13T07:41:29.569 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.570 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T07:41:29.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T07:41:29.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec2038f43e7916de-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.574 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.575 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.575 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.576 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.576 DEBUG:boto:Method: GET 2022-09-13T07:41:29.577 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.577 DEBUG:boto:Data: 2022-09-13T07:41:29.578 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.578 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.579 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.579 DEBUG:boto:Params: {} 2022-09-13T07:41:29.580 DEBUG:boto:Token: None 2022-09-13T07:41:29.581 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.581 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T07:41:29.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T07:41:29.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000243ec6f399f4c55c-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.584 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.585 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.586 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.586 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.587 DEBUG:boto:Method: GET 2022-09-13T07:41:29.587 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.588 DEBUG:boto:Data: 2022-09-13T07:41:29.588 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.589 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.589 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.590 DEBUG:boto:Params: {} 2022-09-13T07:41:29.591 DEBUG:boto:Token: None 2022-09-13T07:41:29.591 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/ 2022-09-13T07:41:29.592 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T07:41:29.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T07:41:29.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4a2ffbb3ffdf73d-0063203429-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.596 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:29.597 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.597 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.598 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.598 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.599 DEBUG:boto:Method: GET 2022-09-13T07:41:29.599 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.600 DEBUG:boto:Data: 2022-09-13T07:41:29.601 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.601 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.602 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.602 DEBUG:boto:Params: {} 2022-09-13T07:41:29.603 DEBUG:boto:Token: None 2022-09-13T07:41:29.603 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/ 2022-09-13T07:41:29.604 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T07:41:29.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T07:41:29.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bc188a8eee0498f-0063203429-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.608 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:29.609 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.609 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.610 DEBUG:boto:Method: HEAD 2022-09-13T07:41:29.610 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:41:29.611 DEBUG:boto:Data: 2022-09-13T07:41:29.611 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.612 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.612 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.613 DEBUG:boto:Params: {} 2022-09-13T07:41:29.614 DEBUG:boto:Token: None 2022-09-13T07:41:29.614 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj 2022-09-13T07:41:29.615 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64= 2022-09-13T07:41:29.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64=', 'Content-Length': '0'} 2022-09-13T07:41:29.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000034c51afe1dca80d8-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.619 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.620 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.621 DEBUG:boto:Method: HEAD 2022-09-13T07:41:29.622 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:41:29.623 DEBUG:boto:Data: 2022-09-13T07:41:29.624 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.625 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.625 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.626 DEBUG:boto:Params: {} 2022-09-13T07:41:29.627 DEBUG:boto:Token: None 2022-09-13T07:41:29.628 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj 2022-09-13T07:41:29.629 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64= 2022-09-13T07:41:29.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64=', 'Content-Length': '0'} 2022-09-13T07:41:29.632 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000adaf8a8808a462a8-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:29.633 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.634 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.634 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.635 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.635 DEBUG:boto:Method: GET 2022-09-13T07:41:29.636 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.637 DEBUG:boto:Data: 2022-09-13T07:41:29.637 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.638 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.638 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.639 DEBUG:boto:Params: {} 2022-09-13T07:41:29.639 DEBUG:boto:Token: None 2022-09-13T07:41:29.640 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.641 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T07:41:29.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T07:41:29.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c12bce474144877-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.645 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.646 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.647 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.648 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.649 DEBUG:boto:Method: GET 2022-09-13T07:41:29.650 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.651 DEBUG:boto:Data: 2022-09-13T07:41:29.652 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.652 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.653 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.653 DEBUG:boto:Params: {} 2022-09-13T07:41:29.654 DEBUG:boto:Token: None 2022-09-13T07:41:29.655 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.655 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T07:41:29.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T07:41:29.658 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080a788896b4de21b-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.659 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.660 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.660 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.661 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.661 DEBUG:boto:Method: GET 2022-09-13T07:41:29.662 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.663 DEBUG:boto:Data: 2022-09-13T07:41:29.663 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.664 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.664 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.665 DEBUG:boto:Params: {} 2022-09-13T07:41:29.666 DEBUG:boto:Token: None 2022-09-13T07:41:29.666 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/ 2022-09-13T07:41:29.667 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T07:41:29.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T07:41:29.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c26f0c4c87680b8a-0063203429-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.714 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:29.716 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.717 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.718 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.718 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.719 DEBUG:boto:Method: GET 2022-09-13T07:41:29.721 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.722 DEBUG:boto:Data: 2022-09-13T07:41:29.723 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.724 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.725 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.726 DEBUG:boto:Params: {} 2022-09-13T07:41:29.727 DEBUG:boto:Token: None 2022-09-13T07:41:29.728 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/ 2022-09-13T07:41:29.729 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T07:41:29.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T07:41:29.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce4f18181c34a30b-0063203429-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.735 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:29.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:29.737 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.738 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.739 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.739 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.740 DEBUG:boto:Method: GET 2022-09-13T07:41:29.741 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.742 DEBUG:boto:Data: 2022-09-13T07:41:29.743 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.744 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.745 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.746 DEBUG:boto:Params: {} 2022-09-13T07:41:29.747 DEBUG:boto:Token: None 2022-09-13T07:41:29.748 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.749 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc= 2022-09-13T07:41:29.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc=', 'Content-Length': '0'} 2022-09-13T07:41:29.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000078adb442c5762865-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.754 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.754 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.755 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.756 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.757 DEBUG:boto:Method: GET 2022-09-13T07:41:29.758 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.759 DEBUG:boto:Data: 2022-09-13T07:41:29.759 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.760 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.761 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.762 DEBUG:boto:Params: {} 2022-09-13T07:41:29.763 DEBUG:boto:Token: None 2022-09-13T07:41:29.764 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.765 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc= 2022-09-13T07:41:29.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc=', 'Content-Length': '0'} 2022-09-13T07:41:29.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7224b985c9c6127-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.771 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.772 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.773 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.774 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.775 DEBUG:boto:Method: HEAD 2022-09-13T07:41:29.776 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.776 DEBUG:boto:Data: 2022-09-13T07:41:29.777 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.778 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.779 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.780 DEBUG:boto:Params: {} 2022-09-13T07:41:29.781 DEBUG:boto:Token: None 2022-09-13T07:41:29.782 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.783 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pOs6vzl5rgaaVjpZ3US0Zl0YoGM= 2022-09-13T07:41:29.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pOs6vzl5rgaaVjpZ3US0Zl0YoGM=', 'Content-Length': '0'} 2022-09-13T07:41:29.787 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001571641238fcbc7e-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.787 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.788 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.788 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.789 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.789 DEBUG:boto:Method: HEAD 2022-09-13T07:41:29.790 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.791 DEBUG:boto:Data: 2022-09-13T07:41:29.791 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.792 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.792 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.793 DEBUG:boto:Params: {} 2022-09-13T07:41:29.793 DEBUG:boto:Token: None 2022-09-13T07:41:29.794 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.794 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pOs6vzl5rgaaVjpZ3US0Zl0YoGM= 2022-09-13T07:41:29.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pOs6vzl5rgaaVjpZ3US0Zl0YoGM=', 'Content-Length': '0'} 2022-09-13T07:41:29.797 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007ee829837c5c14ed-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:29.798 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.799 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.799 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.800 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.800 DEBUG:boto:Method: GET 2022-09-13T07:41:29.801 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.801 DEBUG:boto:Data: 2022-09-13T07:41:29.802 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.802 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.803 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.803 DEBUG:boto:Params: {} 2022-09-13T07:41:29.804 DEBUG:boto:Token: None 2022-09-13T07:41:29.805 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.805 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc= 2022-09-13T07:41:29.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc=', 'Content-Length': '0'} 2022-09-13T07:41:29.808 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b80e2a64f3ccba2d-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.809 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.809 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.810 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.810 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.811 DEBUG:boto:Method: GET 2022-09-13T07:41:29.811 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.812 DEBUG:boto:Data: 2022-09-13T07:41:29.812 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.813 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.813 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.814 DEBUG:boto:Params: {} 2022-09-13T07:41:29.815 DEBUG:boto:Token: None 2022-09-13T07:41:29.815 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:29.816 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc= 2022-09-13T07:41:29.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc=', 'Content-Length': '0'} 2022-09-13T07:41:29.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000078405db93292c7b6-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.819 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.820 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.820 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.821 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.821 DEBUG:boto:Method: GET 2022-09-13T07:41:29.822 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.822 DEBUG:boto:Data: 2022-09-13T07:41:29.823 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.823 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.824 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.824 DEBUG:boto:Params: {} 2022-09-13T07:41:29.825 DEBUG:boto:Token: None 2022-09-13T07:41:29.826 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/ 2022-09-13T07:41:29.826 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T07:41:29.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T07:41:29.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd7e63e1f3a95b7b-0063203429-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.830 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:29.831 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.831 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.832 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.832 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.833 DEBUG:boto:Method: GET 2022-09-13T07:41:29.833 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.834 DEBUG:boto:Data: 2022-09-13T07:41:29.834 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.835 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.835 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.836 DEBUG:boto:Params: {} 2022-09-13T07:41:29.837 DEBUG:boto:Token: None 2022-09-13T07:41:29.837 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/ 2022-09-13T07:41:29.838 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T07:41:29.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T07:41:29.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000115acefa7c2a575c-0063203429-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.842 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:29.843 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.843 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.844 DEBUG:boto:Method: HEAD 2022-09-13T07:41:29.844 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:41:29.845 DEBUG:boto:Data: 2022-09-13T07:41:29.845 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.846 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.846 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.847 DEBUG:boto:Params: {} 2022-09-13T07:41:29.847 DEBUG:boto:Token: None 2022-09-13T07:41:29.848 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj 2022-09-13T07:41:29.849 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64= 2022-09-13T07:41:29.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64=', 'Content-Length': '0'} 2022-09-13T07:41:29.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba6dea8da2cf4377-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.852 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.853 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.853 DEBUG:boto:Method: HEAD 2022-09-13T07:41:29.854 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:41:29.854 DEBUG:boto:Data: 2022-09-13T07:41:29.855 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.855 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.856 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.856 DEBUG:boto:Params: {} 2022-09-13T07:41:29.857 DEBUG:boto:Token: None 2022-09-13T07:41:29.857 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj 2022-09-13T07:41:29.858 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64= 2022-09-13T07:41:29.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64=', 'Content-Length': '0'} 2022-09-13T07:41:29.861 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e33a4cf0843d0a3b-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:29.862 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.863 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.863 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.864 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.864 DEBUG:boto:Method: GET 2022-09-13T07:41:29.865 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.865 DEBUG:boto:Data: 2022-09-13T07:41:29.866 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.866 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.867 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.867 DEBUG:boto:Params: {} 2022-09-13T07:41:29.868 DEBUG:boto:Token: None 2022-09-13T07:41:29.869 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.869 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T07:41:29.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T07:41:29.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a344a338515f8ad8-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.873 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:29.873 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:29.874 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.874 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.875 DEBUG:boto:Method: GET 2022-09-13T07:41:29.875 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.876 DEBUG:boto:Data: 2022-09-13T07:41:29.876 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.877 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.877 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.878 DEBUG:boto:Params: {} 2022-09-13T07:41:29.878 DEBUG:boto:Token: None 2022-09-13T07:41:29.879 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:29.880 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T07:41:29.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T07:41:29.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000de4231781e766acd-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.883 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.883 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.884 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.884 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.885 DEBUG:boto:Method: GET 2022-09-13T07:41:29.886 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.886 DEBUG:boto:Data: 2022-09-13T07:41:29.887 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.887 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:29.888 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.888 DEBUG:boto:Params: {} 2022-09-13T07:41:29.889 DEBUG:boto:Token: None 2022-09-13T07:41:29.889 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/ 2022-09-13T07:41:29.890 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T07:41:29.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T07:41:29.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008682c58b4ce590b1-0063203429-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.894 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:29.895 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.895 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.896 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.896 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.897 DEBUG:boto:Method: GET 2022-09-13T07:41:29.897 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:29.898 DEBUG:boto:Data: 2022-09-13T07:41:29.899 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.899 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.900 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.900 DEBUG:boto:Params: {} 2022-09-13T07:41:29.901 DEBUG:boto:Token: None 2022-09-13T07:41:29.901 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/ 2022-09-13T07:41:29.902 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T07:41:29.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T07:41:29.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001671d21f3b6d3447-0063203429-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.906 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:29.907 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-20 zones={test-zone2, test-zone1} 2022-09-13T07:41:29.907 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-20 zones={test-zone3, test-zone1} 2022-09-13T07:41:29.908 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.908 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.909 DEBUG:boto:Method: HEAD 2022-09-13T07:41:29.909 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:41:29.910 DEBUG:boto:Data: 2022-09-13T07:41:29.910 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.911 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:29.911 DEBUG:boto:Port: 8001 2022-09-13T07:41:29.912 DEBUG:boto:Params: {} 2022-09-13T07:41:29.912 DEBUG:boto:Token: None 2022-09-13T07:41:29.913 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/ 2022-09-13T07:41:29.913 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo= 2022-09-13T07:41:29.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo=', 'Content-Length': '0'} 2022-09-13T07:41:29.917 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e1385daaf252966-0063203429-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.918 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.918 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.919 DEBUG:boto:Method: HEAD 2022-09-13T07:41:29.920 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:41:29.920 DEBUG:boto:Data: 2022-09-13T07:41:29.921 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.921 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.922 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.922 DEBUG:boto:Params: {} 2022-09-13T07:41:29.923 DEBUG:boto:Token: None 2022-09-13T07:41:29.923 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/ 2022-09-13T07:41:29.924 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo= 2022-09-13T07:41:29.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo=', 'Content-Length': '0'} 2022-09-13T07:41:29.927 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045c821901507b1cc-0063203429-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.928 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:41:29.928 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.929 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.929 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.930 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.930 DEBUG:boto:Method: GET 2022-09-13T07:41:29.931 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.931 DEBUG:boto:Data: 2022-09-13T07:41:29.932 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.932 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:29.933 DEBUG:boto:Port: 8001 2022-09-13T07:41:29.933 DEBUG:boto:Params: {} 2022-09-13T07:41:29.934 DEBUG:boto:Token: None 2022-09-13T07:41:29.935 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/?versions 2022-09-13T07:41:29.935 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs= 2022-09-13T07:41:29.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs=', 'Content-Length': '0'} 2022-09-13T07:41:29.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023ba5a6f30259907-0063203429-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.940 DEBUG:boto:b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:29.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:41:29.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:41:29.942 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:41:29.942 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.943 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.943 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.944 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.944 DEBUG:boto:Method: GET 2022-09-13T07:41:29.945 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.945 DEBUG:boto:Data: 2022-09-13T07:41:29.946 DEBUG:boto:Headers: {} 2022-09-13T07:41:29.946 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:29.947 DEBUG:boto:Port: 8000 2022-09-13T07:41:29.947 DEBUG:boto:Params: {} 2022-09-13T07:41:29.948 DEBUG:boto:Token: None 2022-09-13T07:41:29.949 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:29 GMT /yxvvuh-20/?versions 2022-09-13T07:41:29.949 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs= 2022-09-13T07:41:29.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs=', 'Content-Length': '0'} 2022-09-13T07:41:29.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f39c358f5ab612e-0063203429-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:29.994 DEBUG:boto:b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:29.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:41:29.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:41:29.997 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:29.998 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:29.998 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.999 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:29.999 DEBUG:boto:Method: GET 2022-09-13T07:41:30.000 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.000 DEBUG:boto:Data: 2022-09-13T07:41:30.001 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.002 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.002 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.003 DEBUG:boto:Params: {} 2022-09-13T07:41:30.003 DEBUG:boto:Token: None 2022-09-13T07:41:30.004 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/?versions 2022-09-13T07:41:30.005 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU= 2022-09-13T07:41:30.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU=', 'Content-Length': '0'} 2022-09-13T07:41:30.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089c41f4faeb91455-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.009 DEBUG:boto:b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.009 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.010 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.010 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.011 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.011 DEBUG:boto:Method: GET 2022-09-13T07:41:30.012 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.012 DEBUG:boto:Data: 2022-09-13T07:41:30.013 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.013 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.014 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.015 DEBUG:boto:Params: {} 2022-09-13T07:41:30.015 DEBUG:boto:Token: None 2022-09-13T07:41:30.016 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/?versions 2022-09-13T07:41:30.016 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU= 2022-09-13T07:41:30.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU=', 'Content-Length': '0'} 2022-09-13T07:41:30.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cfcb21c90326de5-006320342a-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.020 DEBUG:boto:b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:30.022 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.022 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.023 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.023 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.024 DEBUG:boto:Method: GET 2022-09-13T07:41:30.024 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.025 DEBUG:boto:Data: 2022-09-13T07:41:30.025 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.026 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.026 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.027 DEBUG:boto:Params: {} 2022-09-13T07:41:30.028 DEBUG:boto:Token: None 2022-09-13T07:41:30.028 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.029 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.031 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036d676aa98b84854-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.032 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.033 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.033 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.034 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.034 DEBUG:boto:Method: GET 2022-09-13T07:41:30.035 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.035 DEBUG:boto:Data: 2022-09-13T07:41:30.036 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.036 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.037 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.037 DEBUG:boto:Params: {} 2022-09-13T07:41:30.038 DEBUG:boto:Token: None 2022-09-13T07:41:30.038 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.039 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001536bb4d23aae2aa-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.042 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.043 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.043 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.044 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.044 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.045 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.045 DEBUG:boto:Data: 2022-09-13T07:41:30.046 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.046 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.047 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.047 DEBUG:boto:Params: {} 2022-09-13T07:41:30.048 DEBUG:boto:Token: None 2022-09-13T07:41:30.048 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.049 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I= 2022-09-13T07:41:30.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I=', 'Content-Length': '0'} 2022-09-13T07:41:30.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005577e6148f665db0-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.052 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.053 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.053 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.054 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.054 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.055 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.055 DEBUG:boto:Data: 2022-09-13T07:41:30.056 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.056 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.057 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.057 DEBUG:boto:Params: {} 2022-09-13T07:41:30.058 DEBUG:boto:Token: None 2022-09-13T07:41:30.058 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.059 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I= 2022-09-13T07:41:30.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I=', 'Content-Length': '0'} 2022-09-13T07:41:30.061 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fc428cb979436f02-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:30.063 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.063 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.064 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.064 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.065 DEBUG:boto:Method: GET 2022-09-13T07:41:30.065 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.066 DEBUG:boto:Data: 2022-09-13T07:41:30.066 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.067 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.067 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.068 DEBUG:boto:Params: {} 2022-09-13T07:41:30.068 DEBUG:boto:Token: None 2022-09-13T07:41:30.069 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.070 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.072 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d2adace9c5b3223b-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.073 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.073 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.074 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.074 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.075 DEBUG:boto:Method: GET 2022-09-13T07:41:30.075 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.076 DEBUG:boto:Data: 2022-09-13T07:41:30.076 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.077 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.077 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.078 DEBUG:boto:Params: {} 2022-09-13T07:41:30.079 DEBUG:boto:Token: None 2022-09-13T07:41:30.079 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.080 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3efa51f24c75db7-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.083 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.084 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.084 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.085 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.085 DEBUG:boto:Method: GET 2022-09-13T07:41:30.086 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.086 DEBUG:boto:Data: 2022-09-13T07:41:30.087 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.087 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.088 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.089 DEBUG:boto:Params: {} 2022-09-13T07:41:30.089 DEBUG:boto:Token: None 2022-09-13T07:41:30.090 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.090 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020c794b59aca81e2-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.094 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.095 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.096 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.096 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.097 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.098 DEBUG:boto:Method: GET 2022-09-13T07:41:30.098 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.099 DEBUG:boto:Data: 2022-09-13T07:41:30.100 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.100 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.101 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.101 DEBUG:boto:Params: {} 2022-09-13T07:41:30.102 DEBUG:boto:Token: None 2022-09-13T07:41:30.102 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.103 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052170e7d071581ee-006320342a-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.107 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.108 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.108 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.109 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.109 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:41:30.110 DEBUG:boto:Data: 2022-09-13T07:41:30.110 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.111 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.111 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.112 DEBUG:boto:Params: {} 2022-09-13T07:41:30.112 DEBUG:boto:Token: None 2022-09-13T07:41:30.113 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj 2022-09-13T07:41:30.113 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T07:41:30.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T07:41:30.117 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000558a6fe0914596bb-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.117 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.118 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.118 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.119 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:41:30.119 DEBUG:boto:Data: 2022-09-13T07:41:30.120 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.120 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.121 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.121 DEBUG:boto:Params: {} 2022-09-13T07:41:30.122 DEBUG:boto:Token: None 2022-09-13T07:41:30.122 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj 2022-09-13T07:41:30.123 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T07:41:30.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T07:41:30.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000396a9dcda792c5bd-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:30.127 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.127 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.128 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.128 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.129 DEBUG:boto:Method: GET 2022-09-13T07:41:30.129 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.130 DEBUG:boto:Data: 2022-09-13T07:41:30.130 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.131 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.131 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.132 DEBUG:boto:Params: {} 2022-09-13T07:41:30.132 DEBUG:boto:Token: None 2022-09-13T07:41:30.133 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.134 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000945d43e79cb77b24-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.137 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.138 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.138 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.139 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.139 DEBUG:boto:Method: GET 2022-09-13T07:41:30.140 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.140 DEBUG:boto:Data: 2022-09-13T07:41:30.141 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.141 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.142 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.142 DEBUG:boto:Params: {} 2022-09-13T07:41:30.143 DEBUG:boto:Token: None 2022-09-13T07:41:30.143 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.144 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.147 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bc460de5455f4326-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.147 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.148 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.148 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.149 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.149 DEBUG:boto:Method: GET 2022-09-13T07:41:30.150 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.150 DEBUG:boto:Data: 2022-09-13T07:41:30.151 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.151 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.152 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.152 DEBUG:boto:Params: {} 2022-09-13T07:41:30.153 DEBUG:boto:Token: None 2022-09-13T07:41:30.153 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.154 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd24cbbb80e7ac7e-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.158 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.159 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.159 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.160 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.160 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.161 DEBUG:boto:Method: GET 2022-09-13T07:41:30.161 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.162 DEBUG:boto:Data: 2022-09-13T07:41:30.162 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.163 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.163 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.164 DEBUG:boto:Params: {} 2022-09-13T07:41:30.164 DEBUG:boto:Token: None 2022-09-13T07:41:30.165 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.165 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba38aa67d3974644-006320342a-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.210 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:30.211 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.212 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.212 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.213 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.213 DEBUG:boto:Method: GET 2022-09-13T07:41:30.214 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.214 DEBUG:boto:Data: 2022-09-13T07:41:30.215 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.215 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.216 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.216 DEBUG:boto:Params: {} 2022-09-13T07:41:30.217 DEBUG:boto:Token: None 2022-09-13T07:41:30.218 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.218 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T07:41:30.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T07:41:30.221 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003559b7a554e9805c-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.222 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.222 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.223 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.223 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.224 DEBUG:boto:Method: GET 2022-09-13T07:41:30.224 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.225 DEBUG:boto:Data: 2022-09-13T07:41:30.225 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.226 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.226 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.227 DEBUG:boto:Params: {} 2022-09-13T07:41:30.227 DEBUG:boto:Token: None 2022-09-13T07:41:30.228 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.228 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T07:41:30.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T07:41:30.231 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006e838c5d42dbe37b-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.232 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.232 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.233 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.233 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.234 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.234 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.235 DEBUG:boto:Data: 2022-09-13T07:41:30.235 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.235 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.236 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.236 DEBUG:boto:Params: {} 2022-09-13T07:41:30.237 DEBUG:boto:Token: None 2022-09-13T07:41:30.237 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.238 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M= 2022-09-13T07:41:30.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M=', 'Content-Length': '0'} 2022-09-13T07:41:30.241 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000031a17e0e7c3091e5-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.241 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.242 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.242 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.243 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.243 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.244 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.244 DEBUG:boto:Data: 2022-09-13T07:41:30.245 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.245 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.246 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.246 DEBUG:boto:Params: {} 2022-09-13T07:41:30.247 DEBUG:boto:Token: None 2022-09-13T07:41:30.247 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.248 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M= 2022-09-13T07:41:30.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M=', 'Content-Length': '0'} 2022-09-13T07:41:30.251 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d4f6796a2f640338-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:30.252 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.252 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.253 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.253 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.254 DEBUG:boto:Method: GET 2022-09-13T07:41:30.254 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.255 DEBUG:boto:Data: 2022-09-13T07:41:30.255 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.256 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.256 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.257 DEBUG:boto:Params: {} 2022-09-13T07:41:30.257 DEBUG:boto:Token: None 2022-09-13T07:41:30.258 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.259 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T07:41:30.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T07:41:30.261 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000716166009d5b7a2f-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.262 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.262 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.263 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.263 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.264 DEBUG:boto:Method: GET 2022-09-13T07:41:30.264 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.265 DEBUG:boto:Data: 2022-09-13T07:41:30.265 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.266 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.266 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.267 DEBUG:boto:Params: {} 2022-09-13T07:41:30.268 DEBUG:boto:Token: None 2022-09-13T07:41:30.268 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.269 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T07:41:30.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T07:41:30.271 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dbdd6c77dc8634b3-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.272 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.273 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.273 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.274 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.274 DEBUG:boto:Method: GET 2022-09-13T07:41:30.275 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.275 DEBUG:boto:Data: 2022-09-13T07:41:30.276 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.276 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.276 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.277 DEBUG:boto:Params: {} 2022-09-13T07:41:30.278 DEBUG:boto:Token: None 2022-09-13T07:41:30.278 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.279 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdf2b06eb872ef2d-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.282 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.283 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.284 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.284 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.285 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.285 DEBUG:boto:Method: GET 2022-09-13T07:41:30.286 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.286 DEBUG:boto:Data: 2022-09-13T07:41:30.287 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.287 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.288 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.288 DEBUG:boto:Params: {} 2022-09-13T07:41:30.289 DEBUG:boto:Token: None 2022-09-13T07:41:30.289 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.290 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e694b7fdc127a66d-006320342a-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.294 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.295 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.295 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.296 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.296 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:41:30.296 DEBUG:boto:Data: 2022-09-13T07:41:30.297 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.297 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.298 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.298 DEBUG:boto:Params: {} 2022-09-13T07:41:30.299 DEBUG:boto:Token: None 2022-09-13T07:41:30.299 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj 2022-09-13T07:41:30.300 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T07:41:30.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T07:41:30.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004d3e91f658c8aa95-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.304 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.304 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.305 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.305 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:41:30.306 DEBUG:boto:Data: 2022-09-13T07:41:30.306 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.307 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.307 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.308 DEBUG:boto:Params: {} 2022-09-13T07:41:30.308 DEBUG:boto:Token: None 2022-09-13T07:41:30.309 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj 2022-09-13T07:41:30.309 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T07:41:30.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T07:41:30.313 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dc18f588e4df86da-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:30.314 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.315 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.315 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.316 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.317 DEBUG:boto:Method: GET 2022-09-13T07:41:30.318 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.319 DEBUG:boto:Data: 2022-09-13T07:41:30.320 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.320 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.321 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.322 DEBUG:boto:Params: {} 2022-09-13T07:41:30.323 DEBUG:boto:Token: None 2022-09-13T07:41:30.324 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.325 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006a3cad874257f33c-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.329 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.330 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.331 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.332 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.333 DEBUG:boto:Method: GET 2022-09-13T07:41:30.334 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.334 DEBUG:boto:Data: 2022-09-13T07:41:30.335 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.336 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.337 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.338 DEBUG:boto:Params: {} 2022-09-13T07:41:30.339 DEBUG:boto:Token: None 2022-09-13T07:41:30.340 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.341 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.344 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005da626c317dea5ab-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.345 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.345 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.346 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.346 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.347 DEBUG:boto:Method: GET 2022-09-13T07:41:30.348 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.348 DEBUG:boto:Data: 2022-09-13T07:41:30.349 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.350 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.350 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.351 DEBUG:boto:Params: {} 2022-09-13T07:41:30.351 DEBUG:boto:Token: None 2022-09-13T07:41:30.352 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.353 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eb2c079582a30c2-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.402 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.403 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.404 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.405 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.405 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.406 DEBUG:boto:Method: GET 2022-09-13T07:41:30.406 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.407 DEBUG:boto:Data: 2022-09-13T07:41:30.407 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.408 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.408 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.409 DEBUG:boto:Params: {} 2022-09-13T07:41:30.410 DEBUG:boto:Token: None 2022-09-13T07:41:30.410 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.411 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023b5a17ac42304d5-006320342a-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.415 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.416 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-20 zones={test-zone3, test-zone1} 2022-09-13T07:41:30.417 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-20 zones={test-zone3, test-zone2} 2022-09-13T07:41:30.417 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.418 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.418 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.419 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:41:30.420 DEBUG:boto:Data: 2022-09-13T07:41:30.420 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.421 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.421 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.422 DEBUG:boto:Params: {} 2022-09-13T07:41:30.422 DEBUG:boto:Token: None 2022-09-13T07:41:30.423 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.423 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cfkVsbBaPWLCGHlKc9e3SyLA+dA= 2022-09-13T07:41:30.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cfkVsbBaPWLCGHlKc9e3SyLA+dA=', 'Content-Length': '0'} 2022-09-13T07:41:30.427 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e073bf541c70a73-006320342a-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.427 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.428 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.429 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.429 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:41:30.430 DEBUG:boto:Data: 2022-09-13T07:41:30.430 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.431 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.431 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.432 DEBUG:boto:Params: {} 2022-09-13T07:41:30.432 DEBUG:boto:Token: None 2022-09-13T07:41:30.433 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.433 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:cfkVsbBaPWLCGHlKc9e3SyLA+dA= 2022-09-13T07:41:30.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cfkVsbBaPWLCGHlKc9e3SyLA+dA=', 'Content-Length': '0'} 2022-09-13T07:41:30.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e5b661bcbada1f2-006320342a-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.437 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:41:30.438 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.438 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.439 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.440 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.440 DEBUG:boto:Method: GET 2022-09-13T07:41:30.441 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.441 DEBUG:boto:Data: 2022-09-13T07:41:30.442 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.442 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.443 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.443 DEBUG:boto:Params: {} 2022-09-13T07:41:30.444 DEBUG:boto:Token: None 2022-09-13T07:41:30.444 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/?versions 2022-09-13T07:41:30.445 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU= 2022-09-13T07:41:30.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU=', 'Content-Length': '0'} 2022-09-13T07:41:30.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1505b70c007b2ac-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.449 DEBUG:boto:b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:41:30.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:41:30.451 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:41:30.452 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.452 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.453 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.453 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.454 DEBUG:boto:Method: GET 2022-09-13T07:41:30.454 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.455 DEBUG:boto:Data: 2022-09-13T07:41:30.456 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.456 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.457 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.457 DEBUG:boto:Params: {} 2022-09-13T07:41:30.458 DEBUG:boto:Token: None 2022-09-13T07:41:30.458 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/?versions 2022-09-13T07:41:30.459 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU= 2022-09-13T07:41:30.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU=', 'Content-Length': '0'} 2022-09-13T07:41:30.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d99f48f46a63a3e3-006320342a-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.463 DEBUG:boto:b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:41:30.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:41:30.464 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.465 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.466 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.466 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.467 DEBUG:boto:Method: GET 2022-09-13T07:41:30.467 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.468 DEBUG:boto:Data: 2022-09-13T07:41:30.468 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.469 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.469 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.470 DEBUG:boto:Params: {} 2022-09-13T07:41:30.470 DEBUG:boto:Token: None 2022-09-13T07:41:30.471 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/?versions 2022-09-13T07:41:30.472 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU= 2022-09-13T07:41:30.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU=', 'Content-Length': '0'} 2022-09-13T07:41:30.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005107f9ba0119b77a-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.475 DEBUG:boto:b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.476 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.477 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.478 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.478 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.479 DEBUG:boto:Method: GET 2022-09-13T07:41:30.479 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:41:30.480 DEBUG:boto:Data: 2022-09-13T07:41:30.480 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.481 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.481 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.482 DEBUG:boto:Params: {} 2022-09-13T07:41:30.482 DEBUG:boto:Token: None 2022-09-13T07:41:30.483 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/?versions 2022-09-13T07:41:30.483 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU= 2022-09-13T07:41:30.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU=', 'Content-Length': '0'} 2022-09-13T07:41:30.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004095cd4f535e0ead-006320342a-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.534 DEBUG:boto:b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:30.536 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.536 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.537 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.537 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.538 DEBUG:boto:Method: GET 2022-09-13T07:41:30.539 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.539 DEBUG:boto:Data: 2022-09-13T07:41:30.540 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.540 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.541 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.541 DEBUG:boto:Params: {} 2022-09-13T07:41:30.542 DEBUG:boto:Token: None 2022-09-13T07:41:30.542 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.543 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000823160c066ca8895-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.547 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.547 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.548 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.548 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.549 DEBUG:boto:Method: GET 2022-09-13T07:41:30.550 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.550 DEBUG:boto:Data: 2022-09-13T07:41:30.551 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.551 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.552 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.552 DEBUG:boto:Params: {} 2022-09-13T07:41:30.553 DEBUG:boto:Token: None 2022-09-13T07:41:30.554 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.554 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cf44f5863395efd3-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.558 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.558 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.559 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.559 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.560 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.560 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.561 DEBUG:boto:Data: 2022-09-13T07:41:30.561 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.562 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.562 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.563 DEBUG:boto:Params: {} 2022-09-13T07:41:30.564 DEBUG:boto:Token: None 2022-09-13T07:41:30.564 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.565 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I= 2022-09-13T07:41:30.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I=', 'Content-Length': '0'} 2022-09-13T07:41:30.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a7df7f3de4ba3b4e-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.568 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.569 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.569 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.570 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.570 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.571 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.571 DEBUG:boto:Data: 2022-09-13T07:41:30.572 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.572 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.573 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.573 DEBUG:boto:Params: {} 2022-09-13T07:41:30.574 DEBUG:boto:Token: None 2022-09-13T07:41:30.574 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.575 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I= 2022-09-13T07:41:30.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I=', 'Content-Length': '0'} 2022-09-13T07:41:30.577 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bb51d38c84ead971-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:30.579 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.580 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.580 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.581 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.582 DEBUG:boto:Method: GET 2022-09-13T07:41:30.582 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.583 DEBUG:boto:Data: 2022-09-13T07:41:30.584 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.584 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.585 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.585 DEBUG:boto:Params: {} 2022-09-13T07:41:30.586 DEBUG:boto:Token: None 2022-09-13T07:41:30.586 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.587 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.590 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009f17046fc3ef071a-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.591 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.591 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.592 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.593 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.593 DEBUG:boto:Method: GET 2022-09-13T07:41:30.594 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.595 DEBUG:boto:Data: 2022-09-13T07:41:30.595 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.596 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.596 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.597 DEBUG:boto:Params: {} 2022-09-13T07:41:30.597 DEBUG:boto:Token: None 2022-09-13T07:41:30.598 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.599 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000025c3b0170dad55ae-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.603 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.604 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.605 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.605 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.606 DEBUG:boto:Method: GET 2022-09-13T07:41:30.607 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.608 DEBUG:boto:Data: 2022-09-13T07:41:30.609 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.610 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.611 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.612 DEBUG:boto:Params: {} 2022-09-13T07:41:30.613 DEBUG:boto:Token: None 2022-09-13T07:41:30.614 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.615 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003131d164b68daa4e-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.662 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.663 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.664 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.664 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.665 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.665 DEBUG:boto:Method: GET 2022-09-13T07:41:30.666 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.666 DEBUG:boto:Data: 2022-09-13T07:41:30.667 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.668 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.668 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.669 DEBUG:boto:Params: {} 2022-09-13T07:41:30.670 DEBUG:boto:Token: None 2022-09-13T07:41:30.670 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.671 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000196e4ceb4b3f1a4f-006320342a-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.675 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.676 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.676 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.677 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.678 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:41:30.678 DEBUG:boto:Data: 2022-09-13T07:41:30.679 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.679 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.680 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.680 DEBUG:boto:Params: {} 2022-09-13T07:41:30.681 DEBUG:boto:Token: None 2022-09-13T07:41:30.681 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj 2022-09-13T07:41:30.682 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T07:41:30.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T07:41:30.685 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ce1639f6f2198a47-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.686 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.686 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.687 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.688 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:41:30.688 DEBUG:boto:Data: 2022-09-13T07:41:30.689 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.689 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.690 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.690 DEBUG:boto:Params: {} 2022-09-13T07:41:30.691 DEBUG:boto:Token: None 2022-09-13T07:41:30.691 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj 2022-09-13T07:41:30.692 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T07:41:30.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T07:41:30.695 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b98df274c61cc057-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:30.697 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.697 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.698 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.698 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.699 DEBUG:boto:Method: GET 2022-09-13T07:41:30.700 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.700 DEBUG:boto:Data: 2022-09-13T07:41:30.701 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.701 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.702 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.702 DEBUG:boto:Params: {} 2022-09-13T07:41:30.703 DEBUG:boto:Token: None 2022-09-13T07:41:30.703 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.704 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006764680cc4462b68-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.708 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.709 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.709 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.710 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.710 DEBUG:boto:Method: GET 2022-09-13T07:41:30.711 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.712 DEBUG:boto:Data: 2022-09-13T07:41:30.713 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.714 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.714 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.715 DEBUG:boto:Params: {} 2022-09-13T07:41:30.716 DEBUG:boto:Token: None 2022-09-13T07:41:30.716 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.717 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c355747b315d3de-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.721 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.721 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.722 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.722 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.723 DEBUG:boto:Method: GET 2022-09-13T07:41:30.724 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.724 DEBUG:boto:Data: 2022-09-13T07:41:30.725 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.725 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.726 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.726 DEBUG:boto:Params: {} 2022-09-13T07:41:30.727 DEBUG:boto:Token: None 2022-09-13T07:41:30.727 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.728 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c35235e3876a7ef-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.732 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.733 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.734 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.734 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.735 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.735 DEBUG:boto:Method: GET 2022-09-13T07:41:30.736 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.736 DEBUG:boto:Data: 2022-09-13T07:41:30.737 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.737 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.738 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.738 DEBUG:boto:Params: {} 2022-09-13T07:41:30.739 DEBUG:boto:Token: None 2022-09-13T07:41:30.739 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.740 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f2fd5155ddde9f1-006320342a-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.786 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:30.788 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.788 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.789 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.790 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.790 DEBUG:boto:Method: GET 2022-09-13T07:41:30.791 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.791 DEBUG:boto:Data: 2022-09-13T07:41:30.792 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.792 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.793 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.793 DEBUG:boto:Params: {} 2022-09-13T07:41:30.794 DEBUG:boto:Token: None 2022-09-13T07:41:30.795 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.796 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T07:41:30.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T07:41:30.800 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000074632187ded208f0-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.800 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.801 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.801 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.802 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.802 DEBUG:boto:Method: GET 2022-09-13T07:41:30.803 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.803 DEBUG:boto:Data: 2022-09-13T07:41:30.804 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.804 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.805 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.805 DEBUG:boto:Params: {} 2022-09-13T07:41:30.806 DEBUG:boto:Token: None 2022-09-13T07:41:30.807 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.807 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T07:41:30.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T07:41:30.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cccc5c7dc66a5755-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.811 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.811 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.812 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.812 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.812 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.813 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.813 DEBUG:boto:Data: 2022-09-13T07:41:30.814 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.814 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.815 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.815 DEBUG:boto:Params: {} 2022-09-13T07:41:30.816 DEBUG:boto:Token: None 2022-09-13T07:41:30.816 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.817 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M= 2022-09-13T07:41:30.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M=', 'Content-Length': '0'} 2022-09-13T07:41:30.819 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007b5e630975f84408-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.820 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.821 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.821 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.822 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.822 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.823 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.823 DEBUG:boto:Data: 2022-09-13T07:41:30.824 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.824 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.825 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.825 DEBUG:boto:Params: {} 2022-09-13T07:41:30.826 DEBUG:boto:Token: None 2022-09-13T07:41:30.826 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.827 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M= 2022-09-13T07:41:30.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M=', 'Content-Length': '0'} 2022-09-13T07:41:30.829 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d02dc8921ae2163a-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:30.831 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.831 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.832 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.832 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.833 DEBUG:boto:Method: GET 2022-09-13T07:41:30.833 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.834 DEBUG:boto:Data: 2022-09-13T07:41:30.834 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.835 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.835 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.836 DEBUG:boto:Params: {} 2022-09-13T07:41:30.836 DEBUG:boto:Token: None 2022-09-13T07:41:30.837 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.837 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T07:41:30.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T07:41:30.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000974b78775a785fc6-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.841 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.842 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.842 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.843 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.843 DEBUG:boto:Method: GET 2022-09-13T07:41:30.844 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.844 DEBUG:boto:Data: 2022-09-13T07:41:30.845 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.845 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.846 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.846 DEBUG:boto:Params: {} 2022-09-13T07:41:30.847 DEBUG:boto:Token: None 2022-09-13T07:41:30.847 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:41:30.848 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T07:41:30.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T07:41:30.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002ca3a0d5229a2da7-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.851 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.852 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.852 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.853 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.853 DEBUG:boto:Method: GET 2022-09-13T07:41:30.854 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.854 DEBUG:boto:Data: 2022-09-13T07:41:30.855 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.855 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.856 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.856 DEBUG:boto:Params: {} 2022-09-13T07:41:30.857 DEBUG:boto:Token: None 2022-09-13T07:41:30.857 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.858 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be846e69d6caf940-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.862 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.863 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.863 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.864 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.864 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.865 DEBUG:boto:Method: GET 2022-09-13T07:41:30.865 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.866 DEBUG:boto:Data: 2022-09-13T07:41:30.866 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.867 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.867 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.868 DEBUG:boto:Params: {} 2022-09-13T07:41:30.868 DEBUG:boto:Token: None 2022-09-13T07:41:30.869 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.869 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1a5ddba3a6b4ad2-006320342a-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.873 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.874 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.875 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.875 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.876 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:41:30.876 DEBUG:boto:Data: 2022-09-13T07:41:30.877 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.877 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.878 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.878 DEBUG:boto:Params: {} 2022-09-13T07:41:30.879 DEBUG:boto:Token: None 2022-09-13T07:41:30.879 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj 2022-09-13T07:41:30.880 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T07:41:30.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T07:41:30.883 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a44b9f5130653620-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.883 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.884 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.884 DEBUG:boto:Method: HEAD 2022-09-13T07:41:30.885 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:41:30.885 DEBUG:boto:Data: 2022-09-13T07:41:30.886 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.886 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.887 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.887 DEBUG:boto:Params: {} 2022-09-13T07:41:30.888 DEBUG:boto:Token: None 2022-09-13T07:41:30.888 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj 2022-09-13T07:41:30.889 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T07:41:30.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T07:41:30.892 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004fc460460c08fea6-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:41:30.893 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.894 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.894 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.895 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.895 DEBUG:boto:Method: GET 2022-09-13T07:41:30.896 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.896 DEBUG:boto:Data: 2022-09-13T07:41:30.897 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.897 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.898 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.898 DEBUG:boto:Params: {} 2022-09-13T07:41:30.899 DEBUG:boto:Token: None 2022-09-13T07:41:30.899 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.900 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000089df5bfe26a5ea62-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.903 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:30.904 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:30.904 DEBUG:boto:path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.905 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.905 DEBUG:boto:Method: GET 2022-09-13T07:41:30.906 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.906 DEBUG:boto:Data: 2022-09-13T07:41:30.907 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.908 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.908 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.909 DEBUG:boto:Params: {} 2022-09-13T07:41:30.909 DEBUG:boto:Token: None 2022-09-13T07:41:30.910 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T07:41:30.911 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T07:41:30.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T07:41:30.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e7e3add489db3a3-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.915 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.915 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.916 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.916 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.917 DEBUG:boto:Method: GET 2022-09-13T07:41:30.918 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.919 DEBUG:boto:Data: 2022-09-13T07:41:30.920 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.921 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:41:30.922 DEBUG:boto:Port: 8001 2022-09-13T07:41:30.922 DEBUG:boto:Params: {} 2022-09-13T07:41:30.923 DEBUG:boto:Token: None 2022-09-13T07:41:30.925 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.925 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd31c40f27b96a3c-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.974 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.975 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.976 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.976 DEBUG:boto:path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.977 DEBUG:boto:auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.977 DEBUG:boto:Method: GET 2022-09-13T07:41:30.978 DEBUG:boto:Path: /yxvvuh-20/?prefix=obj 2022-09-13T07:41:30.978 DEBUG:boto:Data: 2022-09-13T07:41:30.979 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.979 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:41:30.981 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.982 DEBUG:boto:Params: {} 2022-09-13T07:41:30.983 DEBUG:boto:Token: None 2022-09-13T07:41:30.984 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/ 2022-09-13T07:41:30.984 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T07:41:30.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T07:41:30.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7dd4170be753bfc-006320342a-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:30.989 DEBUG:boto:b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:41:30.990 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-20 zones={test-zone3, test-zone2} 2022-09-13T07:41:30.990 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:30.991 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:30.991 DEBUG:boto:path=/yxvvuh-20/?versioning 2022-09-13T07:41:30.992 DEBUG:boto:auth_path=/yxvvuh-20/?versioning 2022-09-13T07:41:30.992 DEBUG:boto:Method: PUT 2022-09-13T07:41:30.993 DEBUG:boto:Path: /yxvvuh-20/?versioning 2022-09-13T07:41:30.993 DEBUG:boto:Data: Suspended Disabled 2022-09-13T07:41:30.995 DEBUG:boto:Headers: {} 2022-09-13T07:41:30.995 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:30.996 DEBUG:boto:Port: 8000 2022-09-13T07:41:30.997 DEBUG:boto:Params: {} 2022-09-13T07:41:30.997 DEBUG:boto:Token: None 2022-09-13T07:41:30.998 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:41:30 GMT /yxvvuh-20/?versioning 2022-09-13T07:41:30.998 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rVm4VllGmmxpekZ1BWdIOIx3gqk= 2022-09-13T07:41:30.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rVm4VllGmmxpekZ1BWdIOIx3gqk=', 'Content-Length': '230'} 2022-09-13T07:41:31.005 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005cf9947917207d96-006320342a-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:31.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:41:31.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:41:31.009 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:41:31.066 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:41:31.066 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:41:31.109 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:41:31.125 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:41:31.125 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:41:31.125 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:41:31.126 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:41:31.126 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:41:31.126 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:41:31.126 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:41:31.127 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:41:31.127 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:41:31.127 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:41:31.128 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054891.002339_151.1", 2022-09-13T07:41:31.129 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:41:31.002339Z" 2022-09-13T07:41:31.129 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:41:31.129 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:41:31.129 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:41:31.130 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T07:41:31.130 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:41:31.130 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:41:31.145 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054891.002339_151.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:41:31.146 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:41:31.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:41:31.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:41:31.148 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:41:31.206 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:41:31.207 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:41:31.238 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:41:31.234+0000 7f5427eae600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:41:31.239 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:41:31.235+0000 7f5427eae600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:41:31.251 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:41:31.252 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:41:31.252 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:41:31.252 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:41:31.252 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:41:31.253 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:41:31.253 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:41:31.253 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:41:31.253 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:41:31.254 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:41:31.254 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:41:31.254 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:41:31.255 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:41:31.255 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:41:31.255 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:41:31.255 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:41:31.256 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:41:31.256 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:41:31.256 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:41:31.257 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:41:31.257 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:41:31.257 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:41:31.257 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:41:31.258 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:41:31.258 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:41:31.258 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:41:31.259 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:41:31.259 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:41:31.259 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:41:31.259 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:41:31.260 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:41:31.260 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:41:31.260 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:41:31.260 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:41:31.261 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:41:31.261 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:41:31.262 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:41:31.262 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054891.002339_151.1", 2022-09-13T07:41:31.262 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:41:31.263 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:41:31.263 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:41:31.263 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:41:31.002339Z", 2022-09-13T07:41:31.263 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:41:31.264 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:41:31.264 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:41:31.264 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:41:31.264 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:41:31.265 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:41:31.265 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:41:31.265 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:41:31.265 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:41:31.266 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:41:31.266 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:41:31.266 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:41:31.267 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:41:31.267 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:41:31.267 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:41:31.267 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:41:31.268 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:41:31.268 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:41:31.268 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:41:31.268 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:41:31.269 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:41:31.269 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:41:31Z" 2022-09-13T07:41:31.269 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:41:31.271 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054891.002339_151.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:41:31.002339Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:41:31Z" } 2022-09-13T07:41:31.271 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054891.002339_151.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:41:31.002339Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:41:31.272 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054891.002339_151.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:41:31.272 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054891.002339_151.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:41:31.273 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:41:31.273 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:41:31.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:41:31.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:41:31.275 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:41:31.329 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:41:31.329 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:41:31.359 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:31.355+0000 7fae2eab1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:41:31.360 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:31.356+0000 7fae2eab1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:41:31.371 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:41:31.372 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:41:31.372 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:41:31.372 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:41:31.373 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:41:31.373 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:41:31.373 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:41:31.374 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:41:31.374 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:41:31.374 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:41:31.374 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:41:31.375 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:41:31.375 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:41:31.375 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:41:31.375 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:41:31.376 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:41:31.376 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:41:31.376 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:41:31.376 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:41:31.377 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:41:31.377 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:41:31.377 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:41:31.377 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:41:31.378 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:41:31.378 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:41:31.378 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:41:31.379 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:41:31.379 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:41:31.379 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:41:31.379 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:41:31.380 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:41:31.380 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:41:31.380 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:41:31.380 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:41:31.381 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:41:31.381 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:41:31.381 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:41:31.381 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054890.998697_150.1", 2022-09-13T07:41:31.382 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:41:31.382 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:41:31.382 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:41:31.382 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:41:30.998697Z", 2022-09-13T07:41:31.383 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:41:31.383 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:41:31.383 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:41:31.383 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:41:31.384 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:41:31.384 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:41:31.384 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:41:31.384 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:41:31.385 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:41:31.385 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:41:31.385 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:41:31.385 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:41:31.386 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:41:31.386 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:41:31.386 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:41:31.386 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:41:31.387 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:41:31.387 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:41:31.387 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:41:31.387 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:41:31.388 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:41:31.388 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:41:31Z" 2022-09-13T07:41:31.388 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:41:31.389 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054890.998697_150.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:41:30.998697Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:41:31Z" } 2022-09-13T07:41:31.390 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054890.998697_150.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:41:30.998697Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:41:31.391 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054891.002339_151.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:41:31.391 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054890.998697_150.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:41:31.391 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1663054891.002339_151.1 target=1_1663054890.998697_150.1 2022-09-13T07:41:36.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:41:36.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:41:36.396 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:41:36.455 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:41:36.455 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:41:36.498 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:36.493+0000 7fe5b0d22600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:41:36.499 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:36.495+0000 7fe5b0d22600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:41:36.520 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:41:36.521 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:41:36.522 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:41:36.523 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:41:36.523 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:41:36.523 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:41:36.524 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:41:36.524 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:41:36.525 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:41:36.525 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:41:36.526 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:41:36.526 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:41:36.527 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:41:36.527 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:41:36.528 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:41:36.528 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:41:36.529 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:41:36.529 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:41:36.529 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:41:36.530 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:41:36.531 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:41:36.531 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:41:36.532 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:41:36.532 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:41:36.533 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:41:36.533 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:41:36.534 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:41:36.534 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:41:36.534 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:41:36.535 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:41:36.535 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:41:36.535 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:41:36.535 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:41:36.536 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:41:36.536 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:41:36.536 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:41:36.537 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:41:36.537 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054891.002339_151.1", 2022-09-13T07:41:36.537 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:41:36.538 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:41:36.538 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:41:36.539 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:41:31.002339Z", 2022-09-13T07:41:36.539 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:41:36.540 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:41:36.540 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:41:36.540 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:41:36.541 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:41:36.541 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:41:36.541 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:41:36.541 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:41:36.542 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:41:36.542 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:41:36.542 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:41:36.543 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:41:36.543 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:41:36.543 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:41:36.543 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:41:36.544 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:41:36.544 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:41:36.545 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:41:36.545 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:41:36.546 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:41:36.547 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:41:36.547 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:41:36Z" 2022-09-13T07:41:36.548 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:41:36.550 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054891.002339_151.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:41:31.002339Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:41:36Z" } 2022-09-13T07:41:36.551 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054891.002339_151.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:41:31.002339Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:41:36.552 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054891.002339_151.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:41:36.552 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054891.002339_151.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:41:36.553 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:41:36.553 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:41:36.554 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:41:36.554 DEBUG:boto:Method: HEAD 2022-09-13T07:41:36.555 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:41:36.555 DEBUG:boto:Data: 2022-09-13T07:41:36.556 DEBUG:boto:Headers: {} 2022-09-13T07:41:36.557 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:36.557 DEBUG:boto:Port: 8000 2022-09-13T07:41:36.558 DEBUG:boto:Params: {} 2022-09-13T07:41:36.558 DEBUG:boto:Token: None 2022-09-13T07:41:36.559 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:41:36 GMT /yxvvuh-20/ 2022-09-13T07:41:36.560 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QPaSPhpU4ACoMJKkRkhSs++rK48= 2022-09-13T07:41:36.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QPaSPhpU4ACoMJKkRkhSs++rK48=', 'Content-Length': '0'} 2022-09-13T07:41:36.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b11bf367ffa556eb-0063203430-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:36.563 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:41:36.564 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:41:36.564 DEBUG:boto:Method: PUT 2022-09-13T07:41:36.565 DEBUG:boto:Path: /yxvvuh-20/obj 2022-09-13T07:41:36.566 DEBUG:boto:Data: 2022-09-13T07:41:36.567 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T07:41:36.567 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:41:36.568 DEBUG:boto:Port: 8000 2022-09-13T07:41:36.568 DEBUG:boto:Params: {} 2022-09-13T07:41:36.569 DEBUG:boto:Token: None 2022-09-13T07:41:36.570 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 13 Sep 2022 07:41:36 GMT /yxvvuh-20/obj 2022-09-13T07:41:36.570 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:76ZbTaNDQozrGw875MGrcRrwrnA= 2022-09-13T07:41:36.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:41:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:76ZbTaNDQozrGw875MGrcRrwrnA='} 2022-09-13T07:41:36.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000871903815a085c67-0063203430-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:41:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:41:36.615 DEBUG:rgw_multi.tests:created null version id=None 2022-09-13T07:41:36.616 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-20 2022-09-13T07:41:36.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:36.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:36.618 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:41:36.682 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:41:36.683 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:41:36.719 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:41:36.715+0000 7faffb600600 1 waiting for incremental sync to catch up: 2022-09-13T07:41:36.720 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T07:41:36.720 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T07:41:41.721 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:41:41.717+0000 7faffb600600 1 waiting for incremental sync to catch up: 2022-09-13T07:41:41.722 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T07:41:41.722 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T07:41:46.722 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:41:46.719+0000 7faffb600600 1 waiting for incremental sync to catch up: 2022-09-13T07:41:46.723 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T07:41:46.723 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T07:41:51.724 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:41:51.721+0000 7faffb600600 1 bucket sync caught up with source: 2022-09-13T07:41:51.725 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T07:41:51.725 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T07:41:51.726 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:41:51.721+0000 7faffb600600 0 bucket checkpoint complete 2022-09-13T07:41:51.743 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-20 2022-09-13T07:41:51.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:51.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:51.746 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:41:51.814 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:41:51.815 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:41:51.866 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:51.862+0000 7fce40f14600 1 waiting for incremental sync to catch up: 2022-09-13T07:41:51.866 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T07:41:51.867 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T07:41:56.869 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:56.864+0000 7fce40f14600 1 bucket sync caught up with source: 2022-09-13T07:41:56.869 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T07:41:56.870 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T07:41:56.870 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:56.864+0000 7fce40f14600 0 bucket checkpoint complete 2022-09-13T07:41:56.884 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-20 2022-09-13T07:41:56.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:56.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:41:56.886 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:41:56.951 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:41:56.952 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:41:56.988 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:41:56.984+0000 7f7ada256600 1 waiting for incremental sync to catch up: 2022-09-13T07:41:56.988 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T07:41:56.989 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000006.70.17, , ] 2022-09-13T07:42:01.991 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:01.986+0000 7f7ada256600 1 waiting for incremental sync to catch up: 2022-09-13T07:42:01.992 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T07:42:01.993 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000006.70.17, , ] 2022-09-13T07:42:06.992 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:06.987+0000 7f7ada256600 1 bucket sync caught up with source: 2022-09-13T07:42:06.992 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000007.72.14, , ] 2022-09-13T07:42:06.993 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000006.70.17, , ] 2022-09-13T07:42:06.993 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:06.987+0000 7f7ada256600 0 bucket checkpoint complete 2022-09-13T07:42:07.009 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-20 2022-09-13T07:42:07.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:42:07.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:42:07.010 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:42:07.075 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:42:07.076 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:42:07.129 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:07.125+0000 7f3452c03600 1 waiting for incremental sync to catch up: 2022-09-13T07:42:07.129 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000006.70.17, , ] 2022-09-13T07:42:07.129 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000007.72.14, , ] 2022-09-13T07:42:12.130 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:12.126+0000 7f3452c03600 1 waiting for incremental sync to catch up: 2022-09-13T07:42:12.131 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000006.70.17, , ] 2022-09-13T07:42:12.132 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000007.72.14, , ] 2022-09-13T07:42:17.132 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:17.128+0000 7f3452c03600 1 bucket sync caught up with source: 2022-09-13T07:42:17.133 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000007.72.14, , ] 2022-09-13T07:42:17.133 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000007.72.14, , ] 2022-09-13T07:42:17.134 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:17.128+0000 7f3452c03600 0 bucket checkpoint complete 2022-09-13T07:42:17.149 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-20 2022-09-13T07:42:17.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:42:17.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:42:17.152 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:42:17.218 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:42:17.219 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:42:17.257 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:17.253+0000 7fad0ff40600 1 bucket sync caught up with source: 2022-09-13T07:42:17.257 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000008.77.14, , ] 2022-09-13T07:42:17.258 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000008.77.14, , ] 2022-09-13T07:42:17.258 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:17.253+0000 7fad0ff40600 0 bucket checkpoint complete 2022-09-13T07:42:17.271 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-20 2022-09-13T07:42:17.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:42:17.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:42:17.274 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:42:17.342 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:42:17.343 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:42:17.389 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:42:17.383+0000 7fb55b741600 1 bucket sync caught up with source: 2022-09-13T07:42:17.389 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000008.77.14, , ] 2022-09-13T07:42:17.390 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000008.77.14, , ] 2022-09-13T07:42:17.390 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:42:17.383+0000 7fb55b741600 0 bucket checkpoint complete 2022-09-13T07:42:17.405 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-20 zones={test-zone2, test-zone1} 2022-09-13T07:42:17.406 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:42:17.407 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:42:17.407 DEBUG:boto:Method: HEAD 2022-09-13T07:42:17.408 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:42:17.408 DEBUG:boto:Data: 2022-09-13T07:42:17.409 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.409 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:42:17.410 DEBUG:boto:Port: 8000 2022-09-13T07:42:17.410 DEBUG:boto:Params: {} 2022-09-13T07:42:17.411 DEBUG:boto:Token: None 2022-09-13T07:42:17.412 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-20/ 2022-09-13T07:42:17.412 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:urpWmF5z9SLCfk+PfbZ8VYfYAYw= 2022-09-13T07:42:17.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:urpWmF5z9SLCfk+PfbZ8VYfYAYw=', 'Content-Length': '0'} 2022-09-13T07:42:17.416 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e7a268231060543-0063203459-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.417 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:42:17.417 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:42:17.418 DEBUG:boto:Method: HEAD 2022-09-13T07:42:17.419 DEBUG:boto:Path: /yxvvuh-20/ 2022-09-13T07:42:17.419 DEBUG:boto:Data: 2022-09-13T07:42:17.420 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.420 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:42:17.421 DEBUG:boto:Port: 8000 2022-09-13T07:42:17.421 DEBUG:boto:Params: {} 2022-09-13T07:42:17.422 DEBUG:boto:Token: None 2022-09-13T07:42:17.423 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-20/ 2022-09-13T07:42:17.423 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:urpWmF5z9SLCfk+PfbZ8VYfYAYw= 2022-09-13T07:42:17.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:urpWmF5z9SLCfk+PfbZ8VYfYAYw=', 'Content-Length': '0'} 2022-09-13T07:42:17.426 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090583e5261689af3-0063203459-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.427 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:42:17.427 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:42:17.428 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:42:17.428 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:42:17.429 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:42:17.429 DEBUG:boto:Method: GET 2022-09-13T07:42:17.430 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:42:17.430 DEBUG:boto:Data: 2022-09-13T07:42:17.431 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.432 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:42:17.432 DEBUG:boto:Port: 8000 2022-09-13T07:42:17.433 DEBUG:boto:Params: {} 2022-09-13T07:42:17.433 DEBUG:boto:Token: None 2022-09-13T07:42:17.434 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-20/?versions 2022-09-13T07:42:17.434 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4= 2022-09-13T07:42:17.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4=', 'Content-Length': '0'} 2022-09-13T07:42:17.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058b5bdf7a4c03347-0063203459-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.438 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:41:36.570Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vfalse2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:42:17.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:42:17.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:42:17.440 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:42:17.441 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:42:17.441 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:42:17.442 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:42:17.442 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:42:17.443 DEBUG:boto:Method: GET 2022-09-13T07:42:17.443 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:42:17.444 DEBUG:boto:Data: 2022-09-13T07:42:17.444 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.445 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:42:17.445 DEBUG:boto:Port: 8000 2022-09-13T07:42:17.446 DEBUG:boto:Params: {} 2022-09-13T07:42:17.446 DEBUG:boto:Token: None 2022-09-13T07:42:17.447 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-20/?versions 2022-09-13T07:42:17.448 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4= 2022-09-13T07:42:17.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4=', 'Content-Length': '0'} 2022-09-13T07:42:17.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008db5e8a30ca30504-0063203459-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.451 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:41:36.570Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vfalse2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:42:17.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:42:17.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:42:17.453 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:42:17.453 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:42:17.454 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:42:17.455 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:42:17.455 DEBUG:boto:Method: GET 2022-09-13T07:42:17.456 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:42:17.457 DEBUG:boto:Data: 2022-09-13T07:42:17.457 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.458 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:42:17.458 DEBUG:boto:Port: 8000 2022-09-13T07:42:17.459 DEBUG:boto:Params: {} 2022-09-13T07:42:17.459 DEBUG:boto:Token: None 2022-09-13T07:42:17.460 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-20/?versions 2022-09-13T07:42:17.460 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4= 2022-09-13T07:42:17.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4=', 'Content-Length': '0'} 2022-09-13T07:42:17.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ea7eea92babc888-0063203459-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.464 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:41:36.570Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vfalse2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:42:17.465 DEBUG:boto:path=/yxvvuh-20/ 2022-09-13T07:42:17.465 DEBUG:boto:auth_path=/yxvvuh-20/ 2022-09-13T07:42:17.466 DEBUG:boto:path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:42:17.466 DEBUG:boto:auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T07:42:17.467 DEBUG:boto:Method: GET 2022-09-13T07:42:17.467 DEBUG:boto:Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T07:42:17.468 DEBUG:boto:Data: 2022-09-13T07:42:17.468 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.469 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:42:17.469 DEBUG:boto:Port: 8000 2022-09-13T07:42:17.470 DEBUG:boto:Params: {} 2022-09-13T07:42:17.470 DEBUG:boto:Token: None 2022-09-13T07:42:17.471 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-20/?versions 2022-09-13T07:42:17.472 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4= 2022-09-13T07:42:17.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4=', 'Content-Length': '0'} 2022-09-13T07:42:17.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b560d40bfa8236b4-0063203459-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.475 DEBUG:boto:b'yxvvuh-20999falseobjnulltrue2022-09-13T07:41:36.570Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vfalse2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:42:17.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:42:17.477 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:42:17.478 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:42:17.478 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.479 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.480 DEBUG:boto:Method: GET 2022-09-13T07:42:17.480 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.481 DEBUG:boto:Data: 2022-09-13T07:42:17.481 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.482 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:42:17.482 DEBUG:boto:Port: 8000 2022-09-13T07:42:17.483 DEBUG:boto:Params: {} 2022-09-13T07:42:17.483 DEBUG:boto:Token: None 2022-09-13T07:42:17.484 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.485 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9bh+8rjcE7s01JTF3kfeMA32Kys= 2022-09-13T07:42:17.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9bh+8rjcE7s01JTF3kfeMA32Kys=', 'Content-Length': '0'} 2022-09-13T07:42:17.487 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002bc4b5c94fe3174a-0063203459-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.488 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:42:17.489 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:42:17.489 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.490 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.490 DEBUG:boto:Method: GET 2022-09-13T07:42:17.491 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.491 DEBUG:boto:Data: 2022-09-13T07:42:17.492 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.492 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:42:17.493 DEBUG:boto:Port: 8000 2022-09-13T07:42:17.494 DEBUG:boto:Params: {} 2022-09-13T07:42:17.494 DEBUG:boto:Token: None 2022-09-13T07:42:17.495 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.495 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9bh+8rjcE7s01JTF3kfeMA32Kys= 2022-09-13T07:42:17.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9bh+8rjcE7s01JTF3kfeMA32Kys=', 'Content-Length': '0'} 2022-09-13T07:42:17.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:41:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000019fd4b3ca0b852f1-0063203459-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.499 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:42:17.500 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:42:17.500 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.501 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.501 DEBUG:boto:Method: HEAD 2022-09-13T07:42:17.502 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.502 DEBUG:boto:Data: 2022-09-13T07:42:17.503 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.504 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:42:17.504 DEBUG:boto:Port: 8000 2022-09-13T07:42:17.504 DEBUG:boto:Params: {} 2022-09-13T07:42:17.505 DEBUG:boto:Token: None 2022-09-13T07:42:17.506 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.507 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ycwFijssen/dc7uoVLamXRvq97M= 2022-09-13T07:42:17.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ycwFijssen/dc7uoVLamXRvq97M=', 'Content-Length': '0'} 2022-09-13T07:42:17.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000087eeaaa6ec53203f-0063203459-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.512 DEBUG:boto:path=/yxvvuh-20/obj 2022-09-13T07:42:17.512 DEBUG:boto:auth_path=/yxvvuh-20/obj 2022-09-13T07:42:17.513 DEBUG:boto:path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.513 DEBUG:boto:auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.513 DEBUG:boto:Method: HEAD 2022-09-13T07:42:17.514 DEBUG:boto:Path: /yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.514 DEBUG:boto:Data: 2022-09-13T07:42:17.515 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.515 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:42:17.516 DEBUG:boto:Port: 8000 2022-09-13T07:42:17.516 DEBUG:boto:Params: {} 2022-09-13T07:42:17.517 DEBUG:boto:Token: None 2022-09-13T07:42:17.517 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-20/obj?versionId=null 2022-09-13T07:42:17.518 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ycwFijssen/dc7uoVLamXRvq97M= 2022-09-13T07:42:17.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ycwFijssen/dc7uoVLamXRvq97M=', 'Content-Length': '0'} 2022-09-13T07:42:17.521 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:41:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e884b945212f9d95-0063203459-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:42:17.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... FAIL 2022-09-13T07:42:17.528 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-21 2022-09-13T07:42:17.529 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:42:17.530 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:42:17.531 DEBUG:boto:Method: PUT 2022-09-13T07:42:17.532 DEBUG:boto:Path: /yxvvuh-21/ 2022-09-13T07:42:17.532 DEBUG:boto:Data: 2022-09-13T07:42:17.533 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.533 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:42:17.534 DEBUG:boto:Port: 8000 2022-09-13T07:42:17.534 DEBUG:boto:Params: {} 2022-09-13T07:42:17.535 DEBUG:boto:Token: None 2022-09-13T07:42:17.535 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-21/ 2022-09-13T07:42:17.536 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QMe0l9PKnZEEbn9CSC/0+f9juKs= 2022-09-13T07:42:17.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QMe0l9PKnZEEbn9CSC/0+f9juKs=', 'Content-Length': '0'} 2022-09-13T07:42:17.551 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000010dcc87c5a69fc6f-0063203459-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.552 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-22 2022-09-13T07:42:17.552 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:42:17.553 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:42:17.553 DEBUG:boto:Method: PUT 2022-09-13T07:42:17.554 DEBUG:boto:Path: /yxvvuh-22/ 2022-09-13T07:42:17.554 DEBUG:boto:Data: 2022-09-13T07:42:17.555 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.556 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:42:17.556 DEBUG:boto:Port: 8000 2022-09-13T07:42:17.557 DEBUG:boto:Params: {} 2022-09-13T07:42:17.557 DEBUG:boto:Token: None 2022-09-13T07:42:17.558 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-22/ 2022-09-13T07:42:17.558 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TuvUkVrYYqksD8FGI+/iaraNGJ8= 2022-09-13T07:42:17.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TuvUkVrYYqksD8FGI+/iaraNGJ8=', 'Content-Length': '0'} 2022-09-13T07:42:17.595 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000066c7339f2500faa0-0063203459-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.596 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-23 2022-09-13T07:42:17.596 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:42:17.597 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:42:17.597 DEBUG:boto:Method: PUT 2022-09-13T07:42:17.598 DEBUG:boto:Path: /yxvvuh-23/ 2022-09-13T07:42:17.598 DEBUG:boto:Data: 2022-09-13T07:42:17.599 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.599 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:42:17.600 DEBUG:boto:Port: 8001 2022-09-13T07:42:17.600 DEBUG:boto:Params: {} 2022-09-13T07:42:17.601 DEBUG:boto:Token: None 2022-09-13T07:42:17.601 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-23/ 2022-09-13T07:42:17.602 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UwQuvUyu+dRgT7od6HASiUKJM1w= 2022-09-13T07:42:17.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UwQuvUyu+dRgT7od6HASiUKJM1w=', 'Content-Length': '0'} 2022-09-13T07:42:17.651 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000006ceccf9caab3540-0063203459-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.652 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:42:17.653 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:42:17.654 DEBUG:boto:path=/yxvvuh-21/?versioning 2022-09-13T07:42:17.655 DEBUG:boto:auth_path=/yxvvuh-21/?versioning 2022-09-13T07:42:17.656 DEBUG:boto:Method: PUT 2022-09-13T07:42:17.656 DEBUG:boto:Path: /yxvvuh-21/?versioning 2022-09-13T07:42:17.657 DEBUG:boto:Data: Enabled Disabled 2022-09-13T07:42:17.658 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.659 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:42:17.660 DEBUG:boto:Port: 8000 2022-09-13T07:42:17.661 DEBUG:boto:Params: {} 2022-09-13T07:42:17.662 DEBUG:boto:Token: None 2022-09-13T07:42:17.663 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-21/?versioning 2022-09-13T07:42:17.663 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Ra5Br3dRPvebTyrrHNeGCyhAHGA= 2022-09-13T07:42:17.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Ra5Br3dRPvebTyrrHNeGCyhAHGA=', 'Content-Length': '228'} 2022-09-13T07:42:17.671 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003d9731642bfb7706-0063203459-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.672 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:42:17.673 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:42:17.674 DEBUG:boto:path=/yxvvuh-22/?versioning 2022-09-13T07:42:17.675 DEBUG:boto:auth_path=/yxvvuh-22/?versioning 2022-09-13T07:42:17.676 DEBUG:boto:Method: PUT 2022-09-13T07:42:17.677 DEBUG:boto:Path: /yxvvuh-22/?versioning 2022-09-13T07:42:17.677 DEBUG:boto:Data: Enabled Disabled 2022-09-13T07:42:17.678 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.679 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:42:17.679 DEBUG:boto:Port: 8000 2022-09-13T07:42:17.679 DEBUG:boto:Params: {} 2022-09-13T07:42:17.680 DEBUG:boto:Token: None 2022-09-13T07:42:17.681 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-22/?versioning 2022-09-13T07:42:17.681 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:PHt508bAbv99ci7OOpabxVuC3/8= 2022-09-13T07:42:17.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:PHt508bAbv99ci7OOpabxVuC3/8=', 'Content-Length': '228'} 2022-09-13T07:42:17.691 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b17f1c865a59d5a2-0063203459-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.692 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:42:17.693 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:42:17.694 DEBUG:boto:path=/yxvvuh-23/?versioning 2022-09-13T07:42:17.694 DEBUG:boto:auth_path=/yxvvuh-23/?versioning 2022-09-13T07:42:17.695 DEBUG:boto:Method: PUT 2022-09-13T07:42:17.696 DEBUG:boto:Path: /yxvvuh-23/?versioning 2022-09-13T07:42:17.696 DEBUG:boto:Data: Enabled Disabled 2022-09-13T07:42:17.697 DEBUG:boto:Headers: {} 2022-09-13T07:42:17.697 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:42:17.698 DEBUG:boto:Port: 8001 2022-09-13T07:42:17.698 DEBUG:boto:Params: {} 2022-09-13T07:42:17.699 DEBUG:boto:Token: None 2022-09-13T07:42:17.699 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:42:17 GMT /yxvvuh-23/?versioning 2022-09-13T07:42:17.700 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:eMcPoqA/qFEjHfLF/1+AYBwX74g= 2022-09-13T07:42:17.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eMcPoqA/qFEjHfLF/1+AYBwX74g=', 'Content-Length': '228'} 2022-09-13T07:42:17.708 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001d8f95ed853a3f1f-0063203459-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:17.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:42:17.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:42:17.710 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:42:17.760 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:42:17.761 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:42:17.811 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:42:17.826 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:42:17.826 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:42:17.827 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:42:17.827 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:42:17.828 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:42:17.828 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:42:17.829 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:42:17.829 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:42:17.830 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:42:17.830 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:42:17.831 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054937.703313_169.1", 2022-09-13T07:42:17.831 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:42:17.703313Z" 2022-09-13T07:42:17.832 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:42:17.832 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:42:17.833 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:42:17.833 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T07:42:17.833 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:42:17.834 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:42:17.835 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054937.703313_169.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:42:17.836 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:42:17.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:42:17.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:42:17.837 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:42:17.887 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:42:17.888 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:42:17.920 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:42:17.916+0000 7f934eb16600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:42:17.921 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:42:17.917+0000 7f934eb16600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:42:17.935 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:42:17.935 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:42:17.935 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:42:17.936 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:42:17.936 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:42:17.936 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:42:17.936 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:42:17.937 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:42:17.937 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:42:17.937 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:42:17.937 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:42:17.938 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:42:17.938 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:42:17.938 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:42:17.938 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:42:17.939 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:42:17.939 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:42:17.939 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:42:17.939 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:42:17.940 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:42:17.940 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:42:17.940 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:42:17.940 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:42:17.941 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:42:17.941 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:42:17.942 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:42:17.942 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:42:17.942 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:42:17.942 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:42:17.943 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:42:17.943 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:42:17.943 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:42:17.943 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:42:17.944 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:42:17.944 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:42:17.944 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:42:17.944 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:42:17.945 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054937.575028_159.1", 2022-09-13T07:42:17.945 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:42:17.945 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:42:17.945 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:42:17.946 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:42:17.575028Z", 2022-09-13T07:42:17.946 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:42:17.946 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:42:17.946 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:42:17.947 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:42:17.947 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:42:17.947 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:42:17.947 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:42:17.948 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:42:17.948 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:42:17.948 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:42:17.948 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:42:17.949 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:42:17.949 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:42:17.949 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:42:17.949 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:42:17.950 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:42:17.950 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:42:17.950 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:42:17.950 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:42:17.951 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:42:17.951 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:42:17.951 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:42:17Z" 2022-09-13T07:42:17.952 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:42:17.953 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054937.575028_159.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:42:17.575028Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:42:17Z" } 2022-09-13T07:42:17.954 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054937.575028_159.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:42:17.575028Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:42:17.955 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054937.703313_169.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:42:17.955 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054937.575028_159.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:42:17.956 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663054937.703313_169.1 target=1_1663054937.575028_159.1 2022-09-13T07:42:22.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:42:22.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:42:22.959 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:42:23.017 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:42:23.018 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:42:23.064 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:42:23.060+0000 7f6976d3e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:42:23.065 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:42:23.061+0000 7f6976d3e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:42:23.083 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:42:23.084 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:42:23.084 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:42:23.084 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:42:23.085 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:42:23.085 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:42:23.085 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:42:23.085 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:42:23.086 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:42:23.086 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:42:23.086 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:42:23.087 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:42:23.087 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:42:23.087 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:42:23.087 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:42:23.088 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:42:23.088 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:42:23.088 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:42:23.088 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:42:23.089 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:42:23.089 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:42:23.089 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:42:23.089 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:42:23.090 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:42:23.090 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:42:23.090 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:42:23.091 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:42:23.091 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:42:23.091 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:42:23.091 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:42:23.092 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:42:23.093 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:42:23.093 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:42:23.094 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:42:23.094 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:42:23.095 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:42:23.095 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:42:23.096 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054937.703313_169.1", 2022-09-13T07:42:23.096 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:42:23.096 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:42:23.097 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:42:23.097 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:42:17.703313Z", 2022-09-13T07:42:23.098 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:42:23.098 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:42:23.098 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:42:23.099 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:42:23.099 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:42:23.099 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:42:23.099 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:42:23.100 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:42:23.100 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:42:23.100 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:42:23.100 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:42:23.101 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:42:23.101 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:42:23.101 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:42:23.101 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:42:23.102 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:42:23.102 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:42:23.102 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:42:23.103 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:42:23.103 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:42:23.103 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:42:23.103 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:42:23Z" 2022-09-13T07:42:23.104 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:42:23.105 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054937.703313_169.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:42:17.703313Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:42:23Z" } 2022-09-13T07:42:23.106 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054937.703313_169.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:42:17.703313Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:42:23.106 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054937.703313_169.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:42:23.107 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054937.703313_169.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:42:23.107 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:42:23.108 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:42:23.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:42:23.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:42:23.110 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:42:23.168 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:42:23.168 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:42:23.198 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:23.194+0000 7f2ba43e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:42:23.199 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:23.195+0000 7f2ba43e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:42:23.225 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:42:23.225 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:42:23.225 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:42:23.226 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:42:23.226 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:42:23.226 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:42:23.227 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:42:23.227 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:42:23.228 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:42:23.229 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:42:23.229 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:42:23.229 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:42:23.230 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:42:23.231 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:42:23.231 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:42:23.232 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:42:23.232 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:42:23.232 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:42:23.233 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:42:23.233 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:42:23.234 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:42:23.234 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:42:23.235 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:42:23.235 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:42:23.236 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:42:23.236 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:42:23.237 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:42:23.237 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:42:23.238 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:42:23.238 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:42:23.239 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:42:23.239 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:42:23.239 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:42:23.240 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:42:23.240 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:42:23.240 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:42:23.240 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:42:23.241 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054937.569219_157.1", 2022-09-13T07:42:23.241 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:42:23.241 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:42:23.241 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:42:23.242 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:42:17.569219Z", 2022-09-13T07:42:23.242 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:42:23.242 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:42:23.242 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:42:23.243 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:42:23.243 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:42:23.243 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:42:23.243 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:42:23.244 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:42:23.244 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:42:23.244 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:42:23.245 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:42:23.245 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:42:23.245 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:42:23.246 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:42:23.246 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:42:23.247 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:42:23.247 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:42:23.248 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:42:23.248 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:42:23.249 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:42:23.249 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:42:23.250 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:42:23Z" 2022-09-13T07:42:23.250 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:42:23.252 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054937.569219_157.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:42:17.569219Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:42:23Z" } 2022-09-13T07:42:23.253 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054937.569219_157.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:42:17.569219Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:42:23.254 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054937.703313_169.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:42:23.254 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054937.569219_157.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:42:23.255 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1663054937.703313_169.1 target=1_1663054937.569219_157.1 2022-09-13T07:42:28.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:42:28.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:42:28.259 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:42:28.319 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:42:28.319 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:42:28.364 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:28.361+0000 7f83b0689600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:42:28.365 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:28.362+0000 7f83b0689600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:42:28.380 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:42:28.380 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:42:28.381 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:42:28.381 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:42:28.381 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:42:28.382 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:42:28.382 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:42:28.382 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:42:28.382 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:42:28.383 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:42:28.383 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:42:28.384 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:42:28.384 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:42:28.384 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:42:28.385 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:42:28.385 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:42:28.385 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:42:28.385 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:42:28.386 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:42:28.386 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:42:28.386 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:42:28.386 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:42:28.387 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:42:28.387 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:42:28.387 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:42:28.387 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:42:28.388 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:42:28.388 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:42:28.388 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:42:28.388 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:42:28.389 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:42:28.389 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:42:28.389 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:42:28.390 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:42:28.390 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:42:28.390 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:42:28.391 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:42:28.391 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054937.703313_169.1", 2022-09-13T07:42:28.391 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:42:28.391 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:42:28.391 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:42:28.392 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:42:17.703313Z", 2022-09-13T07:42:28.392 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:42:28.393 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:42:28.393 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:42:28.393 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:42:28.394 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:42:28.394 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:42:28.394 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:42:28.394 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:42:28.395 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:42:28.395 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:42:28.396 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:42:28.396 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:42:28.396 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:42:28.396 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:42:28.397 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:42:28.397 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:42:28.397 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:42:28.397 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:42:28.398 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:42:28.398 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:42:28.398 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:42:28.399 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:42:28Z" 2022-09-13T07:42:28.399 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:42:28.400 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054937.703313_169.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:42:17.703313Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:42:28Z" } 2022-09-13T07:42:28.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054937.703313_169.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:42:17.703313Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:42:28.401 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054937.703313_169.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:42:28.402 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054937.703313_169.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:42:28.402 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:42:28.403 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:42:28.403 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:42:28.404 DEBUG:boto:Method: HEAD 2022-09-13T07:42:28.405 DEBUG:boto:Path: /yxvvuh-21/ 2022-09-13T07:42:28.405 DEBUG:boto:Data: 2022-09-13T07:42:28.405 DEBUG:boto:Headers: {} 2022-09-13T07:42:28.406 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:42:28.406 DEBUG:boto:Port: 8000 2022-09-13T07:42:28.407 DEBUG:boto:Params: {} 2022-09-13T07:42:28.408 DEBUG:boto:Token: None 2022-09-13T07:42:28.408 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:42:28 GMT /yxvvuh-21/ 2022-09-13T07:42:28.409 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NLIT0z1HT/qfqypGZdf1iTPByk4= 2022-09-13T07:42:28.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NLIT0z1HT/qfqypGZdf1iTPByk4=', 'Content-Length': '0'} 2022-09-13T07:42:28.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088794c86053b0843-0063203464-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:28.413 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:42:28.413 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:42:28.414 DEBUG:boto:Method: PUT 2022-09-13T07:42:28.415 DEBUG:boto:Path: /yxvvuh-21/obj 2022-09-13T07:42:28.415 DEBUG:boto:Data: 2022-09-13T07:42:28.416 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T07:42:28.416 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:42:28.417 DEBUG:boto:Port: 8000 2022-09-13T07:42:28.417 DEBUG:boto:Params: {} 2022-09-13T07:42:28.418 DEBUG:boto:Token: None 2022-09-13T07:42:28.418 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 13 Sep 2022 07:42:28 GMT /yxvvuh-21/obj 2022-09-13T07:42:28.419 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:seD9hLEN22rObzSdbmHLp0DCeC0= 2022-09-13T07:42:28.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:42:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:seD9hLEN22rObzSdbmHLp0DCeC0='} 2022-09-13T07:42:28.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-amz-request-id', 'tx000009ec72bd2d1021266-0063203464-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:28.435 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:42:28.436 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:42:28.436 DEBUG:boto:Method: HEAD 2022-09-13T07:42:28.437 DEBUG:boto:Path: /yxvvuh-21/ 2022-09-13T07:42:28.437 DEBUG:boto:Data: 2022-09-13T07:42:28.438 DEBUG:boto:Headers: {} 2022-09-13T07:42:28.438 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:42:28.439 DEBUG:boto:Port: 8000 2022-09-13T07:42:28.440 DEBUG:boto:Params: {} 2022-09-13T07:42:28.440 DEBUG:boto:Token: None 2022-09-13T07:42:28.441 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:42:28 GMT /yxvvuh-21/ 2022-09-13T07:42:28.441 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NLIT0z1HT/qfqypGZdf1iTPByk4= 2022-09-13T07:42:28.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NLIT0z1HT/qfqypGZdf1iTPByk4=', 'Content-Length': '0'} 2022-09-13T07:42:28.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed4cd544e95bba7d-0063203464-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:28.445 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:42:28.445 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:42:28.446 DEBUG:boto:Method: DELETE 2022-09-13T07:42:28.447 DEBUG:boto:Path: /yxvvuh-21/obj 2022-09-13T07:42:28.447 DEBUG:boto:Data: 2022-09-13T07:42:28.448 DEBUG:boto:Headers: {} 2022-09-13T07:42:28.448 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:42:28.449 DEBUG:boto:Port: 8000 2022-09-13T07:42:28.454 DEBUG:boto:Params: {} 2022-09-13T07:42:28.454 DEBUG:boto:Token: None 2022-09-13T07:42:28.455 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:42:28 GMT /yxvvuh-21/obj 2022-09-13T07:42:28.456 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:v6pK4o45dXDEUNTtA3xq1F5F6+o= 2022-09-13T07:42:28.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:v6pK4o45dXDEUNTtA3xq1F5F6+o=', 'Content-Length': '0'} 2022-09-13T07:42:28.469 DEBUG:boto:Response headers: [('x-amz-version-id', 'OP6J-3ruUv8c.XUyhxYbgHFZV-T9z05'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000c31da7b41366eea8-0063203464-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:28.471 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:42:28.471 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:42:28.472 DEBUG:boto:Method: HEAD 2022-09-13T07:42:28.473 DEBUG:boto:Path: /yxvvuh-22/ 2022-09-13T07:42:28.474 DEBUG:boto:Data: 2022-09-13T07:42:28.474 DEBUG:boto:Headers: {} 2022-09-13T07:42:28.475 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:42:28.476 DEBUG:boto:Port: 8000 2022-09-13T07:42:28.477 DEBUG:boto:Params: {} 2022-09-13T07:42:28.478 DEBUG:boto:Token: None 2022-09-13T07:42:28.479 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:42:28 GMT /yxvvuh-22/ 2022-09-13T07:42:28.479 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oZNJARQobOq2Oay0AvjQ5V8daF8= 2022-09-13T07:42:28.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oZNJARQobOq2Oay0AvjQ5V8daF8=', 'Content-Length': '0'} 2022-09-13T07:42:28.482 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041649df063e8c10b-0063203464-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:28.483 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:42:28.484 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:42:28.484 DEBUG:boto:Method: PUT 2022-09-13T07:42:28.485 DEBUG:boto:Path: /yxvvuh-22/obj 2022-09-13T07:42:28.485 DEBUG:boto:Data: 2022-09-13T07:42:28.486 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T07:42:28.487 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:42:28.487 DEBUG:boto:Port: 8000 2022-09-13T07:42:28.487 DEBUG:boto:Params: {} 2022-09-13T07:42:28.488 DEBUG:boto:Token: None 2022-09-13T07:42:28.489 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 13 Sep 2022 07:42:28 GMT /yxvvuh-22/obj 2022-09-13T07:42:28.489 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/CG5ZW/EIPEVVdnHYfFqfSQJ+NE= 2022-09-13T07:42:28.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:42:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/CG5ZW/EIPEVVdnHYfFqfSQJ+NE='} 2022-09-13T07:42:28.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-amz-request-id', 'tx00000731b994fe56fff70-0063203464-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:28.540 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:42:28.540 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:42:28.541 DEBUG:boto:Method: HEAD 2022-09-13T07:42:28.541 DEBUG:boto:Path: /yxvvuh-22/ 2022-09-13T07:42:28.542 DEBUG:boto:Data: 2022-09-13T07:42:28.543 DEBUG:boto:Headers: {} 2022-09-13T07:42:28.544 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:42:28.545 DEBUG:boto:Port: 8000 2022-09-13T07:42:28.545 DEBUG:boto:Params: {} 2022-09-13T07:42:28.546 DEBUG:boto:Token: None 2022-09-13T07:42:28.547 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:42:28 GMT /yxvvuh-22/ 2022-09-13T07:42:28.548 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oZNJARQobOq2Oay0AvjQ5V8daF8= 2022-09-13T07:42:28.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oZNJARQobOq2Oay0AvjQ5V8daF8=', 'Content-Length': '0'} 2022-09-13T07:42:28.553 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005789758224df1f9c-0063203464-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:28.554 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:42:28.554 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:42:28.555 DEBUG:boto:Method: DELETE 2022-09-13T07:42:28.555 DEBUG:boto:Path: /yxvvuh-22/obj 2022-09-13T07:42:28.556 DEBUG:boto:Data: 2022-09-13T07:42:28.556 DEBUG:boto:Headers: {} 2022-09-13T07:42:28.557 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:42:28.557 DEBUG:boto:Port: 8000 2022-09-13T07:42:28.558 DEBUG:boto:Params: {} 2022-09-13T07:42:28.558 DEBUG:boto:Token: None 2022-09-13T07:42:28.559 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:42:28 GMT /yxvvuh-22/obj 2022-09-13T07:42:28.560 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Cl+hWbi892wnpR8HH1e3mao9Qfw= 2022-09-13T07:42:28.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Cl+hWbi892wnpR8HH1e3mao9Qfw=', 'Content-Length': '0'} 2022-09-13T07:42:28.569 DEBUG:boto:Response headers: [('x-amz-version-id', 'RdNIBMozGNVPQgbGp.aZdKuuPHm5I6H'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000005a8d05e2d6c84018-0063203464-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:28.570 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:42:28.571 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:42:28.571 DEBUG:boto:Method: HEAD 2022-09-13T07:42:28.572 DEBUG:boto:Path: /yxvvuh-23/ 2022-09-13T07:42:28.572 DEBUG:boto:Data: 2022-09-13T07:42:28.573 DEBUG:boto:Headers: {} 2022-09-13T07:42:28.573 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:42:28.574 DEBUG:boto:Port: 8001 2022-09-13T07:42:28.575 DEBUG:boto:Params: {} 2022-09-13T07:42:28.576 DEBUG:boto:Token: None 2022-09-13T07:42:28.577 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:42:28 GMT /yxvvuh-23/ 2022-09-13T07:42:28.578 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Tur3FIB17unMsYvW5gUxpFl7vVQ= 2022-09-13T07:42:28.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Tur3FIB17unMsYvW5gUxpFl7vVQ=', 'Content-Length': '0'} 2022-09-13T07:42:28.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012def73b0cf6fd0f-0063203464-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:28.583 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:42:28.584 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:42:28.585 DEBUG:boto:Method: PUT 2022-09-13T07:42:28.586 DEBUG:boto:Path: /yxvvuh-23/obj 2022-09-13T07:42:28.586 DEBUG:boto:Data: 2022-09-13T07:42:28.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T07:42:28.588 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:42:28.589 DEBUG:boto:Port: 8001 2022-09-13T07:42:28.590 DEBUG:boto:Params: {} 2022-09-13T07:42:28.591 DEBUG:boto:Token: None 2022-09-13T07:42:28.592 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 13 Sep 2022 07:42:28 GMT /yxvvuh-23/obj 2022-09-13T07:42:28.593 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:E1ooevaD5fo4dGP3g7Jiza5nwEg= 2022-09-13T07:42:28.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:42:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:E1ooevaD5fo4dGP3g7Jiza5nwEg='} 2022-09-13T07:42:28.607 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-amz-request-id', 'tx0000071e9d1d593f8a54d-0063203464-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:28.608 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:42:28.609 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:42:28.609 DEBUG:boto:Method: HEAD 2022-09-13T07:42:28.610 DEBUG:boto:Path: /yxvvuh-23/ 2022-09-13T07:42:28.610 DEBUG:boto:Data: 2022-09-13T07:42:28.611 DEBUG:boto:Headers: {} 2022-09-13T07:42:28.611 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:42:28.612 DEBUG:boto:Port: 8001 2022-09-13T07:42:28.612 DEBUG:boto:Params: {} 2022-09-13T07:42:28.613 DEBUG:boto:Token: None 2022-09-13T07:42:28.613 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:42:28 GMT /yxvvuh-23/ 2022-09-13T07:42:28.614 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Tur3FIB17unMsYvW5gUxpFl7vVQ= 2022-09-13T07:42:28.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Tur3FIB17unMsYvW5gUxpFl7vVQ=', 'Content-Length': '0'} 2022-09-13T07:42:28.617 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8f22d06e30f6a83-0063203464-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:28.618 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:42:28.618 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:42:28.619 DEBUG:boto:Method: DELETE 2022-09-13T07:42:28.621 DEBUG:boto:Path: /yxvvuh-23/obj 2022-09-13T07:42:28.622 DEBUG:boto:Data: 2022-09-13T07:42:28.623 DEBUG:boto:Headers: {} 2022-09-13T07:42:28.624 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:42:28.624 DEBUG:boto:Port: 8001 2022-09-13T07:42:28.625 DEBUG:boto:Params: {} 2022-09-13T07:42:28.629 DEBUG:boto:Token: None 2022-09-13T07:42:28.631 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:42:28 GMT /yxvvuh-23/obj 2022-09-13T07:42:28.632 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:MWHvZdKPbibjDjcoANTP36Hf8ro= 2022-09-13T07:42:28.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:28 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:MWHvZdKPbibjDjcoANTP36Hf8ro=', 'Content-Length': '0'} 2022-09-13T07:42:28.643 DEBUG:boto:Response headers: [('x-amz-version-id', 'g8mnQoVEf42.Xkv788ZfGhh9A7441T7'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000088406a5bd90de83f-0063203464-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:42:28.644 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-21 2022-09-13T07:42:28.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:42:28.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:42:28.646 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:42:28.710 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:42:28.711 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:42:28.749 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:42:28.746+0000 7f1b93362600 1 waiting to reach incremental sync.. 2022-09-13T07:42:33.750 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:42:33.746+0000 7f1b93362600 1 waiting to reach incremental sync.. 2022-09-13T07:42:38.750 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:42:38.747+0000 7f1b93362600 1 waiting to reach incremental sync.. 2022-09-13T07:42:43.751 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:42:43.746+0000 7f1b93362600 1 waiting to reach incremental sync.. 2022-09-13T07:42:48.753 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:42:48.749+0000 7f1b93362600 1 bucket sync caught up with source: 2022-09-13T07:42:48.753 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000003.22.11, , ] 2022-09-13T07:42:48.754 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000003.22.11, , ] 2022-09-13T07:42:48.754 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:42:48.749+0000 7f1b93362600 0 bucket checkpoint complete 2022-09-13T07:42:48.777 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-21 2022-09-13T07:42:48.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:42:48.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:42:48.780 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:42:48.847 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:42:48.848 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:42:48.895 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:48.892+0000 7fe668880600 1 bucket sync caught up with source: 2022-09-13T07:42:48.895 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000003.22.11, , ] 2022-09-13T07:42:48.895 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000003.22.11, , ] 2022-09-13T07:42:48.896 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:48.892+0000 7fe668880600 0 bucket checkpoint complete 2022-09-13T07:42:48.911 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-21 2022-09-13T07:42:48.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:42:48.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:42:48.913 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:42:48.971 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:42:48.972 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:42:49.013 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:49.010+0000 7fabe68b3600 1 waiting to reach incremental sync.. 2022-09-13T07:42:54.014 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:54.010+0000 7fabe68b3600 1 waiting to reach incremental sync.. 2022-09-13T07:42:59.015 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:42:59.011+0000 7fabe68b3600 1 waiting to reach incremental sync.. 2022-09-13T07:43:04.018 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:04.011+0000 7fabe68b3600 1 waiting to reach incremental sync.. 2022-09-13T07:43:09.017 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:09.013+0000 7fabe68b3600 1 bucket sync caught up with source: 2022-09-13T07:43:09.018 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000002.47.5, , ] 2022-09-13T07:43:09.018 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000002.47.5, , ] 2022-09-13T07:43:09.018 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:09.013+0000 7fabe68b3600 0 bucket checkpoint complete 2022-09-13T07:43:09.036 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-21 2022-09-13T07:43:09.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:09.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:09.038 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:43:09.103 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:09.104 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:09.141 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:09.138+0000 7feed142f600 1 bucket sync caught up with source: 2022-09-13T07:43:09.142 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000002.47.5, , ] 2022-09-13T07:43:09.142 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000002.47.5, , ] 2022-09-13T07:43:09.143 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:09.138+0000 7feed142f600 0 bucket checkpoint complete 2022-09-13T07:43:09.154 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-21 2022-09-13T07:43:09.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:09.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:09.156 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:43:09.216 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:09.217 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:09.248 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:09.244+0000 7f2033b2f600 1 bucket sync caught up with source: 2022-09-13T07:43:09.249 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000002.51.5, , ] 2022-09-13T07:43:09.249 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000002.51.5, , ] 2022-09-13T07:43:09.250 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:09.244+0000 7f2033b2f600 0 bucket checkpoint complete 2022-09-13T07:43:09.264 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-21 2022-09-13T07:43:09.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:09.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:09.267 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:43:09.330 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:09.331 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:09.380 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:09.376+0000 7f376a1e7600 1 bucket sync caught up with source: 2022-09-13T07:43:09.381 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000002.51.5, , ] 2022-09-13T07:43:09.381 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000002.51.5, , ] 2022-09-13T07:43:09.382 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:09.376+0000 7f376a1e7600 0 bucket checkpoint complete 2022-09-13T07:43:09.393 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-21 zones={test-zone2, test-zone1} 2022-09-13T07:43:09.395 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.396 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.397 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.398 DEBUG:boto:Path: /yxvvuh-21/ 2022-09-13T07:43:09.398 DEBUG:boto:Data: 2022-09-13T07:43:09.399 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.400 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:09.401 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.402 DEBUG:boto:Params: {} 2022-09-13T07:43:09.403 DEBUG:boto:Token: None 2022-09-13T07:43:09.404 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/ 2022-09-13T07:43:09.405 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jx4g0n2sqCsHYGMWMqbLdY1kM1A= 2022-09-13T07:43:09.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jx4g0n2sqCsHYGMWMqbLdY1kM1A=', 'Content-Length': '0'} 2022-09-13T07:43:09.411 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002668aa65a973dc88-006320348d-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.412 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.413 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.414 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.415 DEBUG:boto:Path: /yxvvuh-21/ 2022-09-13T07:43:09.415 DEBUG:boto:Data: 2022-09-13T07:43:09.416 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.417 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.418 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.418 DEBUG:boto:Params: {} 2022-09-13T07:43:09.419 DEBUG:boto:Token: None 2022-09-13T07:43:09.419 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/ 2022-09-13T07:43:09.420 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jx4g0n2sqCsHYGMWMqbLdY1kM1A= 2022-09-13T07:43:09.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jx4g0n2sqCsHYGMWMqbLdY1kM1A=', 'Content-Length': '0'} 2022-09-13T07:43:09.423 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f160847629cb55b-006320348d-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.424 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:43:09.425 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.425 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.426 DEBUG:boto:path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.426 DEBUG:boto:auth_path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.427 DEBUG:boto:Method: GET 2022-09-13T07:43:09.427 DEBUG:boto:Path: /yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.428 DEBUG:boto:Data: 2022-09-13T07:43:09.428 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.429 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:09.429 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.430 DEBUG:boto:Params: {} 2022-09-13T07:43:09.431 DEBUG:boto:Token: None 2022-09-13T07:43:09.431 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/?versions 2022-09-13T07:43:09.432 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw= 2022-09-13T07:43:09.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw=', 'Content-Length': '0'} 2022-09-13T07:43:09.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089dd443193dd63c3-006320348d-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.435 DEBUG:boto:b'yxvvuh-21999falseobjOP6J-3ruUv8c.XUyhxYbgHFZV-T9z05true2022-09-13T07:42:28.458Zrgw-multisite-test-userMultisite Test UserNormalobjmjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCbfalse2022-09-13T07:42:28.419Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:09.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:09.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:09.438 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:43:09.438 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.439 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.439 DEBUG:boto:path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.440 DEBUG:boto:auth_path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.440 DEBUG:boto:Method: GET 2022-09-13T07:43:09.441 DEBUG:boto:Path: /yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.442 DEBUG:boto:Data: 2022-09-13T07:43:09.442 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.443 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.443 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.444 DEBUG:boto:Params: {} 2022-09-13T07:43:09.444 DEBUG:boto:Token: None 2022-09-13T07:43:09.445 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/?versions 2022-09-13T07:43:09.446 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw= 2022-09-13T07:43:09.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw=', 'Content-Length': '0'} 2022-09-13T07:43:09.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000210f137ec5560345-006320348d-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.450 DEBUG:boto:b'yxvvuh-21999falseobjOP6J-3ruUv8c.XUyhxYbgHFZV-T9z05true2022-09-13T07:42:28.458Zrgw-multisite-test-userMultisite Test UserNormalobjmjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCbfalse2022-09-13T07:42:28.419Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:09.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:09.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:09.453 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.453 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.454 DEBUG:boto:path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.455 DEBUG:boto:auth_path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.455 DEBUG:boto:Method: GET 2022-09-13T07:43:09.456 DEBUG:boto:Path: /yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.456 DEBUG:boto:Data: 2022-09-13T07:43:09.457 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.457 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:09.458 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.458 DEBUG:boto:Params: {} 2022-09-13T07:43:09.459 DEBUG:boto:Token: None 2022-09-13T07:43:09.460 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/?versions 2022-09-13T07:43:09.460 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw= 2022-09-13T07:43:09.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw=', 'Content-Length': '0'} 2022-09-13T07:43:09.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa05b372907c322f-006320348d-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.506 DEBUG:boto:b'yxvvuh-21999falseobjOP6J-3ruUv8c.XUyhxYbgHFZV-T9z05true2022-09-13T07:42:28.458Zrgw-multisite-test-userMultisite Test UserNormalobjmjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCbfalse2022-09-13T07:42:28.419Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:09.507 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.508 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.508 DEBUG:boto:path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.509 DEBUG:boto:auth_path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.510 DEBUG:boto:Method: GET 2022-09-13T07:43:09.510 DEBUG:boto:Path: /yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.511 DEBUG:boto:Data: 2022-09-13T07:43:09.511 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.512 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.512 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.513 DEBUG:boto:Params: {} 2022-09-13T07:43:09.514 DEBUG:boto:Token: None 2022-09-13T07:43:09.514 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/?versions 2022-09-13T07:43:09.515 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw= 2022-09-13T07:43:09.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw=', 'Content-Length': '0'} 2022-09-13T07:43:09.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043560c78ff42d1a4-006320348d-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.519 DEBUG:boto:b'yxvvuh-21999falseobjOP6J-3ruUv8c.XUyhxYbgHFZV-T9z05true2022-09-13T07:42:28.458Zrgw-multisite-test-userMultisite Test UserNormalobjmjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCbfalse2022-09-13T07:42:28.419Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:09.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:09.520 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.520 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.521 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.521 DEBUG:boto:Path: /yxvvuh-21/obj 2022-09-13T07:43:09.522 DEBUG:boto:Data: 2022-09-13T07:43:09.522 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.523 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:09.523 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.524 DEBUG:boto:Params: {} 2022-09-13T07:43:09.524 DEBUG:boto:Token: None 2022-09-13T07:43:09.525 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj 2022-09-13T07:43:09.526 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA= 2022-09-13T07:43:09.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA=', 'Content-Length': '0'} 2022-09-13T07:43:09.529 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000088c7009c5ed89a94-006320348d-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.530 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.530 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.531 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.532 DEBUG:boto:Path: /yxvvuh-21/obj 2022-09-13T07:43:09.533 DEBUG:boto:Data: 2022-09-13T07:43:09.534 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.535 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.535 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.536 DEBUG:boto:Params: {} 2022-09-13T07:43:09.537 DEBUG:boto:Token: None 2022-09-13T07:43:09.538 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj 2022-09-13T07:43:09.539 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA= 2022-09-13T07:43:09.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA=', 'Content-Length': '0'} 2022-09-13T07:43:09.543 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000fea71c971ff3a1a3-006320348d-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:09.545 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.545 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.546 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.547 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.548 DEBUG:boto:Method: GET 2022-09-13T07:43:09.549 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.550 DEBUG:boto:Data: 2022-09-13T07:43:09.551 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.551 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:09.552 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.553 DEBUG:boto:Params: {} 2022-09-13T07:43:09.554 DEBUG:boto:Token: None 2022-09-13T07:43:09.555 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.556 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms= 2022-09-13T07:43:09.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms=', 'Content-Length': '0'} 2022-09-13T07:43:09.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005dc881da3669bf74-006320348d-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.560 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.561 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.562 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.562 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.564 DEBUG:boto:Method: GET 2022-09-13T07:43:09.566 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.566 DEBUG:boto:Data: 2022-09-13T07:43:09.567 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.567 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.568 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.569 DEBUG:boto:Params: {} 2022-09-13T07:43:09.570 DEBUG:boto:Token: None 2022-09-13T07:43:09.570 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.571 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms= 2022-09-13T07:43:09.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms=', 'Content-Length': '0'} 2022-09-13T07:43:09.574 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab486af391ac41c4-006320348d-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.575 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.576 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.577 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.578 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.579 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.580 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.581 DEBUG:boto:Data: 2022-09-13T07:43:09.582 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.583 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:09.584 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.584 DEBUG:boto:Params: {} 2022-09-13T07:43:09.585 DEBUG:boto:Token: None 2022-09-13T07:43:09.586 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.587 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:n30hOQzCZxkhswjWUpZVxrWVU1U= 2022-09-13T07:43:09.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:n30hOQzCZxkhswjWUpZVxrWVU1U=', 'Content-Length': '0'} 2022-09-13T07:43:09.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f42fd9551223606-006320348d-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.592 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.592 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.593 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.593 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.594 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.595 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.595 DEBUG:boto:Data: 2022-09-13T07:43:09.596 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.597 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.597 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.598 DEBUG:boto:Params: {} 2022-09-13T07:43:09.599 DEBUG:boto:Token: None 2022-09-13T07:43:09.600 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.601 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:n30hOQzCZxkhswjWUpZVxrWVU1U= 2022-09-13T07:43:09.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:n30hOQzCZxkhswjWUpZVxrWVU1U=', 'Content-Length': '0'} 2022-09-13T07:43:09.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000019cd712c964cd87a-006320348d-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:09.605 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.606 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.607 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.608 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.608 DEBUG:boto:Method: GET 2022-09-13T07:43:09.609 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.610 DEBUG:boto:Data: 2022-09-13T07:43:09.611 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.612 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:09.613 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.614 DEBUG:boto:Params: {} 2022-09-13T07:43:09.615 DEBUG:boto:Token: None 2022-09-13T07:43:09.616 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.617 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms= 2022-09-13T07:43:09.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms=', 'Content-Length': '0'} 2022-09-13T07:43:09.619 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bb1d25af00b83eb4-006320348d-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.620 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.621 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.621 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.622 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.622 DEBUG:boto:Method: GET 2022-09-13T07:43:09.623 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.624 DEBUG:boto:Data: 2022-09-13T07:43:09.624 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.625 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.625 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.626 DEBUG:boto:Params: {} 2022-09-13T07:43:09.627 DEBUG:boto:Token: None 2022-09-13T07:43:09.627 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.628 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms= 2022-09-13T07:43:09.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms=', 'Content-Length': '0'} 2022-09-13T07:43:09.631 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009697909e16d30dae-006320348d-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.632 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.633 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.634 DEBUG:boto:path=/yxvvuh-21/?prefix=obj 2022-09-13T07:43:09.635 DEBUG:boto:auth_path=/yxvvuh-21/?prefix=obj 2022-09-13T07:43:09.636 DEBUG:boto:Method: GET 2022-09-13T07:43:09.637 DEBUG:boto:Path: /yxvvuh-21/?prefix=obj 2022-09-13T07:43:09.638 DEBUG:boto:Data: 2022-09-13T07:43:09.639 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.640 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:09.641 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.641 DEBUG:boto:Params: {} 2022-09-13T07:43:09.643 DEBUG:boto:Token: None 2022-09-13T07:43:09.644 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/ 2022-09-13T07:43:09.645 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ax+HFm7XzitcoTBCaWmd2h02nLw= 2022-09-13T07:43:09.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ax+HFm7XzitcoTBCaWmd2h02nLw=', 'Content-Length': '0'} 2022-09-13T07:43:09.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088819775c5f94006-006320348d-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.651 DEBUG:boto:b'yxvvuh-21obj1000false' 2022-09-13T07:43:09.653 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.654 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.655 DEBUG:boto:path=/yxvvuh-21/?prefix=obj 2022-09-13T07:43:09.656 DEBUG:boto:auth_path=/yxvvuh-21/?prefix=obj 2022-09-13T07:43:09.657 DEBUG:boto:Method: GET 2022-09-13T07:43:09.658 DEBUG:boto:Path: /yxvvuh-21/?prefix=obj 2022-09-13T07:43:09.659 DEBUG:boto:Data: 2022-09-13T07:43:09.660 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.661 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.661 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.662 DEBUG:boto:Params: {} 2022-09-13T07:43:09.663 DEBUG:boto:Token: None 2022-09-13T07:43:09.663 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/ 2022-09-13T07:43:09.664 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ax+HFm7XzitcoTBCaWmd2h02nLw= 2022-09-13T07:43:09.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ax+HFm7XzitcoTBCaWmd2h02nLw=', 'Content-Length': '0'} 2022-09-13T07:43:09.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000495a55afedf726b6-006320348d-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.710 DEBUG:boto:b'yxvvuh-21obj1000false' 2022-09-13T07:43:09.712 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.712 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.713 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.713 DEBUG:boto:Path: /yxvvuh-21/obj 2022-09-13T07:43:09.714 DEBUG:boto:Data: 2022-09-13T07:43:09.714 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.715 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:09.715 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.716 DEBUG:boto:Params: {} 2022-09-13T07:43:09.717 DEBUG:boto:Token: None 2022-09-13T07:43:09.717 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj 2022-09-13T07:43:09.718 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA= 2022-09-13T07:43:09.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA=', 'Content-Length': '0'} 2022-09-13T07:43:09.721 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000021cfc6be98c3948b-006320348d-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.721 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.722 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.723 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.723 DEBUG:boto:Path: /yxvvuh-21/obj 2022-09-13T07:43:09.724 DEBUG:boto:Data: 2022-09-13T07:43:09.724 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.725 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.725 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.726 DEBUG:boto:Params: {} 2022-09-13T07:43:09.726 DEBUG:boto:Token: None 2022-09-13T07:43:09.727 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj 2022-09-13T07:43:09.727 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA= 2022-09-13T07:43:09.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA=', 'Content-Length': '0'} 2022-09-13T07:43:09.730 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000d834a9e2722b713b-006320348d-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.731 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-21 zones={test-zone2, test-zone1} 2022-09-13T07:43:09.731 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-21 zones={test-zone3, test-zone1} 2022-09-13T07:43:09.732 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.733 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.733 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.734 DEBUG:boto:Path: /yxvvuh-21/ 2022-09-13T07:43:09.734 DEBUG:boto:Data: 2022-09-13T07:43:09.735 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.735 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:09.736 DEBUG:boto:Port: 8001 2022-09-13T07:43:09.736 DEBUG:boto:Params: {} 2022-09-13T07:43:09.737 DEBUG:boto:Token: None 2022-09-13T07:43:09.737 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/ 2022-09-13T07:43:09.738 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jx4g0n2sqCsHYGMWMqbLdY1kM1A= 2022-09-13T07:43:09.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jx4g0n2sqCsHYGMWMqbLdY1kM1A=', 'Content-Length': '0'} 2022-09-13T07:43:09.741 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbb15e1da471ae73-006320348d-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.742 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.742 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.743 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.744 DEBUG:boto:Path: /yxvvuh-21/ 2022-09-13T07:43:09.744 DEBUG:boto:Data: 2022-09-13T07:43:09.745 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.746 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.746 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.747 DEBUG:boto:Params: {} 2022-09-13T07:43:09.747 DEBUG:boto:Token: None 2022-09-13T07:43:09.748 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/ 2022-09-13T07:43:09.749 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jx4g0n2sqCsHYGMWMqbLdY1kM1A= 2022-09-13T07:43:09.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jx4g0n2sqCsHYGMWMqbLdY1kM1A=', 'Content-Length': '0'} 2022-09-13T07:43:09.752 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006cc3e136acdfb50f-006320348d-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.753 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:43:09.753 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.754 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.755 DEBUG:boto:path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.755 DEBUG:boto:auth_path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.756 DEBUG:boto:Method: GET 2022-09-13T07:43:09.757 DEBUG:boto:Path: /yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.757 DEBUG:boto:Data: 2022-09-13T07:43:09.758 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.758 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:09.759 DEBUG:boto:Port: 8001 2022-09-13T07:43:09.759 DEBUG:boto:Params: {} 2022-09-13T07:43:09.760 DEBUG:boto:Token: None 2022-09-13T07:43:09.761 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/?versions 2022-09-13T07:43:09.761 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw= 2022-09-13T07:43:09.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw=', 'Content-Length': '0'} 2022-09-13T07:43:09.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000786867303afd6945-006320348d-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.765 DEBUG:boto:b'yxvvuh-21999falseobjOP6J-3ruUv8c.XUyhxYbgHFZV-T9z05true2022-09-13T07:42:28.458Zrgw-multisite-test-userMultisite Test UserNormalobjmjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCbfalse2022-09-13T07:42:28.419Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:09.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:09.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:09.767 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:43:09.768 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.768 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.769 DEBUG:boto:path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.769 DEBUG:boto:auth_path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.770 DEBUG:boto:Method: GET 2022-09-13T07:43:09.770 DEBUG:boto:Path: /yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.771 DEBUG:boto:Data: 2022-09-13T07:43:09.771 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.772 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.772 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.773 DEBUG:boto:Params: {} 2022-09-13T07:43:09.774 DEBUG:boto:Token: None 2022-09-13T07:43:09.774 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/?versions 2022-09-13T07:43:09.775 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw= 2022-09-13T07:43:09.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw=', 'Content-Length': '0'} 2022-09-13T07:43:09.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aa3a5921108413c-006320348d-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.779 DEBUG:boto:b'yxvvuh-21999falseobjOP6J-3ruUv8c.XUyhxYbgHFZV-T9z05true2022-09-13T07:42:28.458Zrgw-multisite-test-userMultisite Test UserNormalobjmjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCbfalse2022-09-13T07:42:28.419Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:09.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:09.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:09.781 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.781 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.782 DEBUG:boto:path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.783 DEBUG:boto:auth_path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.783 DEBUG:boto:Method: GET 2022-09-13T07:43:09.784 DEBUG:boto:Path: /yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.784 DEBUG:boto:Data: 2022-09-13T07:43:09.785 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.785 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:09.786 DEBUG:boto:Port: 8001 2022-09-13T07:43:09.786 DEBUG:boto:Params: {} 2022-09-13T07:43:09.787 DEBUG:boto:Token: None 2022-09-13T07:43:09.787 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/?versions 2022-09-13T07:43:09.788 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw= 2022-09-13T07:43:09.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw=', 'Content-Length': '0'} 2022-09-13T07:43:09.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b96f1ca375ce1fc-006320348d-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.792 DEBUG:boto:b'yxvvuh-21999falseobjOP6J-3ruUv8c.XUyhxYbgHFZV-T9z05true2022-09-13T07:42:28.458Zrgw-multisite-test-userMultisite Test UserNormalobjmjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCbfalse2022-09-13T07:42:28.419Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:09.793 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.794 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.794 DEBUG:boto:path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.795 DEBUG:boto:auth_path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.795 DEBUG:boto:Method: GET 2022-09-13T07:43:09.796 DEBUG:boto:Path: /yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:09.797 DEBUG:boto:Data: 2022-09-13T07:43:09.797 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.798 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.798 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.799 DEBUG:boto:Params: {} 2022-09-13T07:43:09.800 DEBUG:boto:Token: None 2022-09-13T07:43:09.800 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/?versions 2022-09-13T07:43:09.801 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw= 2022-09-13T07:43:09.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yewTIlQpp7ZK1+wLWy506sRJTuw=', 'Content-Length': '0'} 2022-09-13T07:43:09.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ef32055bbd57762-006320348d-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.850 DEBUG:boto:b'yxvvuh-21999falseobjOP6J-3ruUv8c.XUyhxYbgHFZV-T9z05true2022-09-13T07:42:28.458Zrgw-multisite-test-userMultisite Test UserNormalobjmjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCbfalse2022-09-13T07:42:28.419Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:09.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:09.852 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.852 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.853 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.853 DEBUG:boto:Path: /yxvvuh-21/obj 2022-09-13T07:43:09.854 DEBUG:boto:Data: 2022-09-13T07:43:09.854 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.855 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:09.855 DEBUG:boto:Port: 8001 2022-09-13T07:43:09.855 DEBUG:boto:Params: {} 2022-09-13T07:43:09.856 DEBUG:boto:Token: None 2022-09-13T07:43:09.857 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj 2022-09-13T07:43:09.857 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA= 2022-09-13T07:43:09.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA=', 'Content-Length': '0'} 2022-09-13T07:43:09.860 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000006e6ee75b2c742ad2-006320348d-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.860 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.861 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.861 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.862 DEBUG:boto:Path: /yxvvuh-21/obj 2022-09-13T07:43:09.862 DEBUG:boto:Data: 2022-09-13T07:43:09.863 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.863 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.864 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.864 DEBUG:boto:Params: {} 2022-09-13T07:43:09.865 DEBUG:boto:Token: None 2022-09-13T07:43:09.865 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj 2022-09-13T07:43:09.866 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA= 2022-09-13T07:43:09.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA=', 'Content-Length': '0'} 2022-09-13T07:43:09.869 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000da0d3c64cdbe9459-006320348d-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:09.870 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.871 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.872 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.872 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.873 DEBUG:boto:Method: GET 2022-09-13T07:43:09.873 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.874 DEBUG:boto:Data: 2022-09-13T07:43:09.874 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.875 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:09.875 DEBUG:boto:Port: 8001 2022-09-13T07:43:09.876 DEBUG:boto:Params: {} 2022-09-13T07:43:09.876 DEBUG:boto:Token: None 2022-09-13T07:43:09.877 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.877 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms= 2022-09-13T07:43:09.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms=', 'Content-Length': '0'} 2022-09-13T07:43:09.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036f6d218b0dc41e7-006320348d-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.882 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.882 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.883 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.883 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.884 DEBUG:boto:Method: GET 2022-09-13T07:43:09.884 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.885 DEBUG:boto:Data: 2022-09-13T07:43:09.885 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.886 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.886 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.887 DEBUG:boto:Params: {} 2022-09-13T07:43:09.887 DEBUG:boto:Token: None 2022-09-13T07:43:09.888 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.889 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms= 2022-09-13T07:43:09.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms=', 'Content-Length': '0'} 2022-09-13T07:43:09.891 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007cc98863330e1f81-006320348d-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.892 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.893 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.893 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.894 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.894 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.895 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.895 DEBUG:boto:Data: 2022-09-13T07:43:09.895 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.896 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:09.896 DEBUG:boto:Port: 8001 2022-09-13T07:43:09.897 DEBUG:boto:Params: {} 2022-09-13T07:43:09.898 DEBUG:boto:Token: None 2022-09-13T07:43:09.898 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.899 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:n30hOQzCZxkhswjWUpZVxrWVU1U= 2022-09-13T07:43:09.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:n30hOQzCZxkhswjWUpZVxrWVU1U=', 'Content-Length': '0'} 2022-09-13T07:43:09.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040a0cf9678b70c3e-006320348d-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.903 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.904 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.905 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.906 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.907 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.907 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.908 DEBUG:boto:Data: 2022-09-13T07:43:09.908 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.909 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.909 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.910 DEBUG:boto:Params: {} 2022-09-13T07:43:09.910 DEBUG:boto:Token: None 2022-09-13T07:43:09.911 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.911 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:n30hOQzCZxkhswjWUpZVxrWVU1U= 2022-09-13T07:43:09.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:n30hOQzCZxkhswjWUpZVxrWVU1U=', 'Content-Length': '0'} 2022-09-13T07:43:09.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000893cf945df31388c-006320348d-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:09.915 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.916 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.916 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.917 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.917 DEBUG:boto:Method: GET 2022-09-13T07:43:09.918 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.918 DEBUG:boto:Data: 2022-09-13T07:43:09.919 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.919 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:09.920 DEBUG:boto:Port: 8001 2022-09-13T07:43:09.920 DEBUG:boto:Params: {} 2022-09-13T07:43:09.921 DEBUG:boto:Token: None 2022-09-13T07:43:09.922 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.922 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms= 2022-09-13T07:43:09.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms=', 'Content-Length': '0'} 2022-09-13T07:43:09.925 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000045e3637a0fa37d8c-006320348d-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.926 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.926 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.927 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.927 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.928 DEBUG:boto:Method: GET 2022-09-13T07:43:09.928 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.929 DEBUG:boto:Data: 2022-09-13T07:43:09.929 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.930 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.930 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.931 DEBUG:boto:Params: {} 2022-09-13T07:43:09.931 DEBUG:boto:Token: None 2022-09-13T07:43:09.932 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:09.933 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms= 2022-09-13T07:43:09.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:p6zo3JTndmOmFlWnAI/M9PUBwms=', 'Content-Length': '0'} 2022-09-13T07:43:09.935 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004f7a451c0ab6a636-006320348d-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.936 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.936 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.937 DEBUG:boto:path=/yxvvuh-21/?prefix=obj 2022-09-13T07:43:09.937 DEBUG:boto:auth_path=/yxvvuh-21/?prefix=obj 2022-09-13T07:43:09.938 DEBUG:boto:Method: GET 2022-09-13T07:43:09.938 DEBUG:boto:Path: /yxvvuh-21/?prefix=obj 2022-09-13T07:43:09.939 DEBUG:boto:Data: 2022-09-13T07:43:09.940 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.941 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:09.941 DEBUG:boto:Port: 8001 2022-09-13T07:43:09.942 DEBUG:boto:Params: {} 2022-09-13T07:43:09.942 DEBUG:boto:Token: None 2022-09-13T07:43:09.943 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/ 2022-09-13T07:43:09.943 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ax+HFm7XzitcoTBCaWmd2h02nLw= 2022-09-13T07:43:09.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ax+HFm7XzitcoTBCaWmd2h02nLw=', 'Content-Length': '0'} 2022-09-13T07:43:09.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c01a2cbcec885e1-006320348d-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.947 DEBUG:boto:b'yxvvuh-21obj1000false' 2022-09-13T07:43:09.948 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.949 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.949 DEBUG:boto:path=/yxvvuh-21/?prefix=obj 2022-09-13T07:43:09.950 DEBUG:boto:auth_path=/yxvvuh-21/?prefix=obj 2022-09-13T07:43:09.950 DEBUG:boto:Method: GET 2022-09-13T07:43:09.951 DEBUG:boto:Path: /yxvvuh-21/?prefix=obj 2022-09-13T07:43:09.951 DEBUG:boto:Data: 2022-09-13T07:43:09.952 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.952 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.953 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.953 DEBUG:boto:Params: {} 2022-09-13T07:43:09.954 DEBUG:boto:Token: None 2022-09-13T07:43:09.954 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/ 2022-09-13T07:43:09.955 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ax+HFm7XzitcoTBCaWmd2h02nLw= 2022-09-13T07:43:09.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ax+HFm7XzitcoTBCaWmd2h02nLw=', 'Content-Length': '0'} 2022-09-13T07:43:09.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002551da9bac876d06-006320348d-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.961 DEBUG:boto:b'yxvvuh-21obj1000false' 2022-09-13T07:43:09.962 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.963 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.964 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.965 DEBUG:boto:Path: /yxvvuh-21/obj 2022-09-13T07:43:09.965 DEBUG:boto:Data: 2022-09-13T07:43:09.966 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.967 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:09.968 DEBUG:boto:Port: 8001 2022-09-13T07:43:09.968 DEBUG:boto:Params: {} 2022-09-13T07:43:09.969 DEBUG:boto:Token: None 2022-09-13T07:43:09.970 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj 2022-09-13T07:43:09.971 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA= 2022-09-13T07:43:09.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA=', 'Content-Length': '0'} 2022-09-13T07:43:09.984 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000000d471909b85d6883-006320348d-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.985 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:09.985 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:09.986 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.986 DEBUG:boto:Path: /yxvvuh-21/obj 2022-09-13T07:43:09.987 DEBUG:boto:Data: 2022-09-13T07:43:09.987 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.988 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:09.988 DEBUG:boto:Port: 8000 2022-09-13T07:43:09.989 DEBUG:boto:Params: {} 2022-09-13T07:43:09.989 DEBUG:boto:Token: None 2022-09-13T07:43:09.990 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:09 GMT /yxvvuh-21/obj 2022-09-13T07:43:09.991 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA= 2022-09-13T07:43:09.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:g6yibv+lrOsxxg0AvG2NpzLlQeA=', 'Content-Length': '0'} 2022-09-13T07:43:09.994 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000011e9e1a4dc12fcb8-006320348d-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:09.995 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-21 zones={test-zone3, test-zone1} 2022-09-13T07:43:09.995 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-21 zones={test-zone3, test-zone2} 2022-09-13T07:43:09.996 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:09.996 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:09.997 DEBUG:boto:Method: HEAD 2022-09-13T07:43:09.997 DEBUG:boto:Path: /yxvvuh-21/ 2022-09-13T07:43:09.998 DEBUG:boto:Data: 2022-09-13T07:43:09.998 DEBUG:boto:Headers: {} 2022-09-13T07:43:09.999 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:09.999 DEBUG:boto:Port: 8001 2022-09-13T07:43:10.000 DEBUG:boto:Params: {} 2022-09-13T07:43:10.000 DEBUG:boto:Token: None 2022-09-13T07:43:10.001 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/ 2022-09-13T07:43:10.002 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NWmk5iBslHy8wZJ+1+x0AcUTr7k= 2022-09-13T07:43:10.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NWmk5iBslHy8wZJ+1+x0AcUTr7k=', 'Content-Length': '0'} 2022-09-13T07:43:10.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e4d507e728ab2db-006320348e-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.005 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:10.006 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:10.006 DEBUG:boto:Method: HEAD 2022-09-13T07:43:10.007 DEBUG:boto:Path: /yxvvuh-21/ 2022-09-13T07:43:10.007 DEBUG:boto:Data: 2022-09-13T07:43:10.008 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.008 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:10.009 DEBUG:boto:Port: 8000 2022-09-13T07:43:10.009 DEBUG:boto:Params: {} 2022-09-13T07:43:10.010 DEBUG:boto:Token: None 2022-09-13T07:43:10.010 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/ 2022-09-13T07:43:10.011 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NWmk5iBslHy8wZJ+1+x0AcUTr7k= 2022-09-13T07:43:10.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NWmk5iBslHy8wZJ+1+x0AcUTr7k=', 'Content-Length': '0'} 2022-09-13T07:43:10.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008255d82b1a70cea9-006320348e-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.016 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:43:10.016 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:10.017 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:10.017 DEBUG:boto:path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:10.018 DEBUG:boto:auth_path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:10.018 DEBUG:boto:Method: GET 2022-09-13T07:43:10.019 DEBUG:boto:Path: /yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:10.020 DEBUG:boto:Data: 2022-09-13T07:43:10.020 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.021 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:10.021 DEBUG:boto:Port: 8001 2022-09-13T07:43:10.022 DEBUG:boto:Params: {} 2022-09-13T07:43:10.022 DEBUG:boto:Token: None 2022-09-13T07:43:10.023 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/?versions 2022-09-13T07:43:10.024 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:sE6UCphH9Nkf/tTDiZHTtVhGtmw= 2022-09-13T07:43:10.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:sE6UCphH9Nkf/tTDiZHTtVhGtmw=', 'Content-Length': '0'} 2022-09-13T07:43:10.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000599827b6424f40c0-006320348e-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.078 DEBUG:boto:b'yxvvuh-21999falseobjOP6J-3ruUv8c.XUyhxYbgHFZV-T9z05true2022-09-13T07:42:28.458Zrgw-multisite-test-userMultisite Test UserNormalobjmjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCbfalse2022-09-13T07:42:28.419Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:10.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:10.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:10.094 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:43:10.095 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:10.096 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:10.097 DEBUG:boto:path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:10.098 DEBUG:boto:auth_path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:10.099 DEBUG:boto:Method: GET 2022-09-13T07:43:10.100 DEBUG:boto:Path: /yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:10.101 DEBUG:boto:Data: 2022-09-13T07:43:10.102 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.103 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:10.104 DEBUG:boto:Port: 8000 2022-09-13T07:43:10.105 DEBUG:boto:Params: {} 2022-09-13T07:43:10.105 DEBUG:boto:Token: None 2022-09-13T07:43:10.106 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/?versions 2022-09-13T07:43:10.107 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:sE6UCphH9Nkf/tTDiZHTtVhGtmw= 2022-09-13T07:43:10.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:sE6UCphH9Nkf/tTDiZHTtVhGtmw=', 'Content-Length': '0'} 2022-09-13T07:43:10.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efe6c832712b7e4f-006320348e-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.112 DEBUG:boto:b'yxvvuh-21999falseobjOP6J-3ruUv8c.XUyhxYbgHFZV-T9z05true2022-09-13T07:42:28.458Zrgw-multisite-test-userMultisite Test UserNormalobjmjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCbfalse2022-09-13T07:42:28.419Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:10.113 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:10.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:10.115 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:10.116 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:10.116 DEBUG:boto:path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:10.117 DEBUG:boto:auth_path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:10.117 DEBUG:boto:Method: GET 2022-09-13T07:43:10.118 DEBUG:boto:Path: /yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:10.119 DEBUG:boto:Data: 2022-09-13T07:43:10.119 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.120 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:10.120 DEBUG:boto:Port: 8001 2022-09-13T07:43:10.121 DEBUG:boto:Params: {} 2022-09-13T07:43:10.121 DEBUG:boto:Token: None 2022-09-13T07:43:10.122 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/?versions 2022-09-13T07:43:10.122 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:sE6UCphH9Nkf/tTDiZHTtVhGtmw= 2022-09-13T07:43:10.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:sE6UCphH9Nkf/tTDiZHTtVhGtmw=', 'Content-Length': '0'} 2022-09-13T07:43:10.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f27f4d0d3d50f34-006320348e-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.126 DEBUG:boto:b'yxvvuh-21999falseobjOP6J-3ruUv8c.XUyhxYbgHFZV-T9z05true2022-09-13T07:42:28.458Zrgw-multisite-test-userMultisite Test UserNormalobjmjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCbfalse2022-09-13T07:42:28.419Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:10.127 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:10.128 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:10.128 DEBUG:boto:path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:10.129 DEBUG:boto:auth_path=/yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:10.130 DEBUG:boto:Method: GET 2022-09-13T07:43:10.130 DEBUG:boto:Path: /yxvvuh-21/?versions&max-keys=999 2022-09-13T07:43:10.131 DEBUG:boto:Data: 2022-09-13T07:43:10.132 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.132 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:10.133 DEBUG:boto:Port: 8000 2022-09-13T07:43:10.134 DEBUG:boto:Params: {} 2022-09-13T07:43:10.134 DEBUG:boto:Token: None 2022-09-13T07:43:10.135 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/?versions 2022-09-13T07:43:10.137 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:sE6UCphH9Nkf/tTDiZHTtVhGtmw= 2022-09-13T07:43:10.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:sE6UCphH9Nkf/tTDiZHTtVhGtmw=', 'Content-Length': '0'} 2022-09-13T07:43:10.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd0d1d52dbd5ae9-006320348e-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.190 DEBUG:boto:b'yxvvuh-21999falseobjOP6J-3ruUv8c.XUyhxYbgHFZV-T9z05true2022-09-13T07:42:28.458Zrgw-multisite-test-userMultisite Test UserNormalobjmjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCbfalse2022-09-13T07:42:28.419Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:10.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:10.192 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:10.192 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:10.193 DEBUG:boto:Method: HEAD 2022-09-13T07:43:10.193 DEBUG:boto:Path: /yxvvuh-21/obj 2022-09-13T07:43:10.194 DEBUG:boto:Data: 2022-09-13T07:43:10.195 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.196 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:10.197 DEBUG:boto:Port: 8001 2022-09-13T07:43:10.197 DEBUG:boto:Params: {} 2022-09-13T07:43:10.198 DEBUG:boto:Token: None 2022-09-13T07:43:10.198 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/obj 2022-09-13T07:43:10.199 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AoJo2CfUNiW1Ot8yjMJ7+KWpiGQ= 2022-09-13T07:43:10.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AoJo2CfUNiW1Ot8yjMJ7+KWpiGQ=', 'Content-Length': '0'} 2022-09-13T07:43:10.202 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000013a32de12da5bbd4-006320348e-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.202 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:10.203 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:10.204 DEBUG:boto:Method: HEAD 2022-09-13T07:43:10.204 DEBUG:boto:Path: /yxvvuh-21/obj 2022-09-13T07:43:10.204 DEBUG:boto:Data: 2022-09-13T07:43:10.205 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.205 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:10.206 DEBUG:boto:Port: 8000 2022-09-13T07:43:10.206 DEBUG:boto:Params: {} 2022-09-13T07:43:10.207 DEBUG:boto:Token: None 2022-09-13T07:43:10.208 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/obj 2022-09-13T07:43:10.208 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AoJo2CfUNiW1Ot8yjMJ7+KWpiGQ= 2022-09-13T07:43:10.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AoJo2CfUNiW1Ot8yjMJ7+KWpiGQ=', 'Content-Length': '0'} 2022-09-13T07:43:10.211 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000002426797223451954-006320348e-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:10.212 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:10.213 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:10.213 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.214 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.215 DEBUG:boto:Method: GET 2022-09-13T07:43:10.215 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.216 DEBUG:boto:Data: 2022-09-13T07:43:10.216 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.217 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:10.217 DEBUG:boto:Port: 8001 2022-09-13T07:43:10.218 DEBUG:boto:Params: {} 2022-09-13T07:43:10.219 DEBUG:boto:Token: None 2022-09-13T07:43:10.219 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.220 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yNUPFlu1Nu2basHE5N90m/2Qdzc= 2022-09-13T07:43:10.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yNUPFlu1Nu2basHE5N90m/2Qdzc=', 'Content-Length': '0'} 2022-09-13T07:43:10.225 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bd7842ae4ee5617a-006320348e-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.226 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:10.227 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:10.227 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.229 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.230 DEBUG:boto:Method: GET 2022-09-13T07:43:10.231 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.231 DEBUG:boto:Data: 2022-09-13T07:43:10.232 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.232 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:10.233 DEBUG:boto:Port: 8000 2022-09-13T07:43:10.233 DEBUG:boto:Params: {} 2022-09-13T07:43:10.234 DEBUG:boto:Token: None 2022-09-13T07:43:10.234 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.235 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yNUPFlu1Nu2basHE5N90m/2Qdzc= 2022-09-13T07:43:10.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yNUPFlu1Nu2basHE5N90m/2Qdzc=', 'Content-Length': '0'} 2022-09-13T07:43:10.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a6999fc4c52c7460-006320348e-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.238 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:10.239 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:10.239 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.240 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.240 DEBUG:boto:Method: HEAD 2022-09-13T07:43:10.241 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.241 DEBUG:boto:Data: 2022-09-13T07:43:10.242 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.242 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:10.243 DEBUG:boto:Port: 8001 2022-09-13T07:43:10.243 DEBUG:boto:Params: {} 2022-09-13T07:43:10.244 DEBUG:boto:Token: None 2022-09-13T07:43:10.244 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.245 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tM2ZDoHJuLwDEYCvf27GHCX3wPk= 2022-09-13T07:43:10.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tM2ZDoHJuLwDEYCvf27GHCX3wPk=', 'Content-Length': '0'} 2022-09-13T07:43:10.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008eb35e1a0902df6e-006320348e-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.248 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:10.249 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:10.249 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.250 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.250 DEBUG:boto:Method: HEAD 2022-09-13T07:43:10.251 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.251 DEBUG:boto:Data: 2022-09-13T07:43:10.252 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.252 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:10.253 DEBUG:boto:Port: 8000 2022-09-13T07:43:10.254 DEBUG:boto:Params: {} 2022-09-13T07:43:10.254 DEBUG:boto:Token: None 2022-09-13T07:43:10.255 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.256 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tM2ZDoHJuLwDEYCvf27GHCX3wPk= 2022-09-13T07:43:10.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tM2ZDoHJuLwDEYCvf27GHCX3wPk=', 'Content-Length': '0'} 2022-09-13T07:43:10.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002fe51c5995bcb2cb-006320348e-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:10.260 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:10.260 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:10.261 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.261 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.262 DEBUG:boto:Method: GET 2022-09-13T07:43:10.262 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.263 DEBUG:boto:Data: 2022-09-13T07:43:10.263 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.264 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:10.264 DEBUG:boto:Port: 8001 2022-09-13T07:43:10.265 DEBUG:boto:Params: {} 2022-09-13T07:43:10.265 DEBUG:boto:Token: None 2022-09-13T07:43:10.266 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.267 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yNUPFlu1Nu2basHE5N90m/2Qdzc= 2022-09-13T07:43:10.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yNUPFlu1Nu2basHE5N90m/2Qdzc=', 'Content-Length': '0'} 2022-09-13T07:43:10.269 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a70440a274508efa-006320348e-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.270 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:10.271 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:10.272 DEBUG:boto:path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.272 DEBUG:boto:auth_path=/yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.273 DEBUG:boto:Method: GET 2022-09-13T07:43:10.273 DEBUG:boto:Path: /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.274 DEBUG:boto:Data: 2022-09-13T07:43:10.274 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.275 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:10.275 DEBUG:boto:Port: 8000 2022-09-13T07:43:10.276 DEBUG:boto:Params: {} 2022-09-13T07:43:10.277 DEBUG:boto:Token: None 2022-09-13T07:43:10.277 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/obj?versionId=mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb 2022-09-13T07:43:10.278 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yNUPFlu1Nu2basHE5N90m/2Qdzc= 2022-09-13T07:43:10.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yNUPFlu1Nu2basHE5N90m/2Qdzc=', 'Content-Length': '0'} 2022-09-13T07:43:10.280 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'mjSn6bHwJuQF8pQ89ZmDszGZ5uwkqCb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fd1862d3a3a2e307-006320348e-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.281 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:10.282 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:10.283 DEBUG:boto:path=/yxvvuh-21/?prefix=obj 2022-09-13T07:43:10.283 DEBUG:boto:auth_path=/yxvvuh-21/?prefix=obj 2022-09-13T07:43:10.284 DEBUG:boto:Method: GET 2022-09-13T07:43:10.285 DEBUG:boto:Path: /yxvvuh-21/?prefix=obj 2022-09-13T07:43:10.286 DEBUG:boto:Data: 2022-09-13T07:43:10.287 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.288 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:10.288 DEBUG:boto:Port: 8001 2022-09-13T07:43:10.289 DEBUG:boto:Params: {} 2022-09-13T07:43:10.290 DEBUG:boto:Token: None 2022-09-13T07:43:10.291 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/ 2022-09-13T07:43:10.293 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dWV2Hup5Uwvp7xuHGbJ6R/wLSds= 2022-09-13T07:43:10.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dWV2Hup5Uwvp7xuHGbJ6R/wLSds=', 'Content-Length': '0'} 2022-09-13T07:43:10.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a926c06b39ad1a9f-006320348e-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.297 DEBUG:boto:b'yxvvuh-21obj1000false' 2022-09-13T07:43:10.299 DEBUG:boto:path=/yxvvuh-21/ 2022-09-13T07:43:10.300 DEBUG:boto:auth_path=/yxvvuh-21/ 2022-09-13T07:43:10.300 DEBUG:boto:path=/yxvvuh-21/?prefix=obj 2022-09-13T07:43:10.302 DEBUG:boto:auth_path=/yxvvuh-21/?prefix=obj 2022-09-13T07:43:10.303 DEBUG:boto:Method: GET 2022-09-13T07:43:10.304 DEBUG:boto:Path: /yxvvuh-21/?prefix=obj 2022-09-13T07:43:10.305 DEBUG:boto:Data: 2022-09-13T07:43:10.306 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.307 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:10.307 DEBUG:boto:Port: 8000 2022-09-13T07:43:10.309 DEBUG:boto:Params: {} 2022-09-13T07:43:10.311 DEBUG:boto:Token: None 2022-09-13T07:43:10.311 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/ 2022-09-13T07:43:10.312 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dWV2Hup5Uwvp7xuHGbJ6R/wLSds= 2022-09-13T07:43:10.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dWV2Hup5Uwvp7xuHGbJ6R/wLSds=', 'Content-Length': '0'} 2022-09-13T07:43:10.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfa3a90524cf7dda-006320348e-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.358 DEBUG:boto:b'yxvvuh-21obj1000false' 2022-09-13T07:43:10.360 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:10.361 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:10.361 DEBUG:boto:Method: HEAD 2022-09-13T07:43:10.362 DEBUG:boto:Path: /yxvvuh-21/obj 2022-09-13T07:43:10.363 DEBUG:boto:Data: 2022-09-13T07:43:10.364 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.365 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:10.366 DEBUG:boto:Port: 8001 2022-09-13T07:43:10.366 DEBUG:boto:Params: {} 2022-09-13T07:43:10.367 DEBUG:boto:Token: None 2022-09-13T07:43:10.368 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/obj 2022-09-13T07:43:10.369 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AoJo2CfUNiW1Ot8yjMJ7+KWpiGQ= 2022-09-13T07:43:10.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AoJo2CfUNiW1Ot8yjMJ7+KWpiGQ=', 'Content-Length': '0'} 2022-09-13T07:43:10.374 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000950ea483146cfb4d-006320348e-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.375 DEBUG:boto:path=/yxvvuh-21/obj 2022-09-13T07:43:10.376 DEBUG:boto:auth_path=/yxvvuh-21/obj 2022-09-13T07:43:10.376 DEBUG:boto:Method: HEAD 2022-09-13T07:43:10.377 DEBUG:boto:Path: /yxvvuh-21/obj 2022-09-13T07:43:10.378 DEBUG:boto:Data: 2022-09-13T07:43:10.379 DEBUG:boto:Headers: {} 2022-09-13T07:43:10.380 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:10.381 DEBUG:boto:Port: 8000 2022-09-13T07:43:10.382 DEBUG:boto:Params: {} 2022-09-13T07:43:10.383 DEBUG:boto:Token: None 2022-09-13T07:43:10.384 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:10 GMT /yxvvuh-21/obj 2022-09-13T07:43:10.385 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:AoJo2CfUNiW1Ot8yjMJ7+KWpiGQ= 2022-09-13T07:43:10.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:AoJo2CfUNiW1Ot8yjMJ7+KWpiGQ=', 'Content-Length': '0'} 2022-09-13T07:43:10.388 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000044d96d5729d2047-006320348e-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:10.390 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-21 zones={test-zone3, test-zone2} 2022-09-13T07:43:10.391 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-22 2022-09-13T07:43:10.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:10.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:10.392 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:43:10.457 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:10.457 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:10.497 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:10.493+0000 7fc76df42600 1 bucket sync caught up with source: 2022-09-13T07:43:10.498 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000002.10.5, , ] 2022-09-13T07:43:10.498 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000002.10.5, , ] 2022-09-13T07:43:10.499 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:10.493+0000 7fc76df42600 0 bucket checkpoint complete 2022-09-13T07:43:10.509 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-22 2022-09-13T07:43:10.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:10.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:10.512 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:43:10.580 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:10.580 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:10.632 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:10.628+0000 7ff23a9b3600 1 bucket sync caught up with source: 2022-09-13T07:43:10.640 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000002.10.5, , ] 2022-09-13T07:43:10.641 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000002.10.5, , ] 2022-09-13T07:43:10.641 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:10.628+0000 7ff23a9b3600 0 bucket checkpoint complete 2022-09-13T07:43:10.652 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-22 2022-09-13T07:43:10.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:10.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:10.654 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:43:10.712 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:10.713 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:10.761 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:10.757+0000 7f66b46d7600 1 bucket sync caught up with source: 2022-09-13T07:43:10.761 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000003.29.11, , ] 2022-09-13T07:43:10.762 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000003.29.11, , ] 2022-09-13T07:43:10.762 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:10.757+0000 7f66b46d7600 0 bucket checkpoint complete 2022-09-13T07:43:10.778 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-22 2022-09-13T07:43:10.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:10.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:10.780 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:43:10.837 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:10.838 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:10.887 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:10.883+0000 7f64d8913600 1 bucket sync caught up with source: 2022-09-13T07:43:10.888 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000003.29.11, , ] 2022-09-13T07:43:10.888 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000003.29.11, , ] 2022-09-13T07:43:10.889 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:10.883+0000 7f64d8913600 0 bucket checkpoint complete 2022-09-13T07:43:10.901 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-22 2022-09-13T07:43:10.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:10.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:10.903 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:43:10.960 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:10.961 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:11.005 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:11.001+0000 7f684ec0b600 1 bucket sync caught up with source: 2022-09-13T07:43:11.005 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000002.52.5, , ] 2022-09-13T07:43:11.005 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000002.52.5, , ] 2022-09-13T07:43:11.006 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:11.001+0000 7f684ec0b600 0 bucket checkpoint complete 2022-09-13T07:43:11.020 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-22 2022-09-13T07:43:11.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:11.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:11.024 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:43:11.084 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:11.084 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:11.142 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:11.138+0000 7fe9f6247600 1 bucket sync caught up with source: 2022-09-13T07:43:11.142 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000002.52.5, , ] 2022-09-13T07:43:11.142 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000002.52.5, , ] 2022-09-13T07:43:11.143 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:11.138+0000 7fe9f6247600 0 bucket checkpoint complete 2022-09-13T07:43:11.165 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-22 zones={test-zone2, test-zone1} 2022-09-13T07:43:11.167 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.168 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.168 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.169 DEBUG:boto:Path: /yxvvuh-22/ 2022-09-13T07:43:11.170 DEBUG:boto:Data: 2022-09-13T07:43:11.171 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.171 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.173 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.173 DEBUG:boto:Params: {} 2022-09-13T07:43:11.174 DEBUG:boto:Token: None 2022-09-13T07:43:11.174 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/ 2022-09-13T07:43:11.175 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HRzoz4po3/aGRW6AZsf7kDC+2rE= 2022-09-13T07:43:11.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HRzoz4po3/aGRW6AZsf7kDC+2rE=', 'Content-Length': '0'} 2022-09-13T07:43:11.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000849473e14a37f592-006320348f-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.178 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.179 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.179 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.180 DEBUG:boto:Path: /yxvvuh-22/ 2022-09-13T07:43:11.180 DEBUG:boto:Data: 2022-09-13T07:43:11.181 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.181 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.182 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.182 DEBUG:boto:Params: {} 2022-09-13T07:43:11.183 DEBUG:boto:Token: None 2022-09-13T07:43:11.183 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/ 2022-09-13T07:43:11.184 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HRzoz4po3/aGRW6AZsf7kDC+2rE= 2022-09-13T07:43:11.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HRzoz4po3/aGRW6AZsf7kDC+2rE=', 'Content-Length': '0'} 2022-09-13T07:43:11.186 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081b63f79412b9491-006320348f-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.187 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:43:11.187 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.188 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.188 DEBUG:boto:path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.189 DEBUG:boto:auth_path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.189 DEBUG:boto:Method: GET 2022-09-13T07:43:11.190 DEBUG:boto:Path: /yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.190 DEBUG:boto:Data: 2022-09-13T07:43:11.191 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.191 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.192 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.192 DEBUG:boto:Params: {} 2022-09-13T07:43:11.193 DEBUG:boto:Token: None 2022-09-13T07:43:11.194 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/?versions 2022-09-13T07:43:11.194 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U= 2022-09-13T07:43:11.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U=', 'Content-Length': '0'} 2022-09-13T07:43:11.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cb2bd9382ae08b1-006320348f-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.199 DEBUG:boto:b'yxvvuh-22999falseobjRdNIBMozGNVPQgbGp.aZdKuuPHm5I6Htrue2022-09-13T07:42:28.560Zrgw-multisite-test-userMultisite Test UserNormalobjr7owBR4UCa9o6NAg0y3qMJWOO8M4HFqfalse2022-09-13T07:42:28.489Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:11.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:11.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:11.207 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:43:11.208 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.208 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.209 DEBUG:boto:path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.210 DEBUG:boto:auth_path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.211 DEBUG:boto:Method: GET 2022-09-13T07:43:11.211 DEBUG:boto:Path: /yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.212 DEBUG:boto:Data: 2022-09-13T07:43:11.212 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.213 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.213 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.214 DEBUG:boto:Params: {} 2022-09-13T07:43:11.215 DEBUG:boto:Token: None 2022-09-13T07:43:11.216 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/?versions 2022-09-13T07:43:11.217 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U= 2022-09-13T07:43:11.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U=', 'Content-Length': '0'} 2022-09-13T07:43:11.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c25866f76a49b4f-006320348f-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.262 DEBUG:boto:b'yxvvuh-22999falseobjRdNIBMozGNVPQgbGp.aZdKuuPHm5I6Htrue2022-09-13T07:42:28.560Zrgw-multisite-test-userMultisite Test UserNormalobjr7owBR4UCa9o6NAg0y3qMJWOO8M4HFqfalse2022-09-13T07:42:28.489Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:11.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:11.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:11.265 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.266 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.267 DEBUG:boto:path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.267 DEBUG:boto:auth_path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.268 DEBUG:boto:Method: GET 2022-09-13T07:43:11.269 DEBUG:boto:Path: /yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.269 DEBUG:boto:Data: 2022-09-13T07:43:11.270 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.270 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.271 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.271 DEBUG:boto:Params: {} 2022-09-13T07:43:11.272 DEBUG:boto:Token: None 2022-09-13T07:43:11.273 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/?versions 2022-09-13T07:43:11.273 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U= 2022-09-13T07:43:11.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U=', 'Content-Length': '0'} 2022-09-13T07:43:11.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e47264f912c05770-006320348f-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.280 DEBUG:boto:b'yxvvuh-22999falseobjRdNIBMozGNVPQgbGp.aZdKuuPHm5I6Htrue2022-09-13T07:42:28.560Zrgw-multisite-test-userMultisite Test UserNormalobjr7owBR4UCa9o6NAg0y3qMJWOO8M4HFqfalse2022-09-13T07:42:28.489Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:11.282 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.283 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.284 DEBUG:boto:path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.285 DEBUG:boto:auth_path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.286 DEBUG:boto:Method: GET 2022-09-13T07:43:11.287 DEBUG:boto:Path: /yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.288 DEBUG:boto:Data: 2022-09-13T07:43:11.289 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.289 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.290 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.291 DEBUG:boto:Params: {} 2022-09-13T07:43:11.293 DEBUG:boto:Token: None 2022-09-13T07:43:11.293 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/?versions 2022-09-13T07:43:11.294 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U= 2022-09-13T07:43:11.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U=', 'Content-Length': '0'} 2022-09-13T07:43:11.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4cfb78aeefaec04-006320348f-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.298 DEBUG:boto:b'yxvvuh-22999falseobjRdNIBMozGNVPQgbGp.aZdKuuPHm5I6Htrue2022-09-13T07:42:28.560Zrgw-multisite-test-userMultisite Test UserNormalobjr7owBR4UCa9o6NAg0y3qMJWOO8M4HFqfalse2022-09-13T07:42:28.489Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:11.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:11.300 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.300 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.301 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.301 DEBUG:boto:Path: /yxvvuh-22/obj 2022-09-13T07:43:11.302 DEBUG:boto:Data: 2022-09-13T07:43:11.302 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.303 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.304 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.304 DEBUG:boto:Params: {} 2022-09-13T07:43:11.305 DEBUG:boto:Token: None 2022-09-13T07:43:11.306 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj 2022-09-13T07:43:11.306 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg= 2022-09-13T07:43:11.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg=', 'Content-Length': '0'} 2022-09-13T07:43:11.309 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000005d1b24f84dc53f6e-006320348f-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.310 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.310 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.311 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.311 DEBUG:boto:Path: /yxvvuh-22/obj 2022-09-13T07:43:11.312 DEBUG:boto:Data: 2022-09-13T07:43:11.313 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.313 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.314 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.314 DEBUG:boto:Params: {} 2022-09-13T07:43:11.315 DEBUG:boto:Token: None 2022-09-13T07:43:11.315 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj 2022-09-13T07:43:11.316 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg= 2022-09-13T07:43:11.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg=', 'Content-Length': '0'} 2022-09-13T07:43:11.319 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000af4f0ef5a184b479-006320348f-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:11.320 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.320 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.321 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.321 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.322 DEBUG:boto:Method: GET 2022-09-13T07:43:11.323 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.323 DEBUG:boto:Data: 2022-09-13T07:43:11.324 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.324 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.325 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.325 DEBUG:boto:Params: {} 2022-09-13T07:43:11.326 DEBUG:boto:Token: None 2022-09-13T07:43:11.326 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.327 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk= 2022-09-13T07:43:11.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk=', 'Content-Length': '0'} 2022-09-13T07:43:11.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000077b66ebaf5677617-006320348f-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.331 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.331 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.332 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.332 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.333 DEBUG:boto:Method: GET 2022-09-13T07:43:11.333 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.334 DEBUG:boto:Data: 2022-09-13T07:43:11.334 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.335 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.335 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.336 DEBUG:boto:Params: {} 2022-09-13T07:43:11.337 DEBUG:boto:Token: None 2022-09-13T07:43:11.337 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.338 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk= 2022-09-13T07:43:11.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk=', 'Content-Length': '0'} 2022-09-13T07:43:11.340 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000155d4829e8e4b1b4-006320348f-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.341 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.341 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.342 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.342 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.343 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.343 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.344 DEBUG:boto:Data: 2022-09-13T07:43:11.344 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.345 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.345 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.346 DEBUG:boto:Params: {} 2022-09-13T07:43:11.346 DEBUG:boto:Token: None 2022-09-13T07:43:11.347 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.347 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0mvlc0XoMVQhxA2kGusmutRtPkY= 2022-09-13T07:43:11.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0mvlc0XoMVQhxA2kGusmutRtPkY=', 'Content-Length': '0'} 2022-09-13T07:43:11.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000557458b9bd0a3204-006320348f-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.351 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.351 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.352 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.352 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.353 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.353 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.353 DEBUG:boto:Data: 2022-09-13T07:43:11.354 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.354 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.355 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.355 DEBUG:boto:Params: {} 2022-09-13T07:43:11.356 DEBUG:boto:Token: None 2022-09-13T07:43:11.356 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.357 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0mvlc0XoMVQhxA2kGusmutRtPkY= 2022-09-13T07:43:11.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0mvlc0XoMVQhxA2kGusmutRtPkY=', 'Content-Length': '0'} 2022-09-13T07:43:11.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006ea92d2c0f064c78-006320348f-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:11.361 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.362 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.362 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.363 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.363 DEBUG:boto:Method: GET 2022-09-13T07:43:11.364 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.364 DEBUG:boto:Data: 2022-09-13T07:43:11.365 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.365 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.366 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.366 DEBUG:boto:Params: {} 2022-09-13T07:43:11.367 DEBUG:boto:Token: None 2022-09-13T07:43:11.367 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.368 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk= 2022-09-13T07:43:11.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk=', 'Content-Length': '0'} 2022-09-13T07:43:11.371 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f3e728a5cac5a02-006320348f-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.371 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.372 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.372 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.373 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.373 DEBUG:boto:Method: GET 2022-09-13T07:43:11.374 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.374 DEBUG:boto:Data: 2022-09-13T07:43:11.375 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.375 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.376 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.376 DEBUG:boto:Params: {} 2022-09-13T07:43:11.377 DEBUG:boto:Token: None 2022-09-13T07:43:11.377 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.378 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk= 2022-09-13T07:43:11.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk=', 'Content-Length': '0'} 2022-09-13T07:43:11.380 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036981ef79fd25556-006320348f-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.381 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.382 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.382 DEBUG:boto:path=/yxvvuh-22/?prefix=obj 2022-09-13T07:43:11.383 DEBUG:boto:auth_path=/yxvvuh-22/?prefix=obj 2022-09-13T07:43:11.383 DEBUG:boto:Method: GET 2022-09-13T07:43:11.384 DEBUG:boto:Path: /yxvvuh-22/?prefix=obj 2022-09-13T07:43:11.384 DEBUG:boto:Data: 2022-09-13T07:43:11.385 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.386 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.387 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.388 DEBUG:boto:Params: {} 2022-09-13T07:43:11.389 DEBUG:boto:Token: None 2022-09-13T07:43:11.390 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/ 2022-09-13T07:43:11.391 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LbPpwbo4+XtjwOF5PGfO79iunS8= 2022-09-13T07:43:11.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LbPpwbo4+XtjwOF5PGfO79iunS8=', 'Content-Length': '0'} 2022-09-13T07:43:11.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002068456fb9acc544-006320348f-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.438 DEBUG:boto:b'yxvvuh-22obj1000false' 2022-09-13T07:43:11.439 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.440 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.441 DEBUG:boto:path=/yxvvuh-22/?prefix=obj 2022-09-13T07:43:11.442 DEBUG:boto:auth_path=/yxvvuh-22/?prefix=obj 2022-09-13T07:43:11.442 DEBUG:boto:Method: GET 2022-09-13T07:43:11.443 DEBUG:boto:Path: /yxvvuh-22/?prefix=obj 2022-09-13T07:43:11.443 DEBUG:boto:Data: 2022-09-13T07:43:11.444 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.445 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.445 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.446 DEBUG:boto:Params: {} 2022-09-13T07:43:11.447 DEBUG:boto:Token: None 2022-09-13T07:43:11.447 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/ 2022-09-13T07:43:11.448 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LbPpwbo4+XtjwOF5PGfO79iunS8= 2022-09-13T07:43:11.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LbPpwbo4+XtjwOF5PGfO79iunS8=', 'Content-Length': '0'} 2022-09-13T07:43:11.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014406793cf9f0334-006320348f-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.452 DEBUG:boto:b'yxvvuh-22obj1000false' 2022-09-13T07:43:11.453 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.453 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.454 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.455 DEBUG:boto:Path: /yxvvuh-22/obj 2022-09-13T07:43:11.455 DEBUG:boto:Data: 2022-09-13T07:43:11.456 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.456 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.457 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.457 DEBUG:boto:Params: {} 2022-09-13T07:43:11.458 DEBUG:boto:Token: None 2022-09-13T07:43:11.459 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj 2022-09-13T07:43:11.459 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg= 2022-09-13T07:43:11.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg=', 'Content-Length': '0'} 2022-09-13T07:43:11.462 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000a77bb0cab53b7304-006320348f-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.463 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.464 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.464 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.465 DEBUG:boto:Path: /yxvvuh-22/obj 2022-09-13T07:43:11.466 DEBUG:boto:Data: 2022-09-13T07:43:11.466 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.467 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.467 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.467 DEBUG:boto:Params: {} 2022-09-13T07:43:11.468 DEBUG:boto:Token: None 2022-09-13T07:43:11.469 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj 2022-09-13T07:43:11.469 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg= 2022-09-13T07:43:11.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg=', 'Content-Length': '0'} 2022-09-13T07:43:11.472 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000f585eee501fa0b2c-006320348f-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.472 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-22 zones={test-zone2, test-zone1} 2022-09-13T07:43:11.473 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-22 zones={test-zone3, test-zone1} 2022-09-13T07:43:11.474 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.474 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.475 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.475 DEBUG:boto:Path: /yxvvuh-22/ 2022-09-13T07:43:11.476 DEBUG:boto:Data: 2022-09-13T07:43:11.476 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.477 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.478 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.478 DEBUG:boto:Params: {} 2022-09-13T07:43:11.479 DEBUG:boto:Token: None 2022-09-13T07:43:11.480 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/ 2022-09-13T07:43:11.480 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HRzoz4po3/aGRW6AZsf7kDC+2rE= 2022-09-13T07:43:11.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HRzoz4po3/aGRW6AZsf7kDC+2rE=', 'Content-Length': '0'} 2022-09-13T07:43:11.483 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ca0afbae05fcba8-006320348f-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.484 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.485 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.486 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.488 DEBUG:boto:Path: /yxvvuh-22/ 2022-09-13T07:43:11.488 DEBUG:boto:Data: 2022-09-13T07:43:11.489 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.490 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.491 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.492 DEBUG:boto:Params: {} 2022-09-13T07:43:11.493 DEBUG:boto:Token: None 2022-09-13T07:43:11.494 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/ 2022-09-13T07:43:11.495 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HRzoz4po3/aGRW6AZsf7kDC+2rE= 2022-09-13T07:43:11.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HRzoz4po3/aGRW6AZsf7kDC+2rE=', 'Content-Length': '0'} 2022-09-13T07:43:11.499 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000988fd47ab350c496-006320348f-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.500 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:43:11.501 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.502 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.503 DEBUG:boto:path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.504 DEBUG:boto:auth_path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.505 DEBUG:boto:Method: GET 2022-09-13T07:43:11.506 DEBUG:boto:Path: /yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.507 DEBUG:boto:Data: 2022-09-13T07:43:11.508 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.508 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.509 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.510 DEBUG:boto:Params: {} 2022-09-13T07:43:11.511 DEBUG:boto:Token: None 2022-09-13T07:43:11.512 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/?versions 2022-09-13T07:43:11.514 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U= 2022-09-13T07:43:11.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U=', 'Content-Length': '0'} 2022-09-13T07:43:11.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058642348cb01ef5d-006320348f-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.519 DEBUG:boto:b'yxvvuh-22999falseobjRdNIBMozGNVPQgbGp.aZdKuuPHm5I6Htrue2022-09-13T07:42:28.560Zrgw-multisite-test-userMultisite Test UserNormalobjr7owBR4UCa9o6NAg0y3qMJWOO8M4HFqfalse2022-09-13T07:42:28.489Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:11.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:11.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:11.522 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:43:11.522 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.523 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.524 DEBUG:boto:path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.525 DEBUG:boto:auth_path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.525 DEBUG:boto:Method: GET 2022-09-13T07:43:11.526 DEBUG:boto:Path: /yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.526 DEBUG:boto:Data: 2022-09-13T07:43:11.527 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.528 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.528 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.529 DEBUG:boto:Params: {} 2022-09-13T07:43:11.530 DEBUG:boto:Token: None 2022-09-13T07:43:11.530 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/?versions 2022-09-13T07:43:11.531 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U= 2022-09-13T07:43:11.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U=', 'Content-Length': '0'} 2022-09-13T07:43:11.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0896679d72acf89-006320348f-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.578 DEBUG:boto:b'yxvvuh-22999falseobjRdNIBMozGNVPQgbGp.aZdKuuPHm5I6Htrue2022-09-13T07:42:28.560Zrgw-multisite-test-userMultisite Test UserNormalobjr7owBR4UCa9o6NAg0y3qMJWOO8M4HFqfalse2022-09-13T07:42:28.489Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:11.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:11.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:11.581 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.581 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.582 DEBUG:boto:path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.582 DEBUG:boto:auth_path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.583 DEBUG:boto:Method: GET 2022-09-13T07:43:11.584 DEBUG:boto:Path: /yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.584 DEBUG:boto:Data: 2022-09-13T07:43:11.585 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.585 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.586 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.587 DEBUG:boto:Params: {} 2022-09-13T07:43:11.587 DEBUG:boto:Token: None 2022-09-13T07:43:11.588 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/?versions 2022-09-13T07:43:11.589 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U= 2022-09-13T07:43:11.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U=', 'Content-Length': '0'} 2022-09-13T07:43:11.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005947d0a30e871a4a-006320348f-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.593 DEBUG:boto:b'yxvvuh-22999falseobjRdNIBMozGNVPQgbGp.aZdKuuPHm5I6Htrue2022-09-13T07:42:28.560Zrgw-multisite-test-userMultisite Test UserNormalobjr7owBR4UCa9o6NAg0y3qMJWOO8M4HFqfalse2022-09-13T07:42:28.489Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:11.593 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.594 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.595 DEBUG:boto:path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.595 DEBUG:boto:auth_path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.596 DEBUG:boto:Method: GET 2022-09-13T07:43:11.596 DEBUG:boto:Path: /yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.597 DEBUG:boto:Data: 2022-09-13T07:43:11.597 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.598 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.599 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.599 DEBUG:boto:Params: {} 2022-09-13T07:43:11.600 DEBUG:boto:Token: None 2022-09-13T07:43:11.601 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/?versions 2022-09-13T07:43:11.602 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U= 2022-09-13T07:43:11.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U=', 'Content-Length': '0'} 2022-09-13T07:43:11.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3f68b75eea51c04-006320348f-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.606 DEBUG:boto:b'yxvvuh-22999falseobjRdNIBMozGNVPQgbGp.aZdKuuPHm5I6Htrue2022-09-13T07:42:28.560Zrgw-multisite-test-userMultisite Test UserNormalobjr7owBR4UCa9o6NAg0y3qMJWOO8M4HFqfalse2022-09-13T07:42:28.489Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:11.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:11.610 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.611 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.611 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.612 DEBUG:boto:Path: /yxvvuh-22/obj 2022-09-13T07:43:11.612 DEBUG:boto:Data: 2022-09-13T07:43:11.613 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.613 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.614 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.614 DEBUG:boto:Params: {} 2022-09-13T07:43:11.615 DEBUG:boto:Token: None 2022-09-13T07:43:11.615 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj 2022-09-13T07:43:11.616 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg= 2022-09-13T07:43:11.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg=', 'Content-Length': '0'} 2022-09-13T07:43:11.619 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000002ed64a73320b1194-006320348f-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.619 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.620 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.621 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.622 DEBUG:boto:Path: /yxvvuh-22/obj 2022-09-13T07:43:11.623 DEBUG:boto:Data: 2022-09-13T07:43:11.624 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.625 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.625 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.626 DEBUG:boto:Params: {} 2022-09-13T07:43:11.627 DEBUG:boto:Token: None 2022-09-13T07:43:11.628 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj 2022-09-13T07:43:11.629 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg= 2022-09-13T07:43:11.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg=', 'Content-Length': '0'} 2022-09-13T07:43:11.633 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000d023abc96b3b2abb-006320348f-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:11.635 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.636 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.637 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.638 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.639 DEBUG:boto:Method: GET 2022-09-13T07:43:11.640 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.641 DEBUG:boto:Data: 2022-09-13T07:43:11.642 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.642 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.643 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.644 DEBUG:boto:Params: {} 2022-09-13T07:43:11.645 DEBUG:boto:Token: None 2022-09-13T07:43:11.646 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.647 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk= 2022-09-13T07:43:11.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk=', 'Content-Length': '0'} 2022-09-13T07:43:11.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d35bbda78a94f234-006320348f-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.652 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.653 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.654 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.655 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.656 DEBUG:boto:Method: GET 2022-09-13T07:43:11.657 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.658 DEBUG:boto:Data: 2022-09-13T07:43:11.659 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.660 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.661 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.662 DEBUG:boto:Params: {} 2022-09-13T07:43:11.663 DEBUG:boto:Token: None 2022-09-13T07:43:11.664 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.665 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk= 2022-09-13T07:43:11.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk=', 'Content-Length': '0'} 2022-09-13T07:43:11.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000156484a79ef06682-006320348f-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.669 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.670 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.670 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.671 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.671 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.672 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.672 DEBUG:boto:Data: 2022-09-13T07:43:11.673 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.673 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.674 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.674 DEBUG:boto:Params: {} 2022-09-13T07:43:11.675 DEBUG:boto:Token: None 2022-09-13T07:43:11.675 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.676 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0mvlc0XoMVQhxA2kGusmutRtPkY= 2022-09-13T07:43:11.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0mvlc0XoMVQhxA2kGusmutRtPkY=', 'Content-Length': '0'} 2022-09-13T07:43:11.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000317e0f89d064d1f4-006320348f-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.679 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.679 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.680 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.680 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.680 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.681 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.681 DEBUG:boto:Data: 2022-09-13T07:43:11.682 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.682 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.683 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.683 DEBUG:boto:Params: {} 2022-09-13T07:43:11.684 DEBUG:boto:Token: None 2022-09-13T07:43:11.684 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.685 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0mvlc0XoMVQhxA2kGusmutRtPkY= 2022-09-13T07:43:11.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0mvlc0XoMVQhxA2kGusmutRtPkY=', 'Content-Length': '0'} 2022-09-13T07:43:11.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000167745b207249425-006320348f-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:11.688 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.689 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.689 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.690 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.690 DEBUG:boto:Method: GET 2022-09-13T07:43:11.691 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.691 DEBUG:boto:Data: 2022-09-13T07:43:11.692 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.692 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.693 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.693 DEBUG:boto:Params: {} 2022-09-13T07:43:11.694 DEBUG:boto:Token: None 2022-09-13T07:43:11.695 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.695 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk= 2022-09-13T07:43:11.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk=', 'Content-Length': '0'} 2022-09-13T07:43:11.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d57735d63505bef4-006320348f-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.699 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.699 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.700 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.700 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.701 DEBUG:boto:Method: GET 2022-09-13T07:43:11.701 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.702 DEBUG:boto:Data: 2022-09-13T07:43:11.702 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.703 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.703 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.704 DEBUG:boto:Params: {} 2022-09-13T07:43:11.704 DEBUG:boto:Token: None 2022-09-13T07:43:11.705 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.705 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk= 2022-09-13T07:43:11.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk=', 'Content-Length': '0'} 2022-09-13T07:43:11.708 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000523ee155f37d7fd-006320348f-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.709 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.709 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.710 DEBUG:boto:path=/yxvvuh-22/?prefix=obj 2022-09-13T07:43:11.710 DEBUG:boto:auth_path=/yxvvuh-22/?prefix=obj 2022-09-13T07:43:11.711 DEBUG:boto:Method: GET 2022-09-13T07:43:11.712 DEBUG:boto:Path: /yxvvuh-22/?prefix=obj 2022-09-13T07:43:11.712 DEBUG:boto:Data: 2022-09-13T07:43:11.713 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.714 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.714 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.715 DEBUG:boto:Params: {} 2022-09-13T07:43:11.716 DEBUG:boto:Token: None 2022-09-13T07:43:11.717 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/ 2022-09-13T07:43:11.717 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LbPpwbo4+XtjwOF5PGfO79iunS8= 2022-09-13T07:43:11.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LbPpwbo4+XtjwOF5PGfO79iunS8=', 'Content-Length': '0'} 2022-09-13T07:43:11.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fe452632e292e09-006320348f-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.762 DEBUG:boto:b'yxvvuh-22obj1000false' 2022-09-13T07:43:11.763 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.764 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.764 DEBUG:boto:path=/yxvvuh-22/?prefix=obj 2022-09-13T07:43:11.765 DEBUG:boto:auth_path=/yxvvuh-22/?prefix=obj 2022-09-13T07:43:11.766 DEBUG:boto:Method: GET 2022-09-13T07:43:11.767 DEBUG:boto:Path: /yxvvuh-22/?prefix=obj 2022-09-13T07:43:11.767 DEBUG:boto:Data: 2022-09-13T07:43:11.768 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.768 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.769 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.769 DEBUG:boto:Params: {} 2022-09-13T07:43:11.770 DEBUG:boto:Token: None 2022-09-13T07:43:11.771 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/ 2022-09-13T07:43:11.771 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LbPpwbo4+XtjwOF5PGfO79iunS8= 2022-09-13T07:43:11.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LbPpwbo4+XtjwOF5PGfO79iunS8=', 'Content-Length': '0'} 2022-09-13T07:43:11.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086ad33538f56e652-006320348f-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.775 DEBUG:boto:b'yxvvuh-22obj1000false' 2022-09-13T07:43:11.776 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.777 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.777 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.778 DEBUG:boto:Path: /yxvvuh-22/obj 2022-09-13T07:43:11.778 DEBUG:boto:Data: 2022-09-13T07:43:11.779 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.779 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.780 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.780 DEBUG:boto:Params: {} 2022-09-13T07:43:11.781 DEBUG:boto:Token: None 2022-09-13T07:43:11.781 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj 2022-09-13T07:43:11.782 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg= 2022-09-13T07:43:11.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg=', 'Content-Length': '0'} 2022-09-13T07:43:11.785 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000068ec6b4dcf4160c4-006320348f-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.786 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.787 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.788 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.789 DEBUG:boto:Path: /yxvvuh-22/obj 2022-09-13T07:43:11.789 DEBUG:boto:Data: 2022-09-13T07:43:11.790 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.791 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:11.792 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.793 DEBUG:boto:Params: {} 2022-09-13T07:43:11.793 DEBUG:boto:Token: None 2022-09-13T07:43:11.794 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj 2022-09-13T07:43:11.794 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg= 2022-09-13T07:43:11.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg=', 'Content-Length': '0'} 2022-09-13T07:43:11.797 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000db1e3304cad9bfb1-006320348f-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.798 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-22 zones={test-zone3, test-zone1} 2022-09-13T07:43:11.799 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-22 zones={test-zone3, test-zone2} 2022-09-13T07:43:11.799 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.800 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.801 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.801 DEBUG:boto:Path: /yxvvuh-22/ 2022-09-13T07:43:11.802 DEBUG:boto:Data: 2022-09-13T07:43:11.803 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.804 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.805 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.805 DEBUG:boto:Params: {} 2022-09-13T07:43:11.806 DEBUG:boto:Token: None 2022-09-13T07:43:11.807 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/ 2022-09-13T07:43:11.808 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HRzoz4po3/aGRW6AZsf7kDC+2rE= 2022-09-13T07:43:11.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HRzoz4po3/aGRW6AZsf7kDC+2rE=', 'Content-Length': '0'} 2022-09-13T07:43:11.811 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b48806fd4ae88425-006320348f-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.812 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.812 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.813 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.813 DEBUG:boto:Path: /yxvvuh-22/ 2022-09-13T07:43:11.814 DEBUG:boto:Data: 2022-09-13T07:43:11.814 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.815 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.815 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.816 DEBUG:boto:Params: {} 2022-09-13T07:43:11.816 DEBUG:boto:Token: None 2022-09-13T07:43:11.817 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/ 2022-09-13T07:43:11.817 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:HRzoz4po3/aGRW6AZsf7kDC+2rE= 2022-09-13T07:43:11.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HRzoz4po3/aGRW6AZsf7kDC+2rE=', 'Content-Length': '0'} 2022-09-13T07:43:11.820 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b03dfeacab6657a-006320348f-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.821 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:43:11.822 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.822 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.823 DEBUG:boto:path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.823 DEBUG:boto:auth_path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.824 DEBUG:boto:Method: GET 2022-09-13T07:43:11.824 DEBUG:boto:Path: /yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.824 DEBUG:boto:Data: 2022-09-13T07:43:11.825 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.825 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.826 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.826 DEBUG:boto:Params: {} 2022-09-13T07:43:11.827 DEBUG:boto:Token: None 2022-09-13T07:43:11.828 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/?versions 2022-09-13T07:43:11.828 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U= 2022-09-13T07:43:11.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U=', 'Content-Length': '0'} 2022-09-13T07:43:11.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ac0fc585cec94f3-006320348f-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.832 DEBUG:boto:b'yxvvuh-22999falseobjRdNIBMozGNVPQgbGp.aZdKuuPHm5I6Htrue2022-09-13T07:42:28.560Zrgw-multisite-test-userMultisite Test UserNormalobjr7owBR4UCa9o6NAg0y3qMJWOO8M4HFqfalse2022-09-13T07:42:28.489Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:11.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:11.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:11.834 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:43:11.834 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.835 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.835 DEBUG:boto:path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.836 DEBUG:boto:auth_path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.836 DEBUG:boto:Method: GET 2022-09-13T07:43:11.837 DEBUG:boto:Path: /yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.837 DEBUG:boto:Data: 2022-09-13T07:43:11.838 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.838 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.839 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.839 DEBUG:boto:Params: {} 2022-09-13T07:43:11.840 DEBUG:boto:Token: None 2022-09-13T07:43:11.840 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/?versions 2022-09-13T07:43:11.841 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U= 2022-09-13T07:43:11.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U=', 'Content-Length': '0'} 2022-09-13T07:43:11.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000555a1abd47bc116c-006320348f-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.845 DEBUG:boto:b'yxvvuh-22999falseobjRdNIBMozGNVPQgbGp.aZdKuuPHm5I6Htrue2022-09-13T07:42:28.560Zrgw-multisite-test-userMultisite Test UserNormalobjr7owBR4UCa9o6NAg0y3qMJWOO8M4HFqfalse2022-09-13T07:42:28.489Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:11.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:11.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:11.847 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.847 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.848 DEBUG:boto:path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.848 DEBUG:boto:auth_path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.849 DEBUG:boto:Method: GET 2022-09-13T07:43:11.849 DEBUG:boto:Path: /yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.850 DEBUG:boto:Data: 2022-09-13T07:43:11.850 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.850 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.851 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.852 DEBUG:boto:Params: {} 2022-09-13T07:43:11.852 DEBUG:boto:Token: None 2022-09-13T07:43:11.853 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/?versions 2022-09-13T07:43:11.853 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U= 2022-09-13T07:43:11.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U=', 'Content-Length': '0'} 2022-09-13T07:43:11.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cf2078917d3a4b7-006320348f-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.898 DEBUG:boto:b'yxvvuh-22999falseobjRdNIBMozGNVPQgbGp.aZdKuuPHm5I6Htrue2022-09-13T07:42:28.560Zrgw-multisite-test-userMultisite Test UserNormalobjr7owBR4UCa9o6NAg0y3qMJWOO8M4HFqfalse2022-09-13T07:42:28.489Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:11.900 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.900 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.901 DEBUG:boto:path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.902 DEBUG:boto:auth_path=/yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.903 DEBUG:boto:Method: GET 2022-09-13T07:43:11.903 DEBUG:boto:Path: /yxvvuh-22/?versions&max-keys=999 2022-09-13T07:43:11.904 DEBUG:boto:Data: 2022-09-13T07:43:11.904 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.905 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.905 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.906 DEBUG:boto:Params: {} 2022-09-13T07:43:11.906 DEBUG:boto:Token: None 2022-09-13T07:43:11.907 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/?versions 2022-09-13T07:43:11.907 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U= 2022-09-13T07:43:11.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dXCwtC1D78kn2ltu6u2tB/Gqr+U=', 'Content-Length': '0'} 2022-09-13T07:43:11.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2cf9af1ec506b26-006320348f-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.911 DEBUG:boto:b'yxvvuh-22999falseobjRdNIBMozGNVPQgbGp.aZdKuuPHm5I6Htrue2022-09-13T07:42:28.560Zrgw-multisite-test-userMultisite Test UserNormalobjr7owBR4UCa9o6NAg0y3qMJWOO8M4HFqfalse2022-09-13T07:42:28.489Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:11.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:11.912 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.913 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.913 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.914 DEBUG:boto:Path: /yxvvuh-22/obj 2022-09-13T07:43:11.914 DEBUG:boto:Data: 2022-09-13T07:43:11.915 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.915 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.916 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.916 DEBUG:boto:Params: {} 2022-09-13T07:43:11.917 DEBUG:boto:Token: None 2022-09-13T07:43:11.917 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj 2022-09-13T07:43:11.918 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg= 2022-09-13T07:43:11.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg=', 'Content-Length': '0'} 2022-09-13T07:43:11.921 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000009e0cfe4055deedce-006320348f-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.922 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.923 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.923 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.923 DEBUG:boto:Path: /yxvvuh-22/obj 2022-09-13T07:43:11.924 DEBUG:boto:Data: 2022-09-13T07:43:11.924 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.925 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.925 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.926 DEBUG:boto:Params: {} 2022-09-13T07:43:11.927 DEBUG:boto:Token: None 2022-09-13T07:43:11.927 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj 2022-09-13T07:43:11.928 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg= 2022-09-13T07:43:11.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5GgGBeYDyWTH28Yg89BSONKHkSg=', 'Content-Length': '0'} 2022-09-13T07:43:11.931 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000fcbbeddbb90e2af8-006320348f-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:11.932 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.932 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.933 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.933 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.934 DEBUG:boto:Method: GET 2022-09-13T07:43:11.935 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.935 DEBUG:boto:Data: 2022-09-13T07:43:11.936 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.937 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.937 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.938 DEBUG:boto:Params: {} 2022-09-13T07:43:11.938 DEBUG:boto:Token: None 2022-09-13T07:43:11.939 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.939 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk= 2022-09-13T07:43:11.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk=', 'Content-Length': '0'} 2022-09-13T07:43:11.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000982f163e9600d135-006320348f-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.943 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.944 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.944 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.945 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.945 DEBUG:boto:Method: GET 2022-09-13T07:43:11.946 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.946 DEBUG:boto:Data: 2022-09-13T07:43:11.947 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.948 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.948 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.949 DEBUG:boto:Params: {} 2022-09-13T07:43:11.949 DEBUG:boto:Token: None 2022-09-13T07:43:11.950 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.951 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk= 2022-09-13T07:43:11.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk=', 'Content-Length': '0'} 2022-09-13T07:43:11.953 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008788122c4ea41b47-006320348f-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.955 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.955 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.956 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.956 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.957 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.957 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.958 DEBUG:boto:Data: 2022-09-13T07:43:11.958 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.959 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.959 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.960 DEBUG:boto:Params: {} 2022-09-13T07:43:11.961 DEBUG:boto:Token: None 2022-09-13T07:43:11.961 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.962 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0mvlc0XoMVQhxA2kGusmutRtPkY= 2022-09-13T07:43:11.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0mvlc0XoMVQhxA2kGusmutRtPkY=', 'Content-Length': '0'} 2022-09-13T07:43:11.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b43b40726b1dfea8-006320348f-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.965 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.966 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.966 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.967 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.967 DEBUG:boto:Method: HEAD 2022-09-13T07:43:11.968 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.968 DEBUG:boto:Data: 2022-09-13T07:43:11.969 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.969 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.970 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.970 DEBUG:boto:Params: {} 2022-09-13T07:43:11.971 DEBUG:boto:Token: None 2022-09-13T07:43:11.971 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.972 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0mvlc0XoMVQhxA2kGusmutRtPkY= 2022-09-13T07:43:11.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0mvlc0XoMVQhxA2kGusmutRtPkY=', 'Content-Length': '0'} 2022-09-13T07:43:11.975 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006c067d660c0d2888-006320348f-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:11.976 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.976 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.977 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.977 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.978 DEBUG:boto:Method: GET 2022-09-13T07:43:11.978 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.979 DEBUG:boto:Data: 2022-09-13T07:43:11.979 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.980 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:11.980 DEBUG:boto:Port: 8001 2022-09-13T07:43:11.981 DEBUG:boto:Params: {} 2022-09-13T07:43:11.981 DEBUG:boto:Token: None 2022-09-13T07:43:11.982 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.983 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk= 2022-09-13T07:43:11.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk=', 'Content-Length': '0'} 2022-09-13T07:43:11.985 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002944b7e297f6bc99-006320348f-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.986 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:11.986 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:11.987 DEBUG:boto:path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.987 DEBUG:boto:auth_path=/yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.988 DEBUG:boto:Method: GET 2022-09-13T07:43:11.988 DEBUG:boto:Path: /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.989 DEBUG:boto:Data: 2022-09-13T07:43:11.989 DEBUG:boto:Headers: {} 2022-09-13T07:43:11.990 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:11.990 DEBUG:boto:Port: 8000 2022-09-13T07:43:11.991 DEBUG:boto:Params: {} 2022-09-13T07:43:11.991 DEBUG:boto:Token: None 2022-09-13T07:43:11.992 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:11 GMT /yxvvuh-22/obj?versionId=r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq 2022-09-13T07:43:11.993 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk= 2022-09-13T07:43:11.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hAFxOzXJ790yS7gnpeC96LypLwk=', 'Content-Length': '0'} 2022-09-13T07:43:11.995 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'r7owBR4UCa9o6NAg0y3qMJWOO8M4HFq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f04c80149bae5fd-006320348f-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:11.996 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:11.996 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:11.997 DEBUG:boto:path=/yxvvuh-22/?prefix=obj 2022-09-13T07:43:11.997 DEBUG:boto:auth_path=/yxvvuh-22/?prefix=obj 2022-09-13T07:43:11.998 DEBUG:boto:Method: GET 2022-09-13T07:43:11.998 DEBUG:boto:Path: /yxvvuh-22/?prefix=obj 2022-09-13T07:43:11.999 DEBUG:boto:Data: 2022-09-13T07:43:11.999 DEBUG:boto:Headers: {} 2022-09-13T07:43:12.000 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:12.000 DEBUG:boto:Port: 8001 2022-09-13T07:43:12.001 DEBUG:boto:Params: {} 2022-09-13T07:43:12.002 DEBUG:boto:Token: None 2022-09-13T07:43:12.002 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:12 GMT /yxvvuh-22/ 2022-09-13T07:43:12.003 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iXACAfeh4U0c+nG8XIE4BQyWk38= 2022-09-13T07:43:12.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iXACAfeh4U0c+nG8XIE4BQyWk38=', 'Content-Length': '0'} 2022-09-13T07:43:12.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000444ccf158c2d295e-0063203490-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:12.007 DEBUG:boto:b'yxvvuh-22obj1000false' 2022-09-13T07:43:12.008 DEBUG:boto:path=/yxvvuh-22/ 2022-09-13T07:43:12.008 DEBUG:boto:auth_path=/yxvvuh-22/ 2022-09-13T07:43:12.009 DEBUG:boto:path=/yxvvuh-22/?prefix=obj 2022-09-13T07:43:12.009 DEBUG:boto:auth_path=/yxvvuh-22/?prefix=obj 2022-09-13T07:43:12.010 DEBUG:boto:Method: GET 2022-09-13T07:43:12.010 DEBUG:boto:Path: /yxvvuh-22/?prefix=obj 2022-09-13T07:43:12.011 DEBUG:boto:Data: 2022-09-13T07:43:12.012 DEBUG:boto:Headers: {} 2022-09-13T07:43:12.013 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:12.014 DEBUG:boto:Port: 8000 2022-09-13T07:43:12.014 DEBUG:boto:Params: {} 2022-09-13T07:43:12.015 DEBUG:boto:Token: None 2022-09-13T07:43:12.016 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:12 GMT /yxvvuh-22/ 2022-09-13T07:43:12.018 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iXACAfeh4U0c+nG8XIE4BQyWk38= 2022-09-13T07:43:12.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iXACAfeh4U0c+nG8XIE4BQyWk38=', 'Content-Length': '0'} 2022-09-13T07:43:12.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008550e7f17015f499-0063203490-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:12.062 DEBUG:boto:b'yxvvuh-22obj1000false' 2022-09-13T07:43:12.063 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:12.064 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:12.064 DEBUG:boto:Method: HEAD 2022-09-13T07:43:12.065 DEBUG:boto:Path: /yxvvuh-22/obj 2022-09-13T07:43:12.066 DEBUG:boto:Data: 2022-09-13T07:43:12.066 DEBUG:boto:Headers: {} 2022-09-13T07:43:12.067 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:12.067 DEBUG:boto:Port: 8001 2022-09-13T07:43:12.068 DEBUG:boto:Params: {} 2022-09-13T07:43:12.069 DEBUG:boto:Token: None 2022-09-13T07:43:12.070 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:12 GMT /yxvvuh-22/obj 2022-09-13T07:43:12.071 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:59ZQS5TSeh90HM2sgMFacVIebqk= 2022-09-13T07:43:12.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:59ZQS5TSeh90HM2sgMFacVIebqk=', 'Content-Length': '0'} 2022-09-13T07:43:12.074 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000002310a567a7746dae-0063203490-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:12.075 DEBUG:boto:path=/yxvvuh-22/obj 2022-09-13T07:43:12.076 DEBUG:boto:auth_path=/yxvvuh-22/obj 2022-09-13T07:43:12.077 DEBUG:boto:Method: HEAD 2022-09-13T07:43:12.077 DEBUG:boto:Path: /yxvvuh-22/obj 2022-09-13T07:43:12.078 DEBUG:boto:Data: 2022-09-13T07:43:12.078 DEBUG:boto:Headers: {} 2022-09-13T07:43:12.079 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:12.079 DEBUG:boto:Port: 8000 2022-09-13T07:43:12.080 DEBUG:boto:Params: {} 2022-09-13T07:43:12.081 DEBUG:boto:Token: None 2022-09-13T07:43:12.082 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:12 GMT /yxvvuh-22/obj 2022-09-13T07:43:12.083 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:59ZQS5TSeh90HM2sgMFacVIebqk= 2022-09-13T07:43:12.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:12 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:59ZQS5TSeh90HM2sgMFacVIebqk=', 'Content-Length': '0'} 2022-09-13T07:43:12.087 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000007a965df1e18bc3f0-0063203490-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:12.087 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-22 zones={test-zone3, test-zone2} 2022-09-13T07:43:12.088 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-23 2022-09-13T07:43:12.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:12.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:12.090 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:43:12.152 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:12.153 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:12.212 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:12.209+0000 7f5ea2857600 1 bucket sync caught up with source: 2022-09-13T07:43:12.213 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000002.17.5, , ] 2022-09-13T07:43:12.213 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000002.17.5, , ] 2022-09-13T07:43:12.213 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:12.209+0000 7f5ea2857600 0 bucket checkpoint complete 2022-09-13T07:43:12.227 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-23 2022-09-13T07:43:12.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:12.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:12.230 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:43:12.298 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:12.299 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:12.338 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:12.333+0000 7effdd856600 1 bucket sync caught up with source: 2022-09-13T07:43:12.338 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000002.17.5, , ] 2022-09-13T07:43:12.338 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000002.17.5, , ] 2022-09-13T07:43:12.339 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:12.333+0000 7effdd856600 0 bucket checkpoint complete 2022-09-13T07:43:12.359 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-23 2022-09-13T07:43:12.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:12.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:12.361 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:43:12.420 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:12.420 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:12.470 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:12.466+0000 7f0c96773600 1 bucket sync caught up with source: 2022-09-13T07:43:12.472 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000002.51.5, , ] 2022-09-13T07:43:12.473 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000002.51.5, , ] 2022-09-13T07:43:12.473 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:12.466+0000 7f0c96773600 0 bucket checkpoint complete 2022-09-13T07:43:12.489 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-23 2022-09-13T07:43:12.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:12.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:12.492 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:43:12.550 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:12.550 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:12.594 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:12.590+0000 7f80006d0600 1 bucket sync caught up with source: 2022-09-13T07:43:12.595 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000002.51.5, , ] 2022-09-13T07:43:12.595 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000002.51.5, , ] 2022-09-13T07:43:12.596 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:12.590+0000 7f80006d0600 0 bucket checkpoint complete 2022-09-13T07:43:12.608 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-23 2022-09-13T07:43:12.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:12.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:12.611 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:43:12.669 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:12.669 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:12.709 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:12.705+0000 7f53eb5e9600 1 bucket sync caught up with source: 2022-09-13T07:43:12.710 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000003.72.11, , ] 2022-09-13T07:43:12.711 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000003.72.11, , ] 2022-09-13T07:43:12.711 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:43:12.705+0000 7f53eb5e9600 0 bucket checkpoint complete 2022-09-13T07:43:12.735 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-23 2022-09-13T07:43:12.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:12.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:43:12.737 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:43:12.794 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:12.795 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:12.829 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:12.825+0000 7fb390936600 1 waiting to reach incremental sync.. 2022-09-13T07:43:17.829 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:17.826+0000 7fb390936600 1 waiting to reach incremental sync.. 2022-09-13T07:43:22.830 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:22.825+0000 7fb390936600 1 waiting to reach incremental sync.. 2022-09-13T07:43:27.831 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:27.826+0000 7fb390936600 1 waiting to reach incremental sync.. 2022-09-13T07:43:32.831 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:32.827+0000 7fb390936600 1 waiting to reach incremental sync.. 2022-09-13T07:43:37.832 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:37.828+0000 7fb390936600 1 waiting to reach incremental sync.. 2022-09-13T07:43:42.835 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:42.828+0000 7fb390936600 1 waiting to reach incremental sync.. 2022-09-13T07:43:47.834 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:47.830+0000 7fb390936600 1 bucket sync caught up with source: 2022-09-13T07:43:47.835 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000003.72.11, , ] 2022-09-13T07:43:47.835 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000003.72.11, , ] 2022-09-13T07:43:47.835 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:47.830+0000 7fb390936600 0 bucket checkpoint complete 2022-09-13T07:43:47.851 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-23 zones={test-zone2, test-zone1} 2022-09-13T07:43:47.852 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:47.852 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:47.853 DEBUG:boto:Method: HEAD 2022-09-13T07:43:47.853 DEBUG:boto:Path: /yxvvuh-23/ 2022-09-13T07:43:47.854 DEBUG:boto:Data: 2022-09-13T07:43:47.855 DEBUG:boto:Headers: {} 2022-09-13T07:43:47.855 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:47.856 DEBUG:boto:Port: 8000 2022-09-13T07:43:47.856 DEBUG:boto:Params: {} 2022-09-13T07:43:47.857 DEBUG:boto:Token: None 2022-09-13T07:43:47.857 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:47 GMT /yxvvuh-23/ 2022-09-13T07:43:47.858 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:sjb9HgzhxDmXLQX7c6bK1sejdqA= 2022-09-13T07:43:47.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:sjb9HgzhxDmXLQX7c6bK1sejdqA=', 'Content-Length': '0'} 2022-09-13T07:43:47.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008864e3d30dc1f06-00632034b3-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:47.862 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:47.862 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:47.863 DEBUG:boto:Method: HEAD 2022-09-13T07:43:47.863 DEBUG:boto:Path: /yxvvuh-23/ 2022-09-13T07:43:47.864 DEBUG:boto:Data: 2022-09-13T07:43:47.864 DEBUG:boto:Headers: {} 2022-09-13T07:43:47.865 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:47.867 DEBUG:boto:Port: 8000 2022-09-13T07:43:47.867 DEBUG:boto:Params: {} 2022-09-13T07:43:47.868 DEBUG:boto:Token: None 2022-09-13T07:43:47.869 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:47 GMT /yxvvuh-23/ 2022-09-13T07:43:47.870 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:sjb9HgzhxDmXLQX7c6bK1sejdqA= 2022-09-13T07:43:47.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:sjb9HgzhxDmXLQX7c6bK1sejdqA=', 'Content-Length': '0'} 2022-09-13T07:43:47.874 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008762b48324e98d47-00632034b3-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:47.875 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:43:47.875 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:47.876 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:47.876 DEBUG:boto:path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:47.878 DEBUG:boto:auth_path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:47.878 DEBUG:boto:Method: GET 2022-09-13T07:43:47.879 DEBUG:boto:Path: /yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:47.880 DEBUG:boto:Data: 2022-09-13T07:43:47.881 DEBUG:boto:Headers: {} 2022-09-13T07:43:47.882 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:47.883 DEBUG:boto:Port: 8000 2022-09-13T07:43:47.884 DEBUG:boto:Params: {} 2022-09-13T07:43:47.885 DEBUG:boto:Token: None 2022-09-13T07:43:47.886 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:47 GMT /yxvvuh-23/?versions 2022-09-13T07:43:47.887 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pH/VXNcforOTIWKblSwwB/E7Woo= 2022-09-13T07:43:47.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pH/VXNcforOTIWKblSwwB/E7Woo=', 'Content-Length': '0'} 2022-09-13T07:43:47.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b37c144960423f0a-00632034b3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:47.892 DEBUG:boto:b'yxvvuh-23999falseobjg8mnQoVEf42.Xkv788ZfGhh9A7441T7true2022-09-13T07:42:28.633Zrgw-multisite-test-userMultisite Test UserNormalobjAVpI2lJqznmisZGvN7vOSpveirs9URwfalse2022-09-13T07:42:28.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:47.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:47.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:47.894 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:43:47.894 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:47.895 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:47.895 DEBUG:boto:path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:47.896 DEBUG:boto:auth_path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:47.896 DEBUG:boto:Method: GET 2022-09-13T07:43:47.897 DEBUG:boto:Path: /yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:47.897 DEBUG:boto:Data: 2022-09-13T07:43:47.898 DEBUG:boto:Headers: {} 2022-09-13T07:43:47.898 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:47.899 DEBUG:boto:Port: 8000 2022-09-13T07:43:47.899 DEBUG:boto:Params: {} 2022-09-13T07:43:47.900 DEBUG:boto:Token: None 2022-09-13T07:43:47.901 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:47 GMT /yxvvuh-23/?versions 2022-09-13T07:43:47.901 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pH/VXNcforOTIWKblSwwB/E7Woo= 2022-09-13T07:43:47.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pH/VXNcforOTIWKblSwwB/E7Woo=', 'Content-Length': '0'} 2022-09-13T07:43:47.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036a456a463d7f465-00632034b3-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:47.946 DEBUG:boto:b'yxvvuh-23999falseobjg8mnQoVEf42.Xkv788ZfGhh9A7441T7true2022-09-13T07:42:28.633Zrgw-multisite-test-userMultisite Test UserNormalobjAVpI2lJqznmisZGvN7vOSpveirs9URwfalse2022-09-13T07:42:28.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:47.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:47.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:47.948 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:47.949 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:47.949 DEBUG:boto:path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:47.950 DEBUG:boto:auth_path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:47.950 DEBUG:boto:Method: GET 2022-09-13T07:43:47.951 DEBUG:boto:Path: /yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:47.952 DEBUG:boto:Data: 2022-09-13T07:43:47.952 DEBUG:boto:Headers: {} 2022-09-13T07:43:47.952 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:47.953 DEBUG:boto:Port: 8000 2022-09-13T07:43:47.954 DEBUG:boto:Params: {} 2022-09-13T07:43:47.954 DEBUG:boto:Token: None 2022-09-13T07:43:47.955 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:47 GMT /yxvvuh-23/?versions 2022-09-13T07:43:47.955 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pH/VXNcforOTIWKblSwwB/E7Woo= 2022-09-13T07:43:47.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pH/VXNcforOTIWKblSwwB/E7Woo=', 'Content-Length': '0'} 2022-09-13T07:43:47.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055c8ef649e96ff23-00632034b3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:47.959 DEBUG:boto:b'yxvvuh-23999falseobjg8mnQoVEf42.Xkv788ZfGhh9A7441T7true2022-09-13T07:42:28.633Zrgw-multisite-test-userMultisite Test UserNormalobjAVpI2lJqznmisZGvN7vOSpveirs9URwfalse2022-09-13T07:42:28.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:47.960 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:47.961 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:47.962 DEBUG:boto:path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:47.962 DEBUG:boto:auth_path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:47.963 DEBUG:boto:Method: GET 2022-09-13T07:43:47.963 DEBUG:boto:Path: /yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:47.964 DEBUG:boto:Data: 2022-09-13T07:43:47.965 DEBUG:boto:Headers: {} 2022-09-13T07:43:47.966 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:47.966 DEBUG:boto:Port: 8000 2022-09-13T07:43:47.967 DEBUG:boto:Params: {} 2022-09-13T07:43:47.968 DEBUG:boto:Token: None 2022-09-13T07:43:47.968 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:47 GMT /yxvvuh-23/?versions 2022-09-13T07:43:47.969 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pH/VXNcforOTIWKblSwwB/E7Woo= 2022-09-13T07:43:47.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pH/VXNcforOTIWKblSwwB/E7Woo=', 'Content-Length': '0'} 2022-09-13T07:43:47.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdf750b9f7d05a67-00632034b3-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:47.979 DEBUG:boto:b'yxvvuh-23999falseobjg8mnQoVEf42.Xkv788ZfGhh9A7441T7true2022-09-13T07:42:28.633Zrgw-multisite-test-userMultisite Test UserNormalobjAVpI2lJqznmisZGvN7vOSpveirs9URwfalse2022-09-13T07:42:28.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:47.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:47.981 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:47.982 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:47.982 DEBUG:boto:Method: HEAD 2022-09-13T07:43:47.983 DEBUG:boto:Path: /yxvvuh-23/obj 2022-09-13T07:43:47.984 DEBUG:boto:Data: 2022-09-13T07:43:47.984 DEBUG:boto:Headers: {} 2022-09-13T07:43:47.985 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:47.985 DEBUG:boto:Port: 8000 2022-09-13T07:43:47.986 DEBUG:boto:Params: {} 2022-09-13T07:43:47.986 DEBUG:boto:Token: None 2022-09-13T07:43:47.987 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:47 GMT /yxvvuh-23/obj 2022-09-13T07:43:47.988 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/OEaDn5bStucxdubNArj0U6z3YA= 2022-09-13T07:43:47.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/OEaDn5bStucxdubNArj0U6z3YA=', 'Content-Length': '0'} 2022-09-13T07:43:47.991 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000a4cde51fffc6517c-00632034b3-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:47.992 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:47.992 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:47.993 DEBUG:boto:Method: HEAD 2022-09-13T07:43:47.993 DEBUG:boto:Path: /yxvvuh-23/obj 2022-09-13T07:43:47.994 DEBUG:boto:Data: 2022-09-13T07:43:47.994 DEBUG:boto:Headers: {} 2022-09-13T07:43:47.995 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:47.996 DEBUG:boto:Port: 8000 2022-09-13T07:43:47.996 DEBUG:boto:Params: {} 2022-09-13T07:43:47.997 DEBUG:boto:Token: None 2022-09-13T07:43:47.998 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:47 GMT /yxvvuh-23/obj 2022-09-13T07:43:47.999 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/OEaDn5bStucxdubNArj0U6z3YA= 2022-09-13T07:43:47.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/OEaDn5bStucxdubNArj0U6z3YA=', 'Content-Length': '0'} 2022-09-13T07:43:48.002 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000bff31446dfa7039a-00632034b3-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:48.004 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.005 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.006 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.007 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.008 DEBUG:boto:Method: GET 2022-09-13T07:43:48.008 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.009 DEBUG:boto:Data: 2022-09-13T07:43:48.010 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.011 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:48.012 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.013 DEBUG:boto:Params: {} 2022-09-13T07:43:48.014 DEBUG:boto:Token: None 2022-09-13T07:43:48.015 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.016 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg= 2022-09-13T07:43:48.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg=', 'Content-Length': '0'} 2022-09-13T07:43:48.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c8ff5826c3ee8d49-00632034b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.021 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.022 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.023 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.024 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.024 DEBUG:boto:Method: GET 2022-09-13T07:43:48.025 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.025 DEBUG:boto:Data: 2022-09-13T07:43:48.026 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.026 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.027 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.027 DEBUG:boto:Params: {} 2022-09-13T07:43:48.028 DEBUG:boto:Token: None 2022-09-13T07:43:48.028 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.029 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg= 2022-09-13T07:43:48.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg=', 'Content-Length': '0'} 2022-09-13T07:43:48.032 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d7d55ad6914559d-00632034b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.033 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.033 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.034 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.034 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.035 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.035 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.036 DEBUG:boto:Data: 2022-09-13T07:43:48.036 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.037 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:48.037 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.038 DEBUG:boto:Params: {} 2022-09-13T07:43:48.039 DEBUG:boto:Token: None 2022-09-13T07:43:48.039 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.040 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QwUtNC8xeAiz8NU+b1v7V1I1240= 2022-09-13T07:43:48.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QwUtNC8xeAiz8NU+b1v7V1I1240=', 'Content-Length': '0'} 2022-09-13T07:43:48.044 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c4ec3db34fdfc938-00632034b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.044 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.045 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.046 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.047 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.047 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.048 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.048 DEBUG:boto:Data: 2022-09-13T07:43:48.049 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.049 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.050 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.050 DEBUG:boto:Params: {} 2022-09-13T07:43:48.051 DEBUG:boto:Token: None 2022-09-13T07:43:48.051 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.052 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QwUtNC8xeAiz8NU+b1v7V1I1240= 2022-09-13T07:43:48.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QwUtNC8xeAiz8NU+b1v7V1I1240=', 'Content-Length': '0'} 2022-09-13T07:43:48.055 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f1e898c9f3851ec-00632034b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:48.056 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.057 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.057 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.059 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.059 DEBUG:boto:Method: GET 2022-09-13T07:43:48.060 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.061 DEBUG:boto:Data: 2022-09-13T07:43:48.062 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.063 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:48.064 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.065 DEBUG:boto:Params: {} 2022-09-13T07:43:48.066 DEBUG:boto:Token: None 2022-09-13T07:43:48.067 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.068 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg= 2022-09-13T07:43:48.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg=', 'Content-Length': '0'} 2022-09-13T07:43:48.071 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000025addf455d8382e3-00632034b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.072 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.073 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.073 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.074 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.074 DEBUG:boto:Method: GET 2022-09-13T07:43:48.075 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.076 DEBUG:boto:Data: 2022-09-13T07:43:48.076 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.077 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.077 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.078 DEBUG:boto:Params: {} 2022-09-13T07:43:48.079 DEBUG:boto:Token: None 2022-09-13T07:43:48.079 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.080 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg= 2022-09-13T07:43:48.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg=', 'Content-Length': '0'} 2022-09-13T07:43:48.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005ab78ebc271b1ba3-00632034b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.083 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.084 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.084 DEBUG:boto:path=/yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.085 DEBUG:boto:auth_path=/yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.086 DEBUG:boto:Method: GET 2022-09-13T07:43:48.086 DEBUG:boto:Path: /yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.087 DEBUG:boto:Data: 2022-09-13T07:43:48.087 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.088 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:48.089 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.089 DEBUG:boto:Params: {} 2022-09-13T07:43:48.090 DEBUG:boto:Token: None 2022-09-13T07:43:48.091 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/ 2022-09-13T07:43:48.092 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jP/ph3xBz99fEeUFgHt578ZZaLg= 2022-09-13T07:43:48.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jP/ph3xBz99fEeUFgHt578ZZaLg=', 'Content-Length': '0'} 2022-09-13T07:43:48.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000101f4151a775e83e-00632034b4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.138 DEBUG:boto:b'yxvvuh-23obj1000false' 2022-09-13T07:43:48.139 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.140 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.140 DEBUG:boto:path=/yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.141 DEBUG:boto:auth_path=/yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.142 DEBUG:boto:Method: GET 2022-09-13T07:43:48.142 DEBUG:boto:Path: /yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.143 DEBUG:boto:Data: 2022-09-13T07:43:48.143 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.144 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.145 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.145 DEBUG:boto:Params: {} 2022-09-13T07:43:48.146 DEBUG:boto:Token: None 2022-09-13T07:43:48.147 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/ 2022-09-13T07:43:48.147 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jP/ph3xBz99fEeUFgHt578ZZaLg= 2022-09-13T07:43:48.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jP/ph3xBz99fEeUFgHt578ZZaLg=', 'Content-Length': '0'} 2022-09-13T07:43:48.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d369167be2e55d2f-00632034b4-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.152 DEBUG:boto:b'yxvvuh-23obj1000false' 2022-09-13T07:43:48.153 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.153 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.154 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.154 DEBUG:boto:Path: /yxvvuh-23/obj 2022-09-13T07:43:48.155 DEBUG:boto:Data: 2022-09-13T07:43:48.155 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.156 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:48.156 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.157 DEBUG:boto:Params: {} 2022-09-13T07:43:48.157 DEBUG:boto:Token: None 2022-09-13T07:43:48.158 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj 2022-09-13T07:43:48.158 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs= 2022-09-13T07:43:48.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs=', 'Content-Length': '0'} 2022-09-13T07:43:48.161 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000745d5cb5e96fae36-00632034b4-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.162 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.163 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.163 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.164 DEBUG:boto:Path: /yxvvuh-23/obj 2022-09-13T07:43:48.164 DEBUG:boto:Data: 2022-09-13T07:43:48.165 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.165 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.166 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.166 DEBUG:boto:Params: {} 2022-09-13T07:43:48.167 DEBUG:boto:Token: None 2022-09-13T07:43:48.167 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj 2022-09-13T07:43:48.168 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs= 2022-09-13T07:43:48.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs=', 'Content-Length': '0'} 2022-09-13T07:43:48.171 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000d765e88fe97a570e-00632034b4-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.172 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-23 zones={test-zone2, test-zone1} 2022-09-13T07:43:48.172 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-23 zones={test-zone3, test-zone1} 2022-09-13T07:43:48.173 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.173 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.174 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.174 DEBUG:boto:Path: /yxvvuh-23/ 2022-09-13T07:43:48.175 DEBUG:boto:Data: 2022-09-13T07:43:48.176 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.176 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.177 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.177 DEBUG:boto:Params: {} 2022-09-13T07:43:48.178 DEBUG:boto:Token: None 2022-09-13T07:43:48.179 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/ 2022-09-13T07:43:48.180 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:akEQNXR4o/rzpwlwDQLVE6cY64E= 2022-09-13T07:43:48.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:akEQNXR4o/rzpwlwDQLVE6cY64E=', 'Content-Length': '0'} 2022-09-13T07:43:48.183 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ca9f4e3a172897e-00632034b4-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.184 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.185 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.186 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.186 DEBUG:boto:Path: /yxvvuh-23/ 2022-09-13T07:43:48.187 DEBUG:boto:Data: 2022-09-13T07:43:48.188 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.189 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.189 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.190 DEBUG:boto:Params: {} 2022-09-13T07:43:48.191 DEBUG:boto:Token: None 2022-09-13T07:43:48.192 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/ 2022-09-13T07:43:48.193 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:akEQNXR4o/rzpwlwDQLVE6cY64E= 2022-09-13T07:43:48.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:akEQNXR4o/rzpwlwDQLVE6cY64E=', 'Content-Length': '0'} 2022-09-13T07:43:48.197 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb723c38c3d9b300-00632034b4-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.198 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:43:48.198 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.199 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.200 DEBUG:boto:path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.200 DEBUG:boto:auth_path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.201 DEBUG:boto:Method: GET 2022-09-13T07:43:48.201 DEBUG:boto:Path: /yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.202 DEBUG:boto:Data: 2022-09-13T07:43:48.203 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.203 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.204 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.204 DEBUG:boto:Params: {} 2022-09-13T07:43:48.205 DEBUG:boto:Token: None 2022-09-13T07:43:48.206 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/?versions 2022-09-13T07:43:48.206 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY= 2022-09-13T07:43:48.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY=', 'Content-Length': '0'} 2022-09-13T07:43:48.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f365e5b380235be-00632034b4-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.254 DEBUG:boto:b'yxvvuh-23999falseobjg8mnQoVEf42.Xkv788ZfGhh9A7441T7true2022-09-13T07:42:28.633Zrgw-multisite-test-userMultisite Test UserNormalobjAVpI2lJqznmisZGvN7vOSpveirs9URwfalse2022-09-13T07:42:28.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:48.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:48.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:48.256 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:43:48.257 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.257 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.258 DEBUG:boto:path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.258 DEBUG:boto:auth_path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.259 DEBUG:boto:Method: GET 2022-09-13T07:43:48.260 DEBUG:boto:Path: /yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.260 DEBUG:boto:Data: 2022-09-13T07:43:48.261 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.262 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.263 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.264 DEBUG:boto:Params: {} 2022-09-13T07:43:48.265 DEBUG:boto:Token: None 2022-09-13T07:43:48.266 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/?versions 2022-09-13T07:43:48.267 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY= 2022-09-13T07:43:48.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY=', 'Content-Length': '0'} 2022-09-13T07:43:48.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec13df99692df329-00632034b4-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.272 DEBUG:boto:b'yxvvuh-23999falseobjg8mnQoVEf42.Xkv788ZfGhh9A7441T7true2022-09-13T07:42:28.633Zrgw-multisite-test-userMultisite Test UserNormalobjAVpI2lJqznmisZGvN7vOSpveirs9URwfalse2022-09-13T07:42:28.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:48.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:48.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:48.274 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.275 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.275 DEBUG:boto:path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.276 DEBUG:boto:auth_path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.276 DEBUG:boto:Method: GET 2022-09-13T07:43:48.277 DEBUG:boto:Path: /yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.277 DEBUG:boto:Data: 2022-09-13T07:43:48.278 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.278 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.279 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.279 DEBUG:boto:Params: {} 2022-09-13T07:43:48.280 DEBUG:boto:Token: None 2022-09-13T07:43:48.281 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/?versions 2022-09-13T07:43:48.281 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY= 2022-09-13T07:43:48.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY=', 'Content-Length': '0'} 2022-09-13T07:43:48.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b234e65a954a209-00632034b4-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.285 DEBUG:boto:b'yxvvuh-23999falseobjg8mnQoVEf42.Xkv788ZfGhh9A7441T7true2022-09-13T07:42:28.633Zrgw-multisite-test-userMultisite Test UserNormalobjAVpI2lJqznmisZGvN7vOSpveirs9URwfalse2022-09-13T07:42:28.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:48.286 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.286 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.287 DEBUG:boto:path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.288 DEBUG:boto:auth_path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.289 DEBUG:boto:Method: GET 2022-09-13T07:43:48.289 DEBUG:boto:Path: /yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.290 DEBUG:boto:Data: 2022-09-13T07:43:48.291 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.292 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.292 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.293 DEBUG:boto:Params: {} 2022-09-13T07:43:48.294 DEBUG:boto:Token: None 2022-09-13T07:43:48.295 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/?versions 2022-09-13T07:43:48.296 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY= 2022-09-13T07:43:48.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY=', 'Content-Length': '0'} 2022-09-13T07:43:48.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c17b724f9931106e-00632034b4-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.346 DEBUG:boto:b'yxvvuh-23999falseobjg8mnQoVEf42.Xkv788ZfGhh9A7441T7true2022-09-13T07:42:28.633Zrgw-multisite-test-userMultisite Test UserNormalobjAVpI2lJqznmisZGvN7vOSpveirs9URwfalse2022-09-13T07:42:28.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:48.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:48.349 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.350 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.350 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.351 DEBUG:boto:Path: /yxvvuh-23/obj 2022-09-13T07:43:48.352 DEBUG:boto:Data: 2022-09-13T07:43:48.353 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.355 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.356 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.356 DEBUG:boto:Params: {} 2022-09-13T07:43:48.357 DEBUG:boto:Token: None 2022-09-13T07:43:48.358 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj 2022-09-13T07:43:48.360 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs= 2022-09-13T07:43:48.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs=', 'Content-Length': '0'} 2022-09-13T07:43:48.364 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000022a61292ff833e54-00632034b4-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.365 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.365 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.366 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.367 DEBUG:boto:Path: /yxvvuh-23/obj 2022-09-13T07:43:48.368 DEBUG:boto:Data: 2022-09-13T07:43:48.369 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.370 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.371 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.371 DEBUG:boto:Params: {} 2022-09-13T07:43:48.372 DEBUG:boto:Token: None 2022-09-13T07:43:48.372 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj 2022-09-13T07:43:48.373 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs= 2022-09-13T07:43:48.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs=', 'Content-Length': '0'} 2022-09-13T07:43:48.377 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000008556e1de80b47a21-00632034b4-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:48.379 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.379 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.380 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.381 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.382 DEBUG:boto:Method: GET 2022-09-13T07:43:48.383 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.383 DEBUG:boto:Data: 2022-09-13T07:43:48.384 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.385 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.386 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.387 DEBUG:boto:Params: {} 2022-09-13T07:43:48.388 DEBUG:boto:Token: None 2022-09-13T07:43:48.389 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.389 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg= 2022-09-13T07:43:48.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg=', 'Content-Length': '0'} 2022-09-13T07:43:48.392 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a859d6fa22fccc23-00632034b4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.393 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.393 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.394 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.394 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.395 DEBUG:boto:Method: GET 2022-09-13T07:43:48.395 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.396 DEBUG:boto:Data: 2022-09-13T07:43:48.396 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.397 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.397 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.398 DEBUG:boto:Params: {} 2022-09-13T07:43:48.399 DEBUG:boto:Token: None 2022-09-13T07:43:48.399 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.400 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg= 2022-09-13T07:43:48.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg=', 'Content-Length': '0'} 2022-09-13T07:43:48.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002ad9a696c329da40-00632034b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.403 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.404 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.404 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.405 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.405 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.406 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.406 DEBUG:boto:Data: 2022-09-13T07:43:48.407 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.407 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.408 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.408 DEBUG:boto:Params: {} 2022-09-13T07:43:48.409 DEBUG:boto:Token: None 2022-09-13T07:43:48.409 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.410 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QwUtNC8xeAiz8NU+b1v7V1I1240= 2022-09-13T07:43:48.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QwUtNC8xeAiz8NU+b1v7V1I1240=', 'Content-Length': '0'} 2022-09-13T07:43:48.412 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049a12239bd18a33d-00632034b4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.413 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.413 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.414 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.414 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.415 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.415 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.416 DEBUG:boto:Data: 2022-09-13T07:43:48.416 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.417 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.417 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.417 DEBUG:boto:Params: {} 2022-09-13T07:43:48.418 DEBUG:boto:Token: None 2022-09-13T07:43:48.419 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.419 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QwUtNC8xeAiz8NU+b1v7V1I1240= 2022-09-13T07:43:48.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QwUtNC8xeAiz8NU+b1v7V1I1240=', 'Content-Length': '0'} 2022-09-13T07:43:48.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000860af02ed8138df9-00632034b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:48.423 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.424 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.424 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.425 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.425 DEBUG:boto:Method: GET 2022-09-13T07:43:48.426 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.426 DEBUG:boto:Data: 2022-09-13T07:43:48.427 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.427 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.427 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.428 DEBUG:boto:Params: {} 2022-09-13T07:43:48.429 DEBUG:boto:Token: None 2022-09-13T07:43:48.429 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.430 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg= 2022-09-13T07:43:48.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg=', 'Content-Length': '0'} 2022-09-13T07:43:48.432 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000873055b7cfd803c0-00632034b4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.433 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.434 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.434 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.435 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.435 DEBUG:boto:Method: GET 2022-09-13T07:43:48.436 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.436 DEBUG:boto:Data: 2022-09-13T07:43:48.437 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.437 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.438 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.438 DEBUG:boto:Params: {} 2022-09-13T07:43:48.439 DEBUG:boto:Token: None 2022-09-13T07:43:48.440 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.440 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg= 2022-09-13T07:43:48.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg=', 'Content-Length': '0'} 2022-09-13T07:43:48.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e6608303faf4a999-00632034b4-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.444 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.445 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.445 DEBUG:boto:path=/yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.446 DEBUG:boto:auth_path=/yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.446 DEBUG:boto:Method: GET 2022-09-13T07:43:48.447 DEBUG:boto:Path: /yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.447 DEBUG:boto:Data: 2022-09-13T07:43:48.448 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.448 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.449 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.449 DEBUG:boto:Params: {} 2022-09-13T07:43:48.450 DEBUG:boto:Token: None 2022-09-13T07:43:48.451 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/ 2022-09-13T07:43:48.452 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jP/ph3xBz99fEeUFgHt578ZZaLg= 2022-09-13T07:43:48.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jP/ph3xBz99fEeUFgHt578ZZaLg=', 'Content-Length': '0'} 2022-09-13T07:43:48.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7c59321196923d1-00632034b4-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.458 DEBUG:boto:b'yxvvuh-23obj1000false' 2022-09-13T07:43:48.459 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.460 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.460 DEBUG:boto:path=/yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.461 DEBUG:boto:auth_path=/yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.461 DEBUG:boto:Method: GET 2022-09-13T07:43:48.462 DEBUG:boto:Path: /yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.462 DEBUG:boto:Data: 2022-09-13T07:43:48.463 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.463 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.464 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.464 DEBUG:boto:Params: {} 2022-09-13T07:43:48.465 DEBUG:boto:Token: None 2022-09-13T07:43:48.465 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/ 2022-09-13T07:43:48.466 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jP/ph3xBz99fEeUFgHt578ZZaLg= 2022-09-13T07:43:48.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jP/ph3xBz99fEeUFgHt578ZZaLg=', 'Content-Length': '0'} 2022-09-13T07:43:48.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cb7ed9c0bc33ada-00632034b4-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.470 DEBUG:boto:b'yxvvuh-23obj1000false' 2022-09-13T07:43:48.471 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.471 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.472 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.472 DEBUG:boto:Path: /yxvvuh-23/obj 2022-09-13T07:43:48.473 DEBUG:boto:Data: 2022-09-13T07:43:48.473 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.474 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.474 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.475 DEBUG:boto:Params: {} 2022-09-13T07:43:48.475 DEBUG:boto:Token: None 2022-09-13T07:43:48.476 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj 2022-09-13T07:43:48.476 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs= 2022-09-13T07:43:48.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs=', 'Content-Length': '0'} 2022-09-13T07:43:48.486 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000085161770c665b94-00632034b4-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.487 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.488 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.488 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.489 DEBUG:boto:Path: /yxvvuh-23/obj 2022-09-13T07:43:48.489 DEBUG:boto:Data: 2022-09-13T07:43:48.490 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.490 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.491 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.491 DEBUG:boto:Params: {} 2022-09-13T07:43:48.492 DEBUG:boto:Token: None 2022-09-13T07:43:48.493 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj 2022-09-13T07:43:48.493 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs= 2022-09-13T07:43:48.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs=', 'Content-Length': '0'} 2022-09-13T07:43:48.496 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000006e19c12501497cdf-00632034b4-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.498 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-23 zones={test-zone3, test-zone1} 2022-09-13T07:43:48.498 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-23 zones={test-zone3, test-zone2} 2022-09-13T07:43:48.499 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.500 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.501 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.502 DEBUG:boto:Path: /yxvvuh-23/ 2022-09-13T07:43:48.503 DEBUG:boto:Data: 2022-09-13T07:43:48.504 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.505 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.506 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.506 DEBUG:boto:Params: {} 2022-09-13T07:43:48.507 DEBUG:boto:Token: None 2022-09-13T07:43:48.508 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/ 2022-09-13T07:43:48.509 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:akEQNXR4o/rzpwlwDQLVE6cY64E= 2022-09-13T07:43:48.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:akEQNXR4o/rzpwlwDQLVE6cY64E=', 'Content-Length': '0'} 2022-09-13T07:43:48.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da4c7c363daf3ba9-00632034b4-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.514 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.515 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.516 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.517 DEBUG:boto:Path: /yxvvuh-23/ 2022-09-13T07:43:48.518 DEBUG:boto:Data: 2022-09-13T07:43:48.519 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.520 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:48.521 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.522 DEBUG:boto:Params: {} 2022-09-13T07:43:48.523 DEBUG:boto:Token: None 2022-09-13T07:43:48.524 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/ 2022-09-13T07:43:48.524 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:akEQNXR4o/rzpwlwDQLVE6cY64E= 2022-09-13T07:43:48.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:akEQNXR4o/rzpwlwDQLVE6cY64E=', 'Content-Length': '0'} 2022-09-13T07:43:48.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ae60079d8bbe4b0-00632034b4-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.528 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:43:48.529 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.531 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.532 DEBUG:boto:path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.533 DEBUG:boto:auth_path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.534 DEBUG:boto:Method: GET 2022-09-13T07:43:48.535 DEBUG:boto:Path: /yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.536 DEBUG:boto:Data: 2022-09-13T07:43:48.536 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.537 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.538 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.539 DEBUG:boto:Params: {} 2022-09-13T07:43:48.540 DEBUG:boto:Token: None 2022-09-13T07:43:48.541 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/?versions 2022-09-13T07:43:48.542 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY= 2022-09-13T07:43:48.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY=', 'Content-Length': '0'} 2022-09-13T07:43:48.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077211ef8e26e216a-00632034b4-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.547 DEBUG:boto:b'yxvvuh-23999falseobjg8mnQoVEf42.Xkv788ZfGhh9A7441T7true2022-09-13T07:42:28.633Zrgw-multisite-test-userMultisite Test UserNormalobjAVpI2lJqznmisZGvN7vOSpveirs9URwfalse2022-09-13T07:42:28.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:48.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:48.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:48.549 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:43:48.550 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.550 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.551 DEBUG:boto:path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.551 DEBUG:boto:auth_path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.552 DEBUG:boto:Method: GET 2022-09-13T07:43:48.552 DEBUG:boto:Path: /yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.553 DEBUG:boto:Data: 2022-09-13T07:43:48.553 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.554 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:48.555 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.556 DEBUG:boto:Params: {} 2022-09-13T07:43:48.556 DEBUG:boto:Token: None 2022-09-13T07:43:48.557 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/?versions 2022-09-13T07:43:48.557 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY= 2022-09-13T07:43:48.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY=', 'Content-Length': '0'} 2022-09-13T07:43:48.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019b71e5fa37a43c7-00632034b4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.566 DEBUG:boto:b'yxvvuh-23999falseobjg8mnQoVEf42.Xkv788ZfGhh9A7441T7true2022-09-13T07:42:28.633Zrgw-multisite-test-userMultisite Test UserNormalobjAVpI2lJqznmisZGvN7vOSpveirs9URwfalse2022-09-13T07:42:28.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:48.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:48.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:43:48.568 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.568 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.569 DEBUG:boto:path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.569 DEBUG:boto:auth_path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.570 DEBUG:boto:Method: GET 2022-09-13T07:43:48.570 DEBUG:boto:Path: /yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.571 DEBUG:boto:Data: 2022-09-13T07:43:48.571 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.572 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.572 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.573 DEBUG:boto:Params: {} 2022-09-13T07:43:48.574 DEBUG:boto:Token: None 2022-09-13T07:43:48.574 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/?versions 2022-09-13T07:43:48.575 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY= 2022-09-13T07:43:48.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY=', 'Content-Length': '0'} 2022-09-13T07:43:48.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6603dfad78cd859-00632034b4-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.626 DEBUG:boto:b'yxvvuh-23999falseobjg8mnQoVEf42.Xkv788ZfGhh9A7441T7true2022-09-13T07:42:28.633Zrgw-multisite-test-userMultisite Test UserNormalobjAVpI2lJqznmisZGvN7vOSpveirs9URwfalse2022-09-13T07:42:28.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:48.627 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.628 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.628 DEBUG:boto:path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.629 DEBUG:boto:auth_path=/yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.629 DEBUG:boto:Method: GET 2022-09-13T07:43:48.630 DEBUG:boto:Path: /yxvvuh-23/?versions&max-keys=999 2022-09-13T07:43:48.630 DEBUG:boto:Data: 2022-09-13T07:43:48.631 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.631 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:48.632 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.632 DEBUG:boto:Params: {} 2022-09-13T07:43:48.634 DEBUG:boto:Token: None 2022-09-13T07:43:48.635 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/?versions 2022-09-13T07:43:48.636 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY= 2022-09-13T07:43:48.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t7K9p3PklkXz3HnB/40eMZgG0UY=', 'Content-Length': '0'} 2022-09-13T07:43:48.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006013a945d54f296-00632034b4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.640 DEBUG:boto:b'yxvvuh-23999falseobjg8mnQoVEf42.Xkv788ZfGhh9A7441T7true2022-09-13T07:42:28.633Zrgw-multisite-test-userMultisite Test UserNormalobjAVpI2lJqznmisZGvN7vOSpveirs9URwfalse2022-09-13T07:42:28.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:43:48.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:48.641 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.642 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.642 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.643 DEBUG:boto:Path: /yxvvuh-23/obj 2022-09-13T07:43:48.643 DEBUG:boto:Data: 2022-09-13T07:43:48.644 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.645 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.645 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.646 DEBUG:boto:Params: {} 2022-09-13T07:43:48.647 DEBUG:boto:Token: None 2022-09-13T07:43:48.648 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj 2022-09-13T07:43:48.649 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs= 2022-09-13T07:43:48.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs=', 'Content-Length': '0'} 2022-09-13T07:43:48.652 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000064ce1e5e1d2090d0-00632034b4-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.653 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.653 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.654 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.654 DEBUG:boto:Path: /yxvvuh-23/obj 2022-09-13T07:43:48.655 DEBUG:boto:Data: 2022-09-13T07:43:48.655 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.656 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:48.657 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.657 DEBUG:boto:Params: {} 2022-09-13T07:43:48.658 DEBUG:boto:Token: None 2022-09-13T07:43:48.659 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj 2022-09-13T07:43:48.659 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs= 2022-09-13T07:43:48.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs=', 'Content-Length': '0'} 2022-09-13T07:43:48.662 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000005f114fce56d9254d-00632034b4-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:48.664 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.664 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.665 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.666 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.666 DEBUG:boto:Method: GET 2022-09-13T07:43:48.667 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.667 DEBUG:boto:Data: 2022-09-13T07:43:48.668 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.668 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.669 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.670 DEBUG:boto:Params: {} 2022-09-13T07:43:48.670 DEBUG:boto:Token: None 2022-09-13T07:43:48.671 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.673 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg= 2022-09-13T07:43:48.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg=', 'Content-Length': '0'} 2022-09-13T07:43:48.677 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000091ff3f8dd1506a58-00632034b4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.678 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.679 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.679 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.680 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.681 DEBUG:boto:Method: GET 2022-09-13T07:43:48.682 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.682 DEBUG:boto:Data: 2022-09-13T07:43:48.683 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.683 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:48.684 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.684 DEBUG:boto:Params: {} 2022-09-13T07:43:48.685 DEBUG:boto:Token: None 2022-09-13T07:43:48.686 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.687 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg= 2022-09-13T07:43:48.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg=', 'Content-Length': '0'} 2022-09-13T07:43:48.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000549b0220c6e6ddeb-00632034b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.691 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.691 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.692 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.692 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.693 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.693 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.694 DEBUG:boto:Data: 2022-09-13T07:43:48.694 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.695 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.695 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.696 DEBUG:boto:Params: {} 2022-09-13T07:43:48.696 DEBUG:boto:Token: None 2022-09-13T07:43:48.697 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.697 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QwUtNC8xeAiz8NU+b1v7V1I1240= 2022-09-13T07:43:48.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QwUtNC8xeAiz8NU+b1v7V1I1240=', 'Content-Length': '0'} 2022-09-13T07:43:48.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b81e17ed1ef3fb8d-00632034b4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.701 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.702 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.702 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.703 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.703 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.703 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.704 DEBUG:boto:Data: 2022-09-13T07:43:48.704 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.705 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:48.705 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.706 DEBUG:boto:Params: {} 2022-09-13T07:43:48.706 DEBUG:boto:Token: None 2022-09-13T07:43:48.707 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.708 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QwUtNC8xeAiz8NU+b1v7V1I1240= 2022-09-13T07:43:48.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QwUtNC8xeAiz8NU+b1v7V1I1240=', 'Content-Length': '0'} 2022-09-13T07:43:48.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d2d55b3e1f6082ab-00632034b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:43:48.711 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.712 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.712 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.713 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.713 DEBUG:boto:Method: GET 2022-09-13T07:43:48.714 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.714 DEBUG:boto:Data: 2022-09-13T07:43:48.715 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.715 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.716 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.716 DEBUG:boto:Params: {} 2022-09-13T07:43:48.717 DEBUG:boto:Token: None 2022-09-13T07:43:48.718 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.718 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg= 2022-09-13T07:43:48.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg=', 'Content-Length': '0'} 2022-09-13T07:43:48.721 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000623472f1b9fc7b48-00632034b4-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.722 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.722 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.723 DEBUG:boto:path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.724 DEBUG:boto:auth_path=/yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.724 DEBUG:boto:Method: GET 2022-09-13T07:43:48.725 DEBUG:boto:Path: /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.726 DEBUG:boto:Data: 2022-09-13T07:43:48.726 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.727 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:48.728 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.729 DEBUG:boto:Params: {} 2022-09-13T07:43:48.729 DEBUG:boto:Token: None 2022-09-13T07:43:48.730 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj?versionId=AVpI2lJqznmisZGvN7vOSpveirs9URw 2022-09-13T07:43:48.731 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg= 2022-09-13T07:43:48.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NA7zsBp/3AuknWK9dC76RYYdKcg=', 'Content-Length': '0'} 2022-09-13T07:43:48.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:42:28 GMT'), ('x-amz-version-id', 'AVpI2lJqznmisZGvN7vOSpveirs9URw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008aafb317ff7166bc-00632034b4-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.735 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.736 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.737 DEBUG:boto:path=/yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.738 DEBUG:boto:auth_path=/yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.739 DEBUG:boto:Method: GET 2022-09-13T07:43:48.740 DEBUG:boto:Path: /yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.741 DEBUG:boto:Data: 2022-09-13T07:43:48.741 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.742 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.743 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.744 DEBUG:boto:Params: {} 2022-09-13T07:43:48.745 DEBUG:boto:Token: None 2022-09-13T07:43:48.746 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/ 2022-09-13T07:43:48.747 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jP/ph3xBz99fEeUFgHt578ZZaLg= 2022-09-13T07:43:48.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jP/ph3xBz99fEeUFgHt578ZZaLg=', 'Content-Length': '0'} 2022-09-13T07:43:48.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c00ab22efc5c615-00632034b4-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.753 DEBUG:boto:b'yxvvuh-23obj1000false' 2022-09-13T07:43:48.754 DEBUG:boto:path=/yxvvuh-23/ 2022-09-13T07:43:48.754 DEBUG:boto:auth_path=/yxvvuh-23/ 2022-09-13T07:43:48.755 DEBUG:boto:path=/yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.755 DEBUG:boto:auth_path=/yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.756 DEBUG:boto:Method: GET 2022-09-13T07:43:48.757 DEBUG:boto:Path: /yxvvuh-23/?prefix=obj 2022-09-13T07:43:48.757 DEBUG:boto:Data: 2022-09-13T07:43:48.758 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.758 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:48.759 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.759 DEBUG:boto:Params: {} 2022-09-13T07:43:48.760 DEBUG:boto:Token: None 2022-09-13T07:43:48.760 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/ 2022-09-13T07:43:48.761 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jP/ph3xBz99fEeUFgHt578ZZaLg= 2022-09-13T07:43:48.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jP/ph3xBz99fEeUFgHt578ZZaLg=', 'Content-Length': '0'} 2022-09-13T07:43:48.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aeca46f408ca1f0-00632034b4-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.810 DEBUG:boto:b'yxvvuh-23obj1000false' 2022-09-13T07:43:48.812 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.812 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.813 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.813 DEBUG:boto:Path: /yxvvuh-23/obj 2022-09-13T07:43:48.814 DEBUG:boto:Data: 2022-09-13T07:43:48.814 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.814 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.815 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.815 DEBUG:boto:Params: {} 2022-09-13T07:43:48.816 DEBUG:boto:Token: None 2022-09-13T07:43:48.817 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj 2022-09-13T07:43:48.817 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs= 2022-09-13T07:43:48.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs=', 'Content-Length': '0'} 2022-09-13T07:43:48.820 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000001ddca28e08f648f9-00632034b4-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.821 DEBUG:boto:path=/yxvvuh-23/obj 2022-09-13T07:43:48.821 DEBUG:boto:auth_path=/yxvvuh-23/obj 2022-09-13T07:43:48.822 DEBUG:boto:Method: HEAD 2022-09-13T07:43:48.822 DEBUG:boto:Path: /yxvvuh-23/obj 2022-09-13T07:43:48.823 DEBUG:boto:Data: 2022-09-13T07:43:48.823 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.824 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:48.824 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.825 DEBUG:boto:Params: {} 2022-09-13T07:43:48.825 DEBUG:boto:Token: None 2022-09-13T07:43:48.826 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-23/obj 2022-09-13T07:43:48.827 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs= 2022-09-13T07:43:48.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YXm4hgnXrxEc3M1sf9PA9XPdkGs=', 'Content-Length': '0'} 2022-09-13T07:43:48.829 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000033ff2faa9d2b0e4-00632034b4-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.830 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-23 zones={test-zone3, test-zone2} 2022-09-13T07:43:48.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2022-09-13T07:43:48.832 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-24 2022-09-13T07:43:48.832 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:43:48.833 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:43:48.833 DEBUG:boto:Method: PUT 2022-09-13T07:43:48.834 DEBUG:boto:Path: /yxvvuh-24/ 2022-09-13T07:43:48.834 DEBUG:boto:Data: 2022-09-13T07:43:48.835 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.835 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.836 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.836 DEBUG:boto:Params: {} 2022-09-13T07:43:48.837 DEBUG:boto:Token: None 2022-09-13T07:43:48.838 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-24/ 2022-09-13T07:43:48.838 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SIxE2bUIX3VjiO3xufmtiWXWPzA= 2022-09-13T07:43:48.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SIxE2bUIX3VjiO3xufmtiWXWPzA=', 'Content-Length': '0'} 2022-09-13T07:43:48.856 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003371ab8789a2a3bb-00632034b4-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.858 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-25 2022-09-13T07:43:48.858 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:43:48.859 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:43:48.859 DEBUG:boto:Method: PUT 2022-09-13T07:43:48.860 DEBUG:boto:Path: /yxvvuh-25/ 2022-09-13T07:43:48.860 DEBUG:boto:Data: 2022-09-13T07:43:48.861 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.861 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:48.862 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.862 DEBUG:boto:Params: {} 2022-09-13T07:43:48.863 DEBUG:boto:Token: None 2022-09-13T07:43:48.864 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-25/ 2022-09-13T07:43:48.864 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IwONmYIxo31mxXXXZ9TzYQqrHMw= 2022-09-13T07:43:48.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IwONmYIxo31mxXXXZ9TzYQqrHMw=', 'Content-Length': '0'} 2022-09-13T07:43:48.906 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000071ecd9a621d89915-00632034b4-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.907 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-26 2022-09-13T07:43:48.908 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:43:48.908 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:43:48.909 DEBUG:boto:Method: PUT 2022-09-13T07:43:48.910 DEBUG:boto:Path: /yxvvuh-26/ 2022-09-13T07:43:48.910 DEBUG:boto:Data: 2022-09-13T07:43:48.911 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.911 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:48.912 DEBUG:boto:Port: 8001 2022-09-13T07:43:48.912 DEBUG:boto:Params: {} 2022-09-13T07:43:48.913 DEBUG:boto:Token: None 2022-09-13T07:43:48.913 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-26/ 2022-09-13T07:43:48.914 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:sPmBp6gZAHsjvOH2T4TLU2vJodg= 2022-09-13T07:43:48.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:sPmBp6gZAHsjvOH2T4TLU2vJodg=', 'Content-Length': '0'} 2022-09-13T07:43:48.957 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000880f21869289db7b-00632034b4-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.959 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:43:48.960 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:43:48.960 DEBUG:boto:path=/yxvvuh-24/?versioning 2022-09-13T07:43:48.961 DEBUG:boto:auth_path=/yxvvuh-24/?versioning 2022-09-13T07:43:48.962 DEBUG:boto:Method: PUT 2022-09-13T07:43:48.962 DEBUG:boto:Path: /yxvvuh-24/?versioning 2022-09-13T07:43:48.963 DEBUG:boto:Data: Enabled Disabled 2022-09-13T07:43:48.964 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.965 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.966 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.966 DEBUG:boto:Params: {} 2022-09-13T07:43:48.967 DEBUG:boto:Token: None 2022-09-13T07:43:48.968 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-24/?versioning 2022-09-13T07:43:48.969 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NXz/2NXlAGD+teGqoBMbaQsljMg= 2022-09-13T07:43:48.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NXz/2NXlAGD+teGqoBMbaQsljMg=', 'Content-Length': '228'} 2022-09-13T07:43:48.977 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000c56feda158f8752-00632034b4-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.978 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:43:48.979 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:43:48.979 DEBUG:boto:path=/yxvvuh-24/?versioning 2022-09-13T07:43:48.980 DEBUG:boto:auth_path=/yxvvuh-24/?versioning 2022-09-13T07:43:48.980 DEBUG:boto:Method: PUT 2022-09-13T07:43:48.981 DEBUG:boto:Path: /yxvvuh-24/?versioning 2022-09-13T07:43:48.982 DEBUG:boto:Data: Suspended Disabled 2022-09-13T07:43:48.983 DEBUG:boto:Headers: {} 2022-09-13T07:43:48.984 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:43:48.985 DEBUG:boto:Port: 8000 2022-09-13T07:43:48.985 DEBUG:boto:Params: {} 2022-09-13T07:43:48.986 DEBUG:boto:Token: None 2022-09-13T07:43:48.987 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:43:48 GMT /yxvvuh-24/?versioning 2022-09-13T07:43:48.988 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:NXz/2NXlAGD+teGqoBMbaQsljMg= 2022-09-13T07:43:48.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:NXz/2NXlAGD+teGqoBMbaQsljMg=', 'Content-Length': '230'} 2022-09-13T07:43:48.997 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000744fabe563ffe063-00632034b4-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:48.998 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:43:48.998 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:43:48.999 DEBUG:boto:path=/yxvvuh-25/?versioning 2022-09-13T07:43:48.999 DEBUG:boto:auth_path=/yxvvuh-25/?versioning 2022-09-13T07:43:49.000 DEBUG:boto:Method: PUT 2022-09-13T07:43:49.000 DEBUG:boto:Path: /yxvvuh-25/?versioning 2022-09-13T07:43:49.001 DEBUG:boto:Data: Enabled Disabled 2022-09-13T07:43:49.001 DEBUG:boto:Headers: {} 2022-09-13T07:43:49.002 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:49.002 DEBUG:boto:Port: 8000 2022-09-13T07:43:49.002 DEBUG:boto:Params: {} 2022-09-13T07:43:49.003 DEBUG:boto:Token: None 2022-09-13T07:43:49.004 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:43:49 GMT /yxvvuh-25/?versioning 2022-09-13T07:43:49.004 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:so6TyEqxUSfEJIR/lozUthMd9Zw= 2022-09-13T07:43:49.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:so6TyEqxUSfEJIR/lozUthMd9Zw=', 'Content-Length': '228'} 2022-09-13T07:43:49.015 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001a9354c612f83024-00632034b5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:49.016 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:43:49.017 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:43:49.017 DEBUG:boto:path=/yxvvuh-25/?versioning 2022-09-13T07:43:49.017 DEBUG:boto:auth_path=/yxvvuh-25/?versioning 2022-09-13T07:43:49.018 DEBUG:boto:Method: PUT 2022-09-13T07:43:49.018 DEBUG:boto:Path: /yxvvuh-25/?versioning 2022-09-13T07:43:49.019 DEBUG:boto:Data: Suspended Disabled 2022-09-13T07:43:49.019 DEBUG:boto:Headers: {} 2022-09-13T07:43:49.020 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:43:49.020 DEBUG:boto:Port: 8000 2022-09-13T07:43:49.021 DEBUG:boto:Params: {} 2022-09-13T07:43:49.021 DEBUG:boto:Token: None 2022-09-13T07:43:49.022 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:43:49 GMT /yxvvuh-25/?versioning 2022-09-13T07:43:49.022 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:so6TyEqxUSfEJIR/lozUthMd9Zw= 2022-09-13T07:43:49.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:so6TyEqxUSfEJIR/lozUthMd9Zw=', 'Content-Length': '230'} 2022-09-13T07:43:49.032 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f4b7f2b75e63dcb2-00632034b5-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:49.033 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:43:49.033 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:43:49.034 DEBUG:boto:path=/yxvvuh-26/?versioning 2022-09-13T07:43:49.034 DEBUG:boto:auth_path=/yxvvuh-26/?versioning 2022-09-13T07:43:49.035 DEBUG:boto:Method: PUT 2022-09-13T07:43:49.035 DEBUG:boto:Path: /yxvvuh-26/?versioning 2022-09-13T07:43:49.036 DEBUG:boto:Data: Enabled Disabled 2022-09-13T07:43:49.036 DEBUG:boto:Headers: {} 2022-09-13T07:43:49.037 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:49.037 DEBUG:boto:Port: 8001 2022-09-13T07:43:49.038 DEBUG:boto:Params: {} 2022-09-13T07:43:49.039 DEBUG:boto:Token: None 2022-09-13T07:43:49.039 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:43:49 GMT /yxvvuh-26/?versioning 2022-09-13T07:43:49.040 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Z5JFAuJClaU+2DY9Oj6KapiLv5Q= 2022-09-13T07:43:49.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Z5JFAuJClaU+2DY9Oj6KapiLv5Q=', 'Content-Length': '228'} 2022-09-13T07:43:49.049 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004136d45ee887084d-00632034b5-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:49.051 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:43:49.051 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:43:49.052 DEBUG:boto:path=/yxvvuh-26/?versioning 2022-09-13T07:43:49.052 DEBUG:boto:auth_path=/yxvvuh-26/?versioning 2022-09-13T07:43:49.053 DEBUG:boto:Method: PUT 2022-09-13T07:43:49.053 DEBUG:boto:Path: /yxvvuh-26/?versioning 2022-09-13T07:43:49.053 DEBUG:boto:Data: Suspended Disabled 2022-09-13T07:43:49.054 DEBUG:boto:Headers: {} 2022-09-13T07:43:49.054 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:43:49.055 DEBUG:boto:Port: 8001 2022-09-13T07:43:49.056 DEBUG:boto:Params: {} 2022-09-13T07:43:49.056 DEBUG:boto:Token: None 2022-09-13T07:43:49.057 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:43:49 GMT /yxvvuh-26/?versioning 2022-09-13T07:43:49.057 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Z5JFAuJClaU+2DY9Oj6KapiLv5Q= 2022-09-13T07:43:49.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:43:49 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Z5JFAuJClaU+2DY9Oj6KapiLv5Q=', 'Content-Length': '230'} 2022-09-13T07:43:49.065 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eda0cd4af98c7e8f-00632034b5-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:43:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:43:49.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:43:49.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:43:49.067 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:43:49.120 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:49.121 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:49.152 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:43:49.164 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:43:49.165 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:43:49.165 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:43:49.165 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:43:49.166 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:43:49.166 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:43:49.166 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:43:49.167 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:43:49.167 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:43:49.167 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:43:49.167 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663055029.060670_194.1", 2022-09-13T07:43:49.168 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:43:49.060670Z" 2022-09-13T07:43:49.168 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:43:49.168 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:43:49.168 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:43:49.169 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T07:43:49.169 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:43:49.169 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:43:49.170 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055029.060670_194.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:43:49.171 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:43:49.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:43:49.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:43:49.173 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:43:49.230 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:49.231 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:49.262 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:49.257+0000 7f34daa24600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:43:49.263 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:49.258+0000 7f34daa24600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:43:49.278 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:43:49.279 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:43:49.279 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:43:49.280 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:43:49.280 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:43:49.281 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:43:49.281 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:43:49.282 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:49.282 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:43:49.283 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:43:49.283 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:43:49.284 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:43:49.284 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:43:49.285 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:43:49.285 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:43:49.286 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:43:49.286 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:43:49.287 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:43:49.287 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:43:49.288 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:43:49.288 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:49.288 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:43:49.289 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:43:49.289 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:43:49.289 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:43:49.289 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:43:49.290 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:43:49.290 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:43:49.290 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:43:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:43:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:43:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:43:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:43:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:43:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:43:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:43:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054937.703313_169.1", 2022-09-13T07:43:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:43:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:43:49.294 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:43:49.294 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:42:17.703313Z", 2022-09-13T07:43:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:43:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:43:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:43:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:43:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:43:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:43:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:43:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:43:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:43:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:43:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:43:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:43:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:43:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:43:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:43:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:43:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:43:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:43:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:43:49Z" 2022-09-13T07:43:49.300 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:43:49.302 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663054937.703313_169.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:42:17.703313Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:43:49Z" } 2022-09-13T07:43:49.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054937.703313_169.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:42:17.703313Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:43:49.303 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055029.060670_194.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:43:49.304 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054937.703313_169.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:43:49.304 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055029.060670_194.1 target=1_1663054937.703313_169.1 2022-09-13T07:43:54.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:43:54.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:43:54.308 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:43:54.358 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:54.358 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:54.394 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:54.391+0000 7f2e97445600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:43:54.395 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:54.392+0000 7f2e97445600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:43:54.412 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:43:54.413 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:43:54.413 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:43:54.414 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:43:54.414 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:43:54.415 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:43:54.415 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:43:54.416 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:54.416 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:43:54.417 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:43:54.417 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:43:54.417 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:43:54.418 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:43:54.419 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:43:54.419 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:43:54.420 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:43:54.420 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:43:54.421 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:43:54.421 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:43:54.421 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:43:54.422 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:54.422 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:43:54.423 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:43:54.423 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:43:54.424 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:43:54.424 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:43:54.425 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:43:54.425 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:43:54.426 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:43:54.426 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:43:54.427 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:43:54.427 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:43:54.427 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:54.428 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:43:54.428 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:43:54.429 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:43:54.430 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:43:54.431 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055028.925399_179.1", 2022-09-13T07:43:54.431 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:43:54.431 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:43:54.432 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:43:54.432 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:43:48.925399Z", 2022-09-13T07:43:54.433 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:43:54.433 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:43:54.433 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:54.434 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:43:54.434 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:43:54.435 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:43:54.435 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:43:54.435 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:43:54.436 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:43:54.436 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:43:54.437 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:43:54.438 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:43:54.438 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:43:54.439 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:43:54.439 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:43:54.440 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:43:54.440 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:54.440 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:43:54.441 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:43:54.441 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:43:54.442 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:54.442 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:43:54Z" 2022-09-13T07:43:54.443 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:43:54.444 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055028.925399_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:43:48.925399Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:43:54Z" } 2022-09-13T07:43:54.445 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055028.925399_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:43:48.925399Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:43:54.446 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055029.060670_194.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:43:54.446 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055028.925399_179.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:43:54.447 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055029.060670_194.1 target=1_1663055028.925399_179.1 2022-09-13T07:43:59.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:43:59.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:43:59.451 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:43:59.503 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:43:59.503 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:43:59.556 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:59.551+0000 7fdce0ade600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:43:59.556 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:43:59.552+0000 7fdce0ade600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:43:59.578 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:43:59.579 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:43:59.579 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:43:59.580 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:43:59.580 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:43:59.580 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:43:59.581 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:43:59.581 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:59.582 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:43:59.582 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:43:59.583 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:43:59.583 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:43:59.584 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:43:59.584 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:43:59.585 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:43:59.585 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:43:59.585 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:43:59.586 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:43:59.586 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:43:59.587 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:43:59.587 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:59.588 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:43:59.588 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:43:59.589 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:43:59.589 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:43:59.589 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:43:59.590 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:43:59.591 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:43:59.591 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:43:59.591 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:43:59.592 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:43:59.592 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:43:59.592 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:59.592 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:43:59.593 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:43:59.593 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:43:59.593 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:43:59.593 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055028.843973_171.1", 2022-09-13T07:43:59.594 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:43:59.594 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:43:59.595 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:43:59.602 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:43:48.843973Z", 2022-09-13T07:43:59.602 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:43:59.603 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:43:59.603 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:59.603 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:43:59.604 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:43:59.604 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:43:59.604 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:43:59.604 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:43:59.605 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:43:59.605 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:43:59.605 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:43:59.605 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:43:59.606 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:43:59.606 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:43:59.606 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:43:59.607 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:43:59.607 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:59.607 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:43:59.607 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:43:59.608 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:43:59.608 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:43:59.608 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:43:59Z" 2022-09-13T07:43:59.608 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:43:59.610 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055028.843973_171.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:43:48.843973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:43:59Z" } 2022-09-13T07:43:59.610 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055028.843973_171.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:43:48.843973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:43:59.611 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055029.060670_194.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:43:59.611 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055028.843973_171.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:43:59.612 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055029.060670_194.1 target=1_1663055028.843973_171.1 2022-09-13T07:44:04.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:44:04.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:44:04.617 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:44:04.670 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:04.671 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:04.702 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:04.695+0000 7f021914e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:44:04.702 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:04.696+0000 7f021914e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:44:04.717 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:44:04.717 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:44:04.717 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:44:04.718 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:44:04.718 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:44:04.718 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:44:04.718 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:44:04.719 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:44:04.719 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:44:04.719 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:44:04.719 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:44:04.720 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:44:04.720 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:44:04.720 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:44:04.720 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:44:04.721 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:44:04.721 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:44:04.721 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:44:04.722 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:44:04.723 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:44:04.723 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:44:04.723 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:44:04.724 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:44:04.724 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:44:04.724 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:44:04.724 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:44:04.725 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:44:04.725 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:44:04.725 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:44:04.725 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:44:04.726 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:44:04.726 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:44:04.726 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:44:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:44:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:44:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:44:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:44:04.728 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055028.884630_176.1", 2022-09-13T07:44:04.728 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:44:04.728 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:44:04.729 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:44:04.729 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:43:48.884630Z", 2022-09-13T07:44:04.729 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:44:04.729 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:44:04.730 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:44:04.730 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:44:04.730 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:44:04.730 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:44:04.731 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:44:04.731 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:44:04.731 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:44:04.731 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:44:04.732 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:44:04.732 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:44:04.732 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:44:04.732 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:44:04.733 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:44:04.733 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:44:04.733 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:44:04.733 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:44:04.734 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:44:04.734 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:44:04.734 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:44:04.735 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:44:04Z" 2022-09-13T07:44:04.735 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:44:04.735 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055028.884630_176.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:43:48.884630Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:44:04Z" } 2022-09-13T07:44:04.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055028.884630_176.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:43:48.884630Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:44:04.737 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055029.060670_194.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:44:04.737 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055028.884630_176.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:44:04.738 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055029.060670_194.1 target=1_1663055028.884630_176.1 2022-09-13T07:44:09.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:44:09.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:44:09.742 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:44:09.801 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:09.857 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:09.852+0000 7fa37ddd9600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:44:09.858 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:09.853+0000 7fa37ddd9600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:44:09.881 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:44:09.882 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:44:09.882 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:44:09.883 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:44:09.883 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:44:09.883 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:44:09.884 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:44:09.884 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:44:09.885 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:44:09.885 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:44:09.885 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:44:09.886 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:44:09.886 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:44:09.887 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:44:09.887 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:44:09.891 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:44:09.891 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:44:09.891 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:44:09.892 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:44:09.892 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:44:09.892 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:44:09.892 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:44:09.893 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:44:09.893 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:44:09.893 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:44:09.894 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:44:09.894 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:44:09.894 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:44:09.894 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:44:09.895 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:44:09.895 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:44:09.895 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:44:09.896 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:44:09.896 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:44:09.896 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:44:09.896 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:44:09.897 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:44:09.897 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055029.060670_194.1", 2022-09-13T07:44:09.897 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:44:09.897 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:44:09.898 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:44:09.898 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:43:49.060670Z", 2022-09-13T07:44:09.899 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:44:09.899 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:44:09.899 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:44:09.900 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:44:09.900 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:44:09.901 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:44:09.901 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:44:09.901 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:44:09.902 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:44:09.902 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:44:09.902 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:44:09.902 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:44:09.903 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:44:09.903 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:44:09.904 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:44:09.904 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:44:09.904 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:44:09.904 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:44:09.905 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:44:09.905 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:44:09.905 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:44:09.906 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:44:09Z" 2022-09-13T07:44:09.906 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:44:09.907 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055029.060670_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:43:49.060670Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:44:09Z" } 2022-09-13T07:44:09.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055029.060670_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:43:49.060670Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:44:09.909 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055029.060670_194.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:44:09.909 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055029.060670_194.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:44:09.910 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-13T07:44:09.910 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-13T07:44:09.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:44:09.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:44:09.915 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:44:09.975 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:09.976 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:10.009 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:10.006+0000 7f8cd0d7f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:44:10.010 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:10.007+0000 7f8cd0d7f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:44:10.030 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T07:44:10.031 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": { 2022-09-13T07:44:10.031 INFO:teuthology.orchestra.run.smithi036.stdout: "info": { 2022-09-13T07:44:10.032 INFO:teuthology.orchestra.run.smithi036.stdout: "status": "sync", 2022-09-13T07:44:10.032 INFO:teuthology.orchestra.run.smithi036.stdout: "num_shards": 4, 2022-09-13T07:44:10.033 INFO:teuthology.orchestra.run.smithi036.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:44:10.033 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:44:10.033 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:44:10.034 INFO:teuthology.orchestra.run.smithi036.stdout: "markers": [ 2022-09-13T07:44:10.034 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:44:10.035 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 0, 2022-09-13T07:44:10.036 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:44:10.036 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:44:10.037 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:44:10.037 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:44:10.037 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 1, 2022-09-13T07:44:10.038 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:44:10.039 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:44:10.039 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:44:10.040 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:44:10.040 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:44:10.040 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:44:10.041 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 1, 2022-09-13T07:44:10.041 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:44:10.042 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:44:10.042 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:44:10.042 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:44:10.043 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:44:10.043 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:44:10.044 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "0.000000", 2022-09-13T07:44:10.044 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:44:10.044 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:44:10.045 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:44:10.045 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:44:10.047 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 2, 2022-09-13T07:44:10.047 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:44:10.048 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:44:10.048 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663055029.060670_194.1", 2022-09-13T07:44:10.048 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:44:10.049 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:44:10.049 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:44:10.049 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:43:49.060670Z", 2022-09-13T07:44:10.049 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:44:10.050 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:44:10.050 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:44:10.050 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:44:10.051 INFO:teuthology.orchestra.run.smithi036.stdout: "key": 3, 2022-09-13T07:44:10.051 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T07:44:10.051 INFO:teuthology.orchestra.run.smithi036.stdout: "state": 1, 2022-09-13T07:44:10.052 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:44:10.052 INFO:teuthology.orchestra.run.smithi036.stdout: "next_step_marker": "", 2022-09-13T07:44:10.052 INFO:teuthology.orchestra.run.smithi036.stdout: "total_entries": 0, 2022-09-13T07:44:10.052 INFO:teuthology.orchestra.run.smithi036.stdout: "pos": 0, 2022-09-13T07:44:10.053 INFO:teuthology.orchestra.run.smithi036.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:44:10.053 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T07:44:10.053 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:44:10.053 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:44:10.054 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T07:44:10.054 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:44:10.054 INFO:teuthology.orchestra.run.smithi036.stdout: "full_sync": { 2022-09-13T07:44:10.055 INFO:teuthology.orchestra.run.smithi036.stdout: "total": 1, 2022-09-13T07:44:10.055 INFO:teuthology.orchestra.run.smithi036.stdout: "complete": 1 2022-09-13T07:44:10.055 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:44:10.055 INFO:teuthology.orchestra.run.smithi036.stdout: "current_time": "2022-09-13T07:44:10Z" 2022-09-13T07:44:10.056 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T07:44:10.057 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055029.060670_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:43:49.060670Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:44:10Z" } 2022-09-13T07:44:10.058 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055029.060670_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:43:49.060670Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:44:10.058 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055029.060670_194.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:44:10.060 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055029.060670_194.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:44:10.061 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-13T07:44:10.062 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:10.063 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:10.064 DEBUG:boto:Method: HEAD 2022-09-13T07:44:10.065 DEBUG:boto:Path: /yxvvuh-24/ 2022-09-13T07:44:10.066 DEBUG:boto:Data: 2022-09-13T07:44:10.067 DEBUG:boto:Headers: {} 2022-09-13T07:44:10.068 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:10.068 DEBUG:boto:Port: 8000 2022-09-13T07:44:10.069 DEBUG:boto:Params: {} 2022-09-13T07:44:10.070 DEBUG:boto:Token: None 2022-09-13T07:44:10.071 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:10 GMT /yxvvuh-24/ 2022-09-13T07:44:10.072 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:s3+BS7mDOwTWHZs9eScKJEy0Y38= 2022-09-13T07:44:10.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:s3+BS7mDOwTWHZs9eScKJEy0Y38=', 'Content-Length': '0'} 2022-09-13T07:44:10.076 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004550e2baed6888c0-00632034ca-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:10.077 DEBUG:boto:path=/yxvvuh-24/obj 2022-09-13T07:44:10.078 DEBUG:boto:auth_path=/yxvvuh-24/obj 2022-09-13T07:44:10.079 DEBUG:boto:Method: PUT 2022-09-13T07:44:10.079 DEBUG:boto:Path: /yxvvuh-24/obj 2022-09-13T07:44:10.080 DEBUG:boto:Data: 2022-09-13T07:44:10.080 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T07:44:10.081 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:10.081 DEBUG:boto:Port: 8000 2022-09-13T07:44:10.081 DEBUG:boto:Params: {} 2022-09-13T07:44:10.082 DEBUG:boto:Token: None 2022-09-13T07:44:10.083 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 13 Sep 2022 07:44:10 GMT /yxvvuh-24/obj 2022-09-13T07:44:10.083 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:bedv7EckHeVZVEuC/oacA8dK/68= 2022-09-13T07:44:10.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:44:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bedv7EckHeVZVEuC/oacA8dK/68='} 2022-09-13T07:44:10.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea43b42c5cc89ccb-00632034ca-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:44:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:10.092 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:10.093 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:10.094 DEBUG:boto:Method: HEAD 2022-09-13T07:44:10.095 DEBUG:boto:Path: /yxvvuh-24/ 2022-09-13T07:44:10.096 DEBUG:boto:Data: 2022-09-13T07:44:10.097 DEBUG:boto:Headers: {} 2022-09-13T07:44:10.098 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:10.099 DEBUG:boto:Port: 8000 2022-09-13T07:44:10.099 DEBUG:boto:Params: {} 2022-09-13T07:44:10.100 DEBUG:boto:Token: None 2022-09-13T07:44:10.101 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:10 GMT /yxvvuh-24/ 2022-09-13T07:44:10.102 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:s3+BS7mDOwTWHZs9eScKJEy0Y38= 2022-09-13T07:44:10.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:s3+BS7mDOwTWHZs9eScKJEy0Y38=', 'Content-Length': '0'} 2022-09-13T07:44:10.106 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024ff025b2febe88c-00632034ca-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:10.107 DEBUG:boto:path=/yxvvuh-24/obj 2022-09-13T07:44:10.108 DEBUG:boto:auth_path=/yxvvuh-24/obj 2022-09-13T07:44:10.109 DEBUG:boto:Method: DELETE 2022-09-13T07:44:10.110 DEBUG:boto:Path: /yxvvuh-24/obj 2022-09-13T07:44:10.111 DEBUG:boto:Data: 2022-09-13T07:44:10.112 DEBUG:boto:Headers: {} 2022-09-13T07:44:10.112 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:10.113 DEBUG:boto:Port: 8000 2022-09-13T07:44:10.114 DEBUG:boto:Params: {} 2022-09-13T07:44:10.115 DEBUG:boto:Token: None 2022-09-13T07:44:10.115 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:44:10 GMT /yxvvuh-24/obj 2022-09-13T07:44:10.116 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:o9ZPTyAYtZuJ0SdpL2RdgBzpR6U= 2022-09-13T07:44:10.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:o9ZPTyAYtZuJ0SdpL2RdgBzpR6U=', 'Content-Length': '0'} 2022-09-13T07:44:10.133 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000009a6f8d49b98e1f58-00632034ca-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:44:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:10.134 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:10.135 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:10.135 DEBUG:boto:Method: HEAD 2022-09-13T07:44:10.136 DEBUG:boto:Path: /yxvvuh-25/ 2022-09-13T07:44:10.136 DEBUG:boto:Data: 2022-09-13T07:44:10.137 DEBUG:boto:Headers: {} 2022-09-13T07:44:10.137 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:10.138 DEBUG:boto:Port: 8000 2022-09-13T07:44:10.138 DEBUG:boto:Params: {} 2022-09-13T07:44:10.139 DEBUG:boto:Token: None 2022-09-13T07:44:10.139 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:10 GMT /yxvvuh-25/ 2022-09-13T07:44:10.140 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4Ody9815BbfLifC32PbuEPG8DU4= 2022-09-13T07:44:10.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4Ody9815BbfLifC32PbuEPG8DU4=', 'Content-Length': '0'} 2022-09-13T07:44:10.143 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad481b3bfcca5246-00632034ca-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:10.144 DEBUG:boto:path=/yxvvuh-25/obj 2022-09-13T07:44:10.144 DEBUG:boto:auth_path=/yxvvuh-25/obj 2022-09-13T07:44:10.145 DEBUG:boto:Method: PUT 2022-09-13T07:44:10.145 DEBUG:boto:Path: /yxvvuh-25/obj 2022-09-13T07:44:10.146 DEBUG:boto:Data: 2022-09-13T07:44:10.147 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T07:44:10.147 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:10.148 DEBUG:boto:Port: 8000 2022-09-13T07:44:10.149 DEBUG:boto:Params: {} 2022-09-13T07:44:10.149 DEBUG:boto:Token: None 2022-09-13T07:44:10.150 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 13 Sep 2022 07:44:10 GMT /yxvvuh-25/obj 2022-09-13T07:44:10.151 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:x3UR34aetiWWg+eF0AaaOiaEvDI= 2022-09-13T07:44:10.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:44:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x3UR34aetiWWg+eF0AaaOiaEvDI='} 2022-09-13T07:44:10.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000265bf004226617cc-00632034ca-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:44:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:10.195 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:10.196 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:10.196 DEBUG:boto:Method: HEAD 2022-09-13T07:44:10.197 DEBUG:boto:Path: /yxvvuh-25/ 2022-09-13T07:44:10.198 DEBUG:boto:Data: 2022-09-13T07:44:10.198 DEBUG:boto:Headers: {} 2022-09-13T07:44:10.199 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:10.199 DEBUG:boto:Port: 8000 2022-09-13T07:44:10.200 DEBUG:boto:Params: {} 2022-09-13T07:44:10.200 DEBUG:boto:Token: None 2022-09-13T07:44:10.201 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:10 GMT /yxvvuh-25/ 2022-09-13T07:44:10.202 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:4Ody9815BbfLifC32PbuEPG8DU4= 2022-09-13T07:44:10.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:4Ody9815BbfLifC32PbuEPG8DU4=', 'Content-Length': '0'} 2022-09-13T07:44:10.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a809bdfe0f0c9ef9-00632034ca-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:10.205 DEBUG:boto:path=/yxvvuh-25/obj 2022-09-13T07:44:10.206 DEBUG:boto:auth_path=/yxvvuh-25/obj 2022-09-13T07:44:10.206 DEBUG:boto:Method: DELETE 2022-09-13T07:44:10.207 DEBUG:boto:Path: /yxvvuh-25/obj 2022-09-13T07:44:10.207 DEBUG:boto:Data: 2022-09-13T07:44:10.208 DEBUG:boto:Headers: {} 2022-09-13T07:44:10.208 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:10.209 DEBUG:boto:Port: 8000 2022-09-13T07:44:10.210 DEBUG:boto:Params: {} 2022-09-13T07:44:10.211 DEBUG:boto:Token: None 2022-09-13T07:44:10.212 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:44:10 GMT /yxvvuh-25/obj 2022-09-13T07:44:10.212 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Wu1KFL13wZu5jNKxsy7VMUNz/xw= 2022-09-13T07:44:10.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Wu1KFL13wZu5jNKxsy7VMUNz/xw=', 'Content-Length': '0'} 2022-09-13T07:44:10.230 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000005a4b47a99f70185b-00632034ca-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:44:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:10.231 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:10.232 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:10.233 DEBUG:boto:Method: HEAD 2022-09-13T07:44:10.234 DEBUG:boto:Path: /yxvvuh-26/ 2022-09-13T07:44:10.235 DEBUG:boto:Data: 2022-09-13T07:44:10.235 DEBUG:boto:Headers: {} 2022-09-13T07:44:10.236 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:10.237 DEBUG:boto:Port: 8001 2022-09-13T07:44:10.238 DEBUG:boto:Params: {} 2022-09-13T07:44:10.239 DEBUG:boto:Token: None 2022-09-13T07:44:10.240 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:10 GMT /yxvvuh-26/ 2022-09-13T07:44:10.241 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BzgJMR2e7Xpe/yba31MqVfM78cE= 2022-09-13T07:44:10.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BzgJMR2e7Xpe/yba31MqVfM78cE=', 'Content-Length': '0'} 2022-09-13T07:44:10.245 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000252bd2e02b943e2c-00632034ca-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:10.246 DEBUG:boto:path=/yxvvuh-26/obj 2022-09-13T07:44:10.246 DEBUG:boto:auth_path=/yxvvuh-26/obj 2022-09-13T07:44:10.247 DEBUG:boto:Method: PUT 2022-09-13T07:44:10.247 DEBUG:boto:Path: /yxvvuh-26/obj 2022-09-13T07:44:10.248 DEBUG:boto:Data: 2022-09-13T07:44:10.250 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T07:44:10.251 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:10.252 DEBUG:boto:Port: 8001 2022-09-13T07:44:10.253 DEBUG:boto:Params: {} 2022-09-13T07:44:10.254 DEBUG:boto:Token: None 2022-09-13T07:44:10.255 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 13 Sep 2022 07:44:10 GMT /yxvvuh-26/obj 2022-09-13T07:44:10.256 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:yAVeVqltXTtIpwh5NI3CYwnDmRk= 2022-09-13T07:44:10.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:44:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yAVeVqltXTtIpwh5NI3CYwnDmRk='} 2022-09-13T07:44:10.268 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b070b8c42f2476d3-00632034ca-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:44:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:10.269 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:10.270 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:10.271 DEBUG:boto:Method: HEAD 2022-09-13T07:44:10.272 DEBUG:boto:Path: /yxvvuh-26/ 2022-09-13T07:44:10.273 DEBUG:boto:Data: 2022-09-13T07:44:10.274 DEBUG:boto:Headers: {} 2022-09-13T07:44:10.275 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:10.276 DEBUG:boto:Port: 8001 2022-09-13T07:44:10.277 DEBUG:boto:Params: {} 2022-09-13T07:44:10.278 DEBUG:boto:Token: None 2022-09-13T07:44:10.278 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:10 GMT /yxvvuh-26/ 2022-09-13T07:44:10.279 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BzgJMR2e7Xpe/yba31MqVfM78cE= 2022-09-13T07:44:10.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BzgJMR2e7Xpe/yba31MqVfM78cE=', 'Content-Length': '0'} 2022-09-13T07:44:10.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064dace6c9ee2e08e-00632034ca-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:10.285 DEBUG:boto:path=/yxvvuh-26/obj 2022-09-13T07:44:10.286 DEBUG:boto:auth_path=/yxvvuh-26/obj 2022-09-13T07:44:10.287 DEBUG:boto:Method: DELETE 2022-09-13T07:44:10.288 DEBUG:boto:Path: /yxvvuh-26/obj 2022-09-13T07:44:10.289 DEBUG:boto:Data: 2022-09-13T07:44:10.289 DEBUG:boto:Headers: {} 2022-09-13T07:44:10.290 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:10.290 DEBUG:boto:Port: 8001 2022-09-13T07:44:10.291 DEBUG:boto:Params: {} 2022-09-13T07:44:10.291 DEBUG:boto:Token: None 2022-09-13T07:44:10.292 DEBUG:boto:StringToSign: DELETE Tue, 13 Sep 2022 07:44:10 GMT /yxvvuh-26/obj 2022-09-13T07:44:10.293 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZW7lP3KyWvGT3L3qO+ymlhEX8g8= 2022-09-13T07:44:10.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZW7lP3KyWvGT3L3qO+ymlhEX8g8=', 'Content-Length': '0'} 2022-09-13T07:44:10.310 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000098195f9f7346aa29-00632034ca-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:44:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:10.311 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-24 2022-09-13T07:44:10.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:10.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:10.314 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:10.371 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:10.372 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:10.409 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:10.405+0000 7f443563e600 1 waiting to reach incremental sync.. 2022-09-13T07:44:15.409 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:15.405+0000 7f443563e600 1 waiting to reach incremental sync.. 2022-09-13T07:44:20.411 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:20.406+0000 7f443563e600 1 waiting to reach incremental sync.. 2022-09-13T07:44:25.411 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:25.406+0000 7f443563e600 1 waiting to reach incremental sync.. 2022-09-13T07:44:30.413 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:30.408+0000 7f443563e600 1 bucket sync caught up with source: 2022-09-13T07:44:30.413 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000003.28.6, , ] 2022-09-13T07:44:30.414 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000003.28.6, , ] 2022-09-13T07:44:30.414 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:30.408+0000 7f443563e600 0 bucket checkpoint complete 2022-09-13T07:44:30.430 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-24 2022-09-13T07:44:30.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:30.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:30.433 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:30.500 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:30.501 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:30.541 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:30.537+0000 7f0151e0a600 1 bucket sync caught up with source: 2022-09-13T07:44:30.542 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000003.28.6, , ] 2022-09-13T07:44:30.542 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000003.28.6, , ] 2022-09-13T07:44:30.543 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:30.537+0000 7f0151e0a600 0 bucket checkpoint complete 2022-09-13T07:44:30.558 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-24 2022-09-13T07:44:30.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:30.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:30.560 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:30.618 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:30.619 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:30.665 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:30.661+0000 7f298da1e600 1 waiting to reach incremental sync.. 2022-09-13T07:44:35.666 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:35.662+0000 7f298da1e600 1 waiting to reach incremental sync.. 2022-09-13T07:44:40.667 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:40.662+0000 7f298da1e600 1 waiting to reach incremental sync.. 2022-09-13T07:44:45.667 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:45.663+0000 7f298da1e600 1 waiting to reach incremental sync.. 2022-09-13T07:44:50.670 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:50.665+0000 7f298da1e600 1 bucket sync caught up with source: 2022-09-13T07:44:50.670 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000001.58.12, , ] 2022-09-13T07:44:50.670 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000001.58.12, , ] 2022-09-13T07:44:50.671 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:50.665+0000 7f298da1e600 0 bucket checkpoint complete 2022-09-13T07:44:50.689 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-24 2022-09-13T07:44:50.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:50.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:50.692 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:50.756 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:50.757 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:50.805 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:50.801+0000 7f800c672600 1 bucket sync caught up with source: 2022-09-13T07:44:50.806 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000001.58.12, , ] 2022-09-13T07:44:50.806 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000001.58.12, , ] 2022-09-13T07:44:50.806 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:50.801+0000 7f800c672600 0 bucket checkpoint complete 2022-09-13T07:44:50.819 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-24 2022-09-13T07:44:50.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:50.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:50.822 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:50.881 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:50.881 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:50.918 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:50.914+0000 7f9aea156600 1 bucket sync caught up with source: 2022-09-13T07:44:50.918 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000001.81.12, , ] 2022-09-13T07:44:50.918 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000001.81.12, , ] 2022-09-13T07:44:50.919 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:50.914+0000 7f9aea156600 0 bucket checkpoint complete 2022-09-13T07:44:50.931 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-24 2022-09-13T07:44:50.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:50.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:50.933 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:50.997 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:50.998 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:51.044 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:51.038+0000 7f32f8878600 1 bucket sync caught up with source: 2022-09-13T07:44:51.044 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000001.81.12, , ] 2022-09-13T07:44:51.045 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000001.81.12, , ] 2022-09-13T07:44:51.045 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:51.038+0000 7f32f8878600 0 bucket checkpoint complete 2022-09-13T07:44:51.057 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-24 zones={test-zone2, test-zone1} 2022-09-13T07:44:51.059 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.060 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.061 DEBUG:boto:Method: HEAD 2022-09-13T07:44:51.062 DEBUG:boto:Path: /yxvvuh-24/ 2022-09-13T07:44:51.063 DEBUG:boto:Data: 2022-09-13T07:44:51.064 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.065 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:51.066 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.067 DEBUG:boto:Params: {} 2022-09-13T07:44:51.068 DEBUG:boto:Token: None 2022-09-13T07:44:51.069 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/ 2022-09-13T07:44:51.070 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TGKDRbSlfQNP1HsI3BD7VSvnEJY= 2022-09-13T07:44:51.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TGKDRbSlfQNP1HsI3BD7VSvnEJY=', 'Content-Length': '0'} 2022-09-13T07:44:51.075 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c46e1e4921577613-00632034f3-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.076 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.077 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.078 DEBUG:boto:Method: HEAD 2022-09-13T07:44:51.079 DEBUG:boto:Path: /yxvvuh-24/ 2022-09-13T07:44:51.080 DEBUG:boto:Data: 2022-09-13T07:44:51.081 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.082 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:51.083 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.083 DEBUG:boto:Params: {} 2022-09-13T07:44:51.084 DEBUG:boto:Token: None 2022-09-13T07:44:51.084 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/ 2022-09-13T07:44:51.085 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TGKDRbSlfQNP1HsI3BD7VSvnEJY= 2022-09-13T07:44:51.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TGKDRbSlfQNP1HsI3BD7VSvnEJY=', 'Content-Length': '0'} 2022-09-13T07:44:51.089 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af1796cfaf008744-00632034f3-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.090 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:44:51.090 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.091 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.091 DEBUG:boto:path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.092 DEBUG:boto:auth_path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.093 DEBUG:boto:Method: GET 2022-09-13T07:44:51.094 DEBUG:boto:Path: /yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.095 DEBUG:boto:Data: 2022-09-13T07:44:51.096 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.097 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:51.098 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.099 DEBUG:boto:Params: {} 2022-09-13T07:44:51.100 DEBUG:boto:Token: None 2022-09-13T07:44:51.101 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/?versions 2022-09-13T07:44:51.102 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc= 2022-09-13T07:44:51.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc=', 'Content-Length': '0'} 2022-09-13T07:44:51.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003849617f521b2a63-00632034f3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.107 DEBUG:boto:b'yxvvuh-24999falseobjnulltrue2022-09-13T07:44:10.116Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:51.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:51.110 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:44:51.111 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.112 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.113 DEBUG:boto:path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.114 DEBUG:boto:auth_path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.115 DEBUG:boto:Method: GET 2022-09-13T07:44:51.116 DEBUG:boto:Path: /yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.116 DEBUG:boto:Data: 2022-09-13T07:44:51.117 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.117 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:51.118 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.118 DEBUG:boto:Params: {} 2022-09-13T07:44:51.119 DEBUG:boto:Token: None 2022-09-13T07:44:51.120 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/?versions 2022-09-13T07:44:51.120 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc= 2022-09-13T07:44:51.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc=', 'Content-Length': '0'} 2022-09-13T07:44:51.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004649c84392cfbcb4-00632034f3-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.127 DEBUG:boto:b'yxvvuh-24999falseobjnulltrue2022-09-13T07:44:10.116Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:51.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:51.128 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.129 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.129 DEBUG:boto:path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.130 DEBUG:boto:auth_path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.132 DEBUG:boto:Method: GET 2022-09-13T07:44:51.133 DEBUG:boto:Path: /yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.134 DEBUG:boto:Data: 2022-09-13T07:44:51.135 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.136 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:51.137 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.138 DEBUG:boto:Params: {} 2022-09-13T07:44:51.139 DEBUG:boto:Token: None 2022-09-13T07:44:51.140 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/?versions 2022-09-13T07:44:51.141 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc= 2022-09-13T07:44:51.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc=', 'Content-Length': '0'} 2022-09-13T07:44:51.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000829dbed6fa4380ab-00632034f3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.186 DEBUG:boto:b'yxvvuh-24999falseobjnulltrue2022-09-13T07:44:10.116Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:51.187 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.188 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.188 DEBUG:boto:path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.190 DEBUG:boto:auth_path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.190 DEBUG:boto:Method: GET 2022-09-13T07:44:51.191 DEBUG:boto:Path: /yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.191 DEBUG:boto:Data: 2022-09-13T07:44:51.192 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.192 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:51.193 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.193 DEBUG:boto:Params: {} 2022-09-13T07:44:51.194 DEBUG:boto:Token: None 2022-09-13T07:44:51.195 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/?versions 2022-09-13T07:44:51.195 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc= 2022-09-13T07:44:51.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc=', 'Content-Length': '0'} 2022-09-13T07:44:51.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d956a14e27bc5d7-00632034f3-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.212 DEBUG:boto:b'yxvvuh-24999falseobjnulltrue2022-09-13T07:44:10.116Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:51.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:44:51.214 DEBUG:boto:path=/yxvvuh-24/obj 2022-09-13T07:44:51.215 DEBUG:boto:auth_path=/yxvvuh-24/obj 2022-09-13T07:44:51.216 DEBUG:boto:Method: HEAD 2022-09-13T07:44:51.217 DEBUG:boto:Path: /yxvvuh-24/obj 2022-09-13T07:44:51.218 DEBUG:boto:Data: 2022-09-13T07:44:51.219 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.220 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:51.221 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.222 DEBUG:boto:Params: {} 2022-09-13T07:44:51.223 DEBUG:boto:Token: None 2022-09-13T07:44:51.224 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/obj 2022-09-13T07:44:51.225 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:q4aDdnkBH5SITzCfLHhNA0RbQ7E= 2022-09-13T07:44:51.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:q4aDdnkBH5SITzCfLHhNA0RbQ7E=', 'Content-Length': '0'} 2022-09-13T07:44:51.229 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000acb885e8d9dfaeb5-00632034f3-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.230 DEBUG:boto:path=/yxvvuh-24/obj 2022-09-13T07:44:51.231 DEBUG:boto:auth_path=/yxvvuh-24/obj 2022-09-13T07:44:51.232 DEBUG:boto:Method: HEAD 2022-09-13T07:44:51.233 DEBUG:boto:Path: /yxvvuh-24/obj 2022-09-13T07:44:51.234 DEBUG:boto:Data: 2022-09-13T07:44:51.235 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.236 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:51.237 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.238 DEBUG:boto:Params: {} 2022-09-13T07:44:51.239 DEBUG:boto:Token: None 2022-09-13T07:44:51.241 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/obj 2022-09-13T07:44:51.242 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:q4aDdnkBH5SITzCfLHhNA0RbQ7E= 2022-09-13T07:44:51.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:q4aDdnkBH5SITzCfLHhNA0RbQ7E=', 'Content-Length': '0'} 2022-09-13T07:44:51.246 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000034978ffac27dc8cc-00632034f3-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.247 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-24 zones={test-zone2, test-zone1} 2022-09-13T07:44:51.248 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-24 zones={test-zone3, test-zone1} 2022-09-13T07:44:51.248 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.249 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.250 DEBUG:boto:Method: HEAD 2022-09-13T07:44:51.251 DEBUG:boto:Path: /yxvvuh-24/ 2022-09-13T07:44:51.252 DEBUG:boto:Data: 2022-09-13T07:44:51.253 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.254 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:51.255 DEBUG:boto:Port: 8001 2022-09-13T07:44:51.256 DEBUG:boto:Params: {} 2022-09-13T07:44:51.257 DEBUG:boto:Token: None 2022-09-13T07:44:51.258 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/ 2022-09-13T07:44:51.259 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TGKDRbSlfQNP1HsI3BD7VSvnEJY= 2022-09-13T07:44:51.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TGKDRbSlfQNP1HsI3BD7VSvnEJY=', 'Content-Length': '0'} 2022-09-13T07:44:51.264 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000af8d0c3fec0f18f-00632034f3-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.265 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.266 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.267 DEBUG:boto:Method: HEAD 2022-09-13T07:44:51.268 DEBUG:boto:Path: /yxvvuh-24/ 2022-09-13T07:44:51.269 DEBUG:boto:Data: 2022-09-13T07:44:51.270 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.271 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:51.272 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.273 DEBUG:boto:Params: {} 2022-09-13T07:44:51.274 DEBUG:boto:Token: None 2022-09-13T07:44:51.275 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/ 2022-09-13T07:44:51.276 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TGKDRbSlfQNP1HsI3BD7VSvnEJY= 2022-09-13T07:44:51.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TGKDRbSlfQNP1HsI3BD7VSvnEJY=', 'Content-Length': '0'} 2022-09-13T07:44:51.280 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce4a6b89b68dd655-00632034f3-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.281 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:44:51.282 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.283 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.284 DEBUG:boto:path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.285 DEBUG:boto:auth_path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.286 DEBUG:boto:Method: GET 2022-09-13T07:44:51.287 DEBUG:boto:Path: /yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.288 DEBUG:boto:Data: 2022-09-13T07:44:51.289 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.290 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:51.291 DEBUG:boto:Port: 8001 2022-09-13T07:44:51.292 DEBUG:boto:Params: {} 2022-09-13T07:44:51.293 DEBUG:boto:Token: None 2022-09-13T07:44:51.295 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/?versions 2022-09-13T07:44:51.296 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc= 2022-09-13T07:44:51.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc=', 'Content-Length': '0'} 2022-09-13T07:44:51.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000efdc41030af2da6-00632034f3-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.301 DEBUG:boto:b'yxvvuh-24999falseobjnulltrue2022-09-13T07:44:10.116Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:51.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:51.303 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:44:51.303 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.304 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.304 DEBUG:boto:path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.305 DEBUG:boto:auth_path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.305 DEBUG:boto:Method: GET 2022-09-13T07:44:51.306 DEBUG:boto:Path: /yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.306 DEBUG:boto:Data: 2022-09-13T07:44:51.307 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.307 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:51.308 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.308 DEBUG:boto:Params: {} 2022-09-13T07:44:51.309 DEBUG:boto:Token: None 2022-09-13T07:44:51.309 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/?versions 2022-09-13T07:44:51.310 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc= 2022-09-13T07:44:51.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc=', 'Content-Length': '0'} 2022-09-13T07:44:51.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a649789929f5defb-00632034f3-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.314 DEBUG:boto:b'yxvvuh-24999falseobjnulltrue2022-09-13T07:44:10.116Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:51.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:51.315 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.316 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.316 DEBUG:boto:path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.317 DEBUG:boto:auth_path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.317 DEBUG:boto:Method: GET 2022-09-13T07:44:51.318 DEBUG:boto:Path: /yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.318 DEBUG:boto:Data: 2022-09-13T07:44:51.319 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.319 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:51.320 DEBUG:boto:Port: 8001 2022-09-13T07:44:51.320 DEBUG:boto:Params: {} 2022-09-13T07:44:51.321 DEBUG:boto:Token: None 2022-09-13T07:44:51.321 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/?versions 2022-09-13T07:44:51.322 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc= 2022-09-13T07:44:51.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc=', 'Content-Length': '0'} 2022-09-13T07:44:51.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000477887d7565f302-00632034f3-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.326 DEBUG:boto:b'yxvvuh-24999falseobjnulltrue2022-09-13T07:44:10.116Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:51.327 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.327 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.328 DEBUG:boto:path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.328 DEBUG:boto:auth_path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.329 DEBUG:boto:Method: GET 2022-09-13T07:44:51.329 DEBUG:boto:Path: /yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.330 DEBUG:boto:Data: 2022-09-13T07:44:51.331 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.331 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:51.332 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.332 DEBUG:boto:Params: {} 2022-09-13T07:44:51.333 DEBUG:boto:Token: None 2022-09-13T07:44:51.333 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/?versions 2022-09-13T07:44:51.334 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc= 2022-09-13T07:44:51.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc=', 'Content-Length': '0'} 2022-09-13T07:44:51.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009b2c138468bcdc8-00632034f3-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.338 DEBUG:boto:b'yxvvuh-24999falseobjnulltrue2022-09-13T07:44:10.116Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:51.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:44:51.339 DEBUG:boto:path=/yxvvuh-24/obj 2022-09-13T07:44:51.340 DEBUG:boto:auth_path=/yxvvuh-24/obj 2022-09-13T07:44:51.340 DEBUG:boto:Method: HEAD 2022-09-13T07:44:51.341 DEBUG:boto:Path: /yxvvuh-24/obj 2022-09-13T07:44:51.341 DEBUG:boto:Data: 2022-09-13T07:44:51.342 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.342 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:51.343 DEBUG:boto:Port: 8001 2022-09-13T07:44:51.343 DEBUG:boto:Params: {} 2022-09-13T07:44:51.344 DEBUG:boto:Token: None 2022-09-13T07:44:51.345 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/obj 2022-09-13T07:44:51.345 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:q4aDdnkBH5SITzCfLHhNA0RbQ7E= 2022-09-13T07:44:51.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:q4aDdnkBH5SITzCfLHhNA0RbQ7E=', 'Content-Length': '0'} 2022-09-13T07:44:51.347 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000003849a56f74f29dc3-00632034f3-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.348 DEBUG:boto:path=/yxvvuh-24/obj 2022-09-13T07:44:51.348 DEBUG:boto:auth_path=/yxvvuh-24/obj 2022-09-13T07:44:51.349 DEBUG:boto:Method: HEAD 2022-09-13T07:44:51.349 DEBUG:boto:Path: /yxvvuh-24/obj 2022-09-13T07:44:51.350 DEBUG:boto:Data: 2022-09-13T07:44:51.350 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.351 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:51.351 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.352 DEBUG:boto:Params: {} 2022-09-13T07:44:51.352 DEBUG:boto:Token: None 2022-09-13T07:44:51.353 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/obj 2022-09-13T07:44:51.354 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:q4aDdnkBH5SITzCfLHhNA0RbQ7E= 2022-09-13T07:44:51.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:q4aDdnkBH5SITzCfLHhNA0RbQ7E=', 'Content-Length': '0'} 2022-09-13T07:44:51.357 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000fd4ef69021030d3c-00632034f3-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.358 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-24 zones={test-zone3, test-zone1} 2022-09-13T07:44:51.358 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-24 zones={test-zone3, test-zone2} 2022-09-13T07:44:51.359 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.359 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.360 DEBUG:boto:Method: HEAD 2022-09-13T07:44:51.360 DEBUG:boto:Path: /yxvvuh-24/ 2022-09-13T07:44:51.361 DEBUG:boto:Data: 2022-09-13T07:44:51.361 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.362 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:51.362 DEBUG:boto:Port: 8001 2022-09-13T07:44:51.363 DEBUG:boto:Params: {} 2022-09-13T07:44:51.363 DEBUG:boto:Token: None 2022-09-13T07:44:51.364 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/ 2022-09-13T07:44:51.364 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TGKDRbSlfQNP1HsI3BD7VSvnEJY= 2022-09-13T07:44:51.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TGKDRbSlfQNP1HsI3BD7VSvnEJY=', 'Content-Length': '0'} 2022-09-13T07:44:51.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b25c1476ad23522-00632034f3-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.368 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.369 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.369 DEBUG:boto:Method: HEAD 2022-09-13T07:44:51.370 DEBUG:boto:Path: /yxvvuh-24/ 2022-09-13T07:44:51.371 DEBUG:boto:Data: 2022-09-13T07:44:51.371 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.372 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:51.373 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.374 DEBUG:boto:Params: {} 2022-09-13T07:44:51.375 DEBUG:boto:Token: None 2022-09-13T07:44:51.376 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/ 2022-09-13T07:44:51.377 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:TGKDRbSlfQNP1HsI3BD7VSvnEJY= 2022-09-13T07:44:51.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:TGKDRbSlfQNP1HsI3BD7VSvnEJY=', 'Content-Length': '0'} 2022-09-13T07:44:51.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bada75de4d79baab-00632034f3-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.381 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:44:51.382 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.382 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.383 DEBUG:boto:path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.383 DEBUG:boto:auth_path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.384 DEBUG:boto:Method: GET 2022-09-13T07:44:51.384 DEBUG:boto:Path: /yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.385 DEBUG:boto:Data: 2022-09-13T07:44:51.386 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.386 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:51.387 DEBUG:boto:Port: 8001 2022-09-13T07:44:51.387 DEBUG:boto:Params: {} 2022-09-13T07:44:51.388 DEBUG:boto:Token: None 2022-09-13T07:44:51.389 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/?versions 2022-09-13T07:44:51.389 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc= 2022-09-13T07:44:51.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc=', 'Content-Length': '0'} 2022-09-13T07:44:51.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001423953c87a76f4b-00632034f3-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.394 DEBUG:boto:b'yxvvuh-24999falseobjnulltrue2022-09-13T07:44:10.116Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:51.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:51.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:44:51.396 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.397 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.398 DEBUG:boto:path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.398 DEBUG:boto:auth_path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.399 DEBUG:boto:Method: GET 2022-09-13T07:44:51.400 DEBUG:boto:Path: /yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.401 DEBUG:boto:Data: 2022-09-13T07:44:51.402 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.403 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:51.404 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.405 DEBUG:boto:Params: {} 2022-09-13T07:44:51.406 DEBUG:boto:Token: None 2022-09-13T07:44:51.407 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/?versions 2022-09-13T07:44:51.408 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc= 2022-09-13T07:44:51.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc=', 'Content-Length': '0'} 2022-09-13T07:44:51.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf4665c2706f56d2-00632034f3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.413 DEBUG:boto:b'yxvvuh-24999falseobjnulltrue2022-09-13T07:44:10.116Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:51.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:51.416 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.417 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.417 DEBUG:boto:path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.418 DEBUG:boto:auth_path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.419 DEBUG:boto:Method: GET 2022-09-13T07:44:51.420 DEBUG:boto:Path: /yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.421 DEBUG:boto:Data: 2022-09-13T07:44:51.422 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.423 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:51.424 DEBUG:boto:Port: 8001 2022-09-13T07:44:51.425 DEBUG:boto:Params: {} 2022-09-13T07:44:51.426 DEBUG:boto:Token: None 2022-09-13T07:44:51.427 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/?versions 2022-09-13T07:44:51.428 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc= 2022-09-13T07:44:51.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc=', 'Content-Length': '0'} 2022-09-13T07:44:51.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015b06cc88caa635a-00632034f3-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.434 DEBUG:boto:b'yxvvuh-24999falseobjnulltrue2022-09-13T07:44:10.116Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:51.435 DEBUG:boto:path=/yxvvuh-24/ 2022-09-13T07:44:51.436 DEBUG:boto:auth_path=/yxvvuh-24/ 2022-09-13T07:44:51.437 DEBUG:boto:path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.438 DEBUG:boto:auth_path=/yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.439 DEBUG:boto:Method: GET 2022-09-13T07:44:51.440 DEBUG:boto:Path: /yxvvuh-24/?versions&max-keys=999 2022-09-13T07:44:51.441 DEBUG:boto:Data: 2022-09-13T07:44:51.442 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.442 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:51.443 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.444 DEBUG:boto:Params: {} 2022-09-13T07:44:51.445 DEBUG:boto:Token: None 2022-09-13T07:44:51.446 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/?versions 2022-09-13T07:44:51.447 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc= 2022-09-13T07:44:51.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Y79c+Hmiu1+EtKfKeST6ehgZHIc=', 'Content-Length': '0'} 2022-09-13T07:44:51.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d964f42600217476-00632034f3-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.452 DEBUG:boto:b'yxvvuh-24999falseobjnulltrue2022-09-13T07:44:10.116Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:51.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:44:51.455 DEBUG:boto:path=/yxvvuh-24/obj 2022-09-13T07:44:51.456 DEBUG:boto:auth_path=/yxvvuh-24/obj 2022-09-13T07:44:51.457 DEBUG:boto:Method: HEAD 2022-09-13T07:44:51.458 DEBUG:boto:Path: /yxvvuh-24/obj 2022-09-13T07:44:51.459 DEBUG:boto:Data: 2022-09-13T07:44:51.460 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.461 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:51.461 DEBUG:boto:Port: 8001 2022-09-13T07:44:51.462 DEBUG:boto:Params: {} 2022-09-13T07:44:51.464 DEBUG:boto:Token: None 2022-09-13T07:44:51.465 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/obj 2022-09-13T07:44:51.466 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:q4aDdnkBH5SITzCfLHhNA0RbQ7E= 2022-09-13T07:44:51.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:q4aDdnkBH5SITzCfLHhNA0RbQ7E=', 'Content-Length': '0'} 2022-09-13T07:44:51.469 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000005965817e489c8c5c-00632034f3-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.471 DEBUG:boto:path=/yxvvuh-24/obj 2022-09-13T07:44:51.472 DEBUG:boto:auth_path=/yxvvuh-24/obj 2022-09-13T07:44:51.473 DEBUG:boto:Method: HEAD 2022-09-13T07:44:51.473 DEBUG:boto:Path: /yxvvuh-24/obj 2022-09-13T07:44:51.474 DEBUG:boto:Data: 2022-09-13T07:44:51.475 DEBUG:boto:Headers: {} 2022-09-13T07:44:51.476 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:51.477 DEBUG:boto:Port: 8000 2022-09-13T07:44:51.478 DEBUG:boto:Params: {} 2022-09-13T07:44:51.479 DEBUG:boto:Token: None 2022-09-13T07:44:51.480 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:51 GMT /yxvvuh-24/obj 2022-09-13T07:44:51.481 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:q4aDdnkBH5SITzCfLHhNA0RbQ7E= 2022-09-13T07:44:51.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:q4aDdnkBH5SITzCfLHhNA0RbQ7E=', 'Content-Length': '0'} 2022-09-13T07:44:51.485 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000ccbb2015ef85ead9-00632034f3-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:51.486 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-24 zones={test-zone3, test-zone2} 2022-09-13T07:44:51.487 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-25 2022-09-13T07:44:51.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:51.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:51.490 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:51.549 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:51.549 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:51.591 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:51.586+0000 7f15ad2ff600 1 bucket sync caught up with source: 2022-09-13T07:44:51.592 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000001.18.12, , ] 2022-09-13T07:44:51.592 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000001.18.12, , ] 2022-09-13T07:44:51.593 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:51.586+0000 7f15ad2ff600 0 bucket checkpoint complete 2022-09-13T07:44:51.614 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-25 2022-09-13T07:44:51.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:51.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:51.616 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:51.680 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:51.680 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:51.729 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:51.726+0000 7fe493bc0600 1 waiting to reach incremental sync.. 2022-09-13T07:44:56.732 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:56.727+0000 7fe493bc0600 1 bucket sync caught up with source: 2022-09-13T07:44:56.732 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000001.18.12, , ] 2022-09-13T07:44:56.733 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000001.18.12, , ] 2022-09-13T07:44:56.733 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:56.727+0000 7fe493bc0600 0 bucket checkpoint complete 2022-09-13T07:44:56.749 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-25 2022-09-13T07:44:56.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:56.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:56.752 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:56.816 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:56.816 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:56.854 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:56.850+0000 7f9e5358b600 1 bucket sync caught up with source: 2022-09-13T07:44:56.855 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000003.252.6, , ] 2022-09-13T07:44:56.855 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000003.252.6, , ] 2022-09-13T07:44:56.855 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:56.850+0000 7f9e5358b600 0 bucket checkpoint complete 2022-09-13T07:44:56.868 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-25 2022-09-13T07:44:56.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:56.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:56.870 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:56.929 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:56.930 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:56.986 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:56.982+0000 7ffbeac37600 1 bucket sync caught up with source: 2022-09-13T07:44:56.987 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000003.252.6, , ] 2022-09-13T07:44:56.987 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000003.252.6, , ] 2022-09-13T07:44:56.988 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:56.982+0000 7ffbeac37600 0 bucket checkpoint complete 2022-09-13T07:44:57.003 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-25 2022-09-13T07:44:57.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:57.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:57.005 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:57.063 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:57.063 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:57.112 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:57.108+0000 7f3dc5855600 1 bucket sync caught up with source: 2022-09-13T07:44:57.112 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000001.95.12, , ] 2022-09-13T07:44:57.113 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000001.95.12, , ] 2022-09-13T07:44:57.113 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:57.108+0000 7f3dc5855600 0 bucket checkpoint complete 2022-09-13T07:44:57.126 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-25 2022-09-13T07:44:57.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:57.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:57.128 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:57.192 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:57.193 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:57.242 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:57.237+0000 7f2393959600 1 bucket sync caught up with source: 2022-09-13T07:44:57.242 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000001.95.12, , ] 2022-09-13T07:44:57.243 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000001.95.12, , ] 2022-09-13T07:44:57.243 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:57.237+0000 7f2393959600 0 bucket checkpoint complete 2022-09-13T07:44:57.259 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-25 zones={test-zone2, test-zone1} 2022-09-13T07:44:57.260 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.261 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.261 DEBUG:boto:Method: HEAD 2022-09-13T07:44:57.262 DEBUG:boto:Path: /yxvvuh-25/ 2022-09-13T07:44:57.263 DEBUG:boto:Data: 2022-09-13T07:44:57.264 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.265 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:57.266 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.267 DEBUG:boto:Params: {} 2022-09-13T07:44:57.268 DEBUG:boto:Token: None 2022-09-13T07:44:57.269 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/ 2022-09-13T07:44:57.270 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t5DOIUY3pYIaZxfl2J7mIPiYIgo= 2022-09-13T07:44:57.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t5DOIUY3pYIaZxfl2J7mIPiYIgo=', 'Content-Length': '0'} 2022-09-13T07:44:57.274 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c518e5183670db98-00632034f9-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.275 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.276 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.277 DEBUG:boto:Method: HEAD 2022-09-13T07:44:57.277 DEBUG:boto:Path: /yxvvuh-25/ 2022-09-13T07:44:57.278 DEBUG:boto:Data: 2022-09-13T07:44:57.279 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.280 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:57.281 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.282 DEBUG:boto:Params: {} 2022-09-13T07:44:57.283 DEBUG:boto:Token: None 2022-09-13T07:44:57.283 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/ 2022-09-13T07:44:57.284 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t5DOIUY3pYIaZxfl2J7mIPiYIgo= 2022-09-13T07:44:57.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t5DOIUY3pYIaZxfl2J7mIPiYIgo=', 'Content-Length': '0'} 2022-09-13T07:44:57.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038c0a781fd68af16-00632034f9-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.289 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:44:57.290 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.291 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.291 DEBUG:boto:path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.292 DEBUG:boto:auth_path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.293 DEBUG:boto:Method: GET 2022-09-13T07:44:57.294 DEBUG:boto:Path: /yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.295 DEBUG:boto:Data: 2022-09-13T07:44:57.296 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.296 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:57.297 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.298 DEBUG:boto:Params: {} 2022-09-13T07:44:57.299 DEBUG:boto:Token: None 2022-09-13T07:44:57.300 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/?versions 2022-09-13T07:44:57.301 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s= 2022-09-13T07:44:57.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s=', 'Content-Length': '0'} 2022-09-13T07:44:57.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d6ebf0c2dc6fef4-00632034f9-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.306 DEBUG:boto:b'yxvvuh-25999falseobjnulltrue2022-09-13T07:44:10.212Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:57.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:57.308 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:44:57.308 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.309 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.310 DEBUG:boto:path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.311 DEBUG:boto:auth_path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.312 DEBUG:boto:Method: GET 2022-09-13T07:44:57.312 DEBUG:boto:Path: /yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.313 DEBUG:boto:Data: 2022-09-13T07:44:57.314 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.315 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:57.316 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.317 DEBUG:boto:Params: {} 2022-09-13T07:44:57.317 DEBUG:boto:Token: None 2022-09-13T07:44:57.318 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/?versions 2022-09-13T07:44:57.319 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s= 2022-09-13T07:44:57.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s=', 'Content-Length': '0'} 2022-09-13T07:44:57.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076eb92a1ec230b0d-00632034f9-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.324 DEBUG:boto:b'yxvvuh-25999falseobjnulltrue2022-09-13T07:44:10.212Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:57.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:57.326 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.327 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.328 DEBUG:boto:path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.329 DEBUG:boto:auth_path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.330 DEBUG:boto:Method: GET 2022-09-13T07:44:57.330 DEBUG:boto:Path: /yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.331 DEBUG:boto:Data: 2022-09-13T07:44:57.332 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.333 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:57.334 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.335 DEBUG:boto:Params: {} 2022-09-13T07:44:57.336 DEBUG:boto:Token: None 2022-09-13T07:44:57.338 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/?versions 2022-09-13T07:44:57.339 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s= 2022-09-13T07:44:57.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s=', 'Content-Length': '0'} 2022-09-13T07:44:57.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a46034523e97cb7f-00632034f9-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.344 DEBUG:boto:b'yxvvuh-25999falseobjnulltrue2022-09-13T07:44:10.212Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:57.345 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.346 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.347 DEBUG:boto:path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.347 DEBUG:boto:auth_path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.348 DEBUG:boto:Method: GET 2022-09-13T07:44:57.348 DEBUG:boto:Path: /yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.349 DEBUG:boto:Data: 2022-09-13T07:44:57.349 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.350 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:57.350 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.351 DEBUG:boto:Params: {} 2022-09-13T07:44:57.351 DEBUG:boto:Token: None 2022-09-13T07:44:57.352 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/?versions 2022-09-13T07:44:57.353 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s= 2022-09-13T07:44:57.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s=', 'Content-Length': '0'} 2022-09-13T07:44:57.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f7a5e6eb7b48836-00632034f9-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.399 DEBUG:boto:b'yxvvuh-25999falseobjnulltrue2022-09-13T07:44:10.212Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:57.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:44:57.402 DEBUG:boto:path=/yxvvuh-25/obj 2022-09-13T07:44:57.403 DEBUG:boto:auth_path=/yxvvuh-25/obj 2022-09-13T07:44:57.404 DEBUG:boto:Method: HEAD 2022-09-13T07:44:57.404 DEBUG:boto:Path: /yxvvuh-25/obj 2022-09-13T07:44:57.405 DEBUG:boto:Data: 2022-09-13T07:44:57.405 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.406 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:57.407 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.407 DEBUG:boto:Params: {} 2022-09-13T07:44:57.408 DEBUG:boto:Token: None 2022-09-13T07:44:57.408 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/obj 2022-09-13T07:44:57.409 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:v1LhAG4mMITNaLImNxwFRNwanVQ= 2022-09-13T07:44:57.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:v1LhAG4mMITNaLImNxwFRNwanVQ=', 'Content-Length': '0'} 2022-09-13T07:44:57.412 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000005d0e5d2bbfd5d8c4-00632034f9-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.412 DEBUG:boto:path=/yxvvuh-25/obj 2022-09-13T07:44:57.413 DEBUG:boto:auth_path=/yxvvuh-25/obj 2022-09-13T07:44:57.414 DEBUG:boto:Method: HEAD 2022-09-13T07:44:57.414 DEBUG:boto:Path: /yxvvuh-25/obj 2022-09-13T07:44:57.415 DEBUG:boto:Data: 2022-09-13T07:44:57.415 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.416 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:57.416 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.417 DEBUG:boto:Params: {} 2022-09-13T07:44:57.417 DEBUG:boto:Token: None 2022-09-13T07:44:57.418 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/obj 2022-09-13T07:44:57.419 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:v1LhAG4mMITNaLImNxwFRNwanVQ= 2022-09-13T07:44:57.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:v1LhAG4mMITNaLImNxwFRNwanVQ=', 'Content-Length': '0'} 2022-09-13T07:44:57.421 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000004a52a90d4867d796-00632034f9-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.422 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-25 zones={test-zone2, test-zone1} 2022-09-13T07:44:57.423 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-25 zones={test-zone3, test-zone1} 2022-09-13T07:44:57.423 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.424 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.424 DEBUG:boto:Method: HEAD 2022-09-13T07:44:57.425 DEBUG:boto:Path: /yxvvuh-25/ 2022-09-13T07:44:57.425 DEBUG:boto:Data: 2022-09-13T07:44:57.426 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.426 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:57.427 DEBUG:boto:Port: 8001 2022-09-13T07:44:57.427 DEBUG:boto:Params: {} 2022-09-13T07:44:57.428 DEBUG:boto:Token: None 2022-09-13T07:44:57.428 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/ 2022-09-13T07:44:57.429 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t5DOIUY3pYIaZxfl2J7mIPiYIgo= 2022-09-13T07:44:57.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t5DOIUY3pYIaZxfl2J7mIPiYIgo=', 'Content-Length': '0'} 2022-09-13T07:44:57.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042f7e867b7976fb4-00632034f9-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.433 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.433 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.434 DEBUG:boto:Method: HEAD 2022-09-13T07:44:57.434 DEBUG:boto:Path: /yxvvuh-25/ 2022-09-13T07:44:57.435 DEBUG:boto:Data: 2022-09-13T07:44:57.435 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.436 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:57.437 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.437 DEBUG:boto:Params: {} 2022-09-13T07:44:57.438 DEBUG:boto:Token: None 2022-09-13T07:44:57.438 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/ 2022-09-13T07:44:57.439 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t5DOIUY3pYIaZxfl2J7mIPiYIgo= 2022-09-13T07:44:57.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t5DOIUY3pYIaZxfl2J7mIPiYIgo=', 'Content-Length': '0'} 2022-09-13T07:44:57.443 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c9956991ba3a1cc-00632034f9-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.444 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:44:57.444 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.445 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.446 DEBUG:boto:path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.446 DEBUG:boto:auth_path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.447 DEBUG:boto:Method: GET 2022-09-13T07:44:57.447 DEBUG:boto:Path: /yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.448 DEBUG:boto:Data: 2022-09-13T07:44:57.449 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.449 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:57.450 DEBUG:boto:Port: 8001 2022-09-13T07:44:57.451 DEBUG:boto:Params: {} 2022-09-13T07:44:57.451 DEBUG:boto:Token: None 2022-09-13T07:44:57.452 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/?versions 2022-09-13T07:44:57.452 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s= 2022-09-13T07:44:57.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s=', 'Content-Length': '0'} 2022-09-13T07:44:57.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b705235e88ac9ff-00632034f9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.456 DEBUG:boto:b'yxvvuh-25999falseobjnulltrue2022-09-13T07:44:10.212Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:57.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:57.458 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:44:57.458 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.459 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.459 DEBUG:boto:path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.460 DEBUG:boto:auth_path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.461 DEBUG:boto:Method: GET 2022-09-13T07:44:57.462 DEBUG:boto:Path: /yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.462 DEBUG:boto:Data: 2022-09-13T07:44:57.463 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.463 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:57.464 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.464 DEBUG:boto:Params: {} 2022-09-13T07:44:57.465 DEBUG:boto:Token: None 2022-09-13T07:44:57.465 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/?versions 2022-09-13T07:44:57.466 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s= 2022-09-13T07:44:57.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s=', 'Content-Length': '0'} 2022-09-13T07:44:57.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e12596d951125ecd-00632034f9-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.470 DEBUG:boto:b'yxvvuh-25999falseobjnulltrue2022-09-13T07:44:10.212Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:57.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:57.471 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.472 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.472 DEBUG:boto:path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.473 DEBUG:boto:auth_path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.473 DEBUG:boto:Method: GET 2022-09-13T07:44:57.474 DEBUG:boto:Path: /yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.474 DEBUG:boto:Data: 2022-09-13T07:44:57.475 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.475 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:57.476 DEBUG:boto:Port: 8001 2022-09-13T07:44:57.477 DEBUG:boto:Params: {} 2022-09-13T07:44:57.477 DEBUG:boto:Token: None 2022-09-13T07:44:57.478 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/?versions 2022-09-13T07:44:57.479 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s= 2022-09-13T07:44:57.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s=', 'Content-Length': '0'} 2022-09-13T07:44:57.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac7d522f7d4ace59-00632034f9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.483 DEBUG:boto:b'yxvvuh-25999falseobjnulltrue2022-09-13T07:44:10.212Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:57.483 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.484 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.484 DEBUG:boto:path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.485 DEBUG:boto:auth_path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.485 DEBUG:boto:Method: GET 2022-09-13T07:44:57.486 DEBUG:boto:Path: /yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.487 DEBUG:boto:Data: 2022-09-13T07:44:57.487 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.488 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:57.488 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.489 DEBUG:boto:Params: {} 2022-09-13T07:44:57.489 DEBUG:boto:Token: None 2022-09-13T07:44:57.490 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/?versions 2022-09-13T07:44:57.491 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s= 2022-09-13T07:44:57.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s=', 'Content-Length': '0'} 2022-09-13T07:44:57.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000701d78cafd5686ec-00632034f9-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.496 DEBUG:boto:b'yxvvuh-25999falseobjnulltrue2022-09-13T07:44:10.212Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:57.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:44:57.498 DEBUG:boto:path=/yxvvuh-25/obj 2022-09-13T07:44:57.498 DEBUG:boto:auth_path=/yxvvuh-25/obj 2022-09-13T07:44:57.499 DEBUG:boto:Method: HEAD 2022-09-13T07:44:57.500 DEBUG:boto:Path: /yxvvuh-25/obj 2022-09-13T07:44:57.500 DEBUG:boto:Data: 2022-09-13T07:44:57.501 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.502 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:57.503 DEBUG:boto:Port: 8001 2022-09-13T07:44:57.504 DEBUG:boto:Params: {} 2022-09-13T07:44:57.505 DEBUG:boto:Token: None 2022-09-13T07:44:57.506 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/obj 2022-09-13T07:44:57.507 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:v1LhAG4mMITNaLImNxwFRNwanVQ= 2022-09-13T07:44:57.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:v1LhAG4mMITNaLImNxwFRNwanVQ=', 'Content-Length': '0'} 2022-09-13T07:44:57.510 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000003979ec5a81315fe9-00632034f9-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.511 DEBUG:boto:path=/yxvvuh-25/obj 2022-09-13T07:44:57.512 DEBUG:boto:auth_path=/yxvvuh-25/obj 2022-09-13T07:44:57.513 DEBUG:boto:Method: HEAD 2022-09-13T07:44:57.514 DEBUG:boto:Path: /yxvvuh-25/obj 2022-09-13T07:44:57.515 DEBUG:boto:Data: 2022-09-13T07:44:57.516 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.516 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:57.517 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.518 DEBUG:boto:Params: {} 2022-09-13T07:44:57.519 DEBUG:boto:Token: None 2022-09-13T07:44:57.520 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/obj 2022-09-13T07:44:57.521 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:v1LhAG4mMITNaLImNxwFRNwanVQ= 2022-09-13T07:44:57.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:v1LhAG4mMITNaLImNxwFRNwanVQ=', 'Content-Length': '0'} 2022-09-13T07:44:57.524 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000f814ac6c4a392d77-00632034f9-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.525 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-25 zones={test-zone3, test-zone1} 2022-09-13T07:44:57.525 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-25 zones={test-zone3, test-zone2} 2022-09-13T07:44:57.526 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.526 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.527 DEBUG:boto:Method: HEAD 2022-09-13T07:44:57.527 DEBUG:boto:Path: /yxvvuh-25/ 2022-09-13T07:44:57.528 DEBUG:boto:Data: 2022-09-13T07:44:57.528 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.529 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:57.529 DEBUG:boto:Port: 8001 2022-09-13T07:44:57.530 DEBUG:boto:Params: {} 2022-09-13T07:44:57.530 DEBUG:boto:Token: None 2022-09-13T07:44:57.531 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/ 2022-09-13T07:44:57.532 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t5DOIUY3pYIaZxfl2J7mIPiYIgo= 2022-09-13T07:44:57.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t5DOIUY3pYIaZxfl2J7mIPiYIgo=', 'Content-Length': '0'} 2022-09-13T07:44:57.535 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a63e84560bccfa92-00632034f9-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.536 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.536 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.537 DEBUG:boto:Method: HEAD 2022-09-13T07:44:57.537 DEBUG:boto:Path: /yxvvuh-25/ 2022-09-13T07:44:57.538 DEBUG:boto:Data: 2022-09-13T07:44:57.538 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.539 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:57.539 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.540 DEBUG:boto:Params: {} 2022-09-13T07:44:57.541 DEBUG:boto:Token: None 2022-09-13T07:44:57.542 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/ 2022-09-13T07:44:57.542 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:t5DOIUY3pYIaZxfl2J7mIPiYIgo= 2022-09-13T07:44:57.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:t5DOIUY3pYIaZxfl2J7mIPiYIgo=', 'Content-Length': '0'} 2022-09-13T07:44:57.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e819ee905c9665d5-00632034f9-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.546 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:44:57.547 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.548 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.549 DEBUG:boto:path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.549 DEBUG:boto:auth_path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.550 DEBUG:boto:Method: GET 2022-09-13T07:44:57.550 DEBUG:boto:Path: /yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.551 DEBUG:boto:Data: 2022-09-13T07:44:57.551 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.552 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:57.552 DEBUG:boto:Port: 8001 2022-09-13T07:44:57.553 DEBUG:boto:Params: {} 2022-09-13T07:44:57.553 DEBUG:boto:Token: None 2022-09-13T07:44:57.554 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/?versions 2022-09-13T07:44:57.555 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s= 2022-09-13T07:44:57.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s=', 'Content-Length': '0'} 2022-09-13T07:44:57.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f51b89eff18f7ced-00632034f9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.598 DEBUG:boto:b'yxvvuh-25999falseobjnulltrue2022-09-13T07:44:10.212Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:57.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:57.600 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:44:57.600 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.601 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.601 DEBUG:boto:path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.602 DEBUG:boto:auth_path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.602 DEBUG:boto:Method: GET 2022-09-13T07:44:57.603 DEBUG:boto:Path: /yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.604 DEBUG:boto:Data: 2022-09-13T07:44:57.604 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.605 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:57.606 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.607 DEBUG:boto:Params: {} 2022-09-13T07:44:57.608 DEBUG:boto:Token: None 2022-09-13T07:44:57.609 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/?versions 2022-09-13T07:44:57.610 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s= 2022-09-13T07:44:57.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s=', 'Content-Length': '0'} 2022-09-13T07:44:57.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ce7a934e68e6e36-00632034f9-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.615 DEBUG:boto:b'yxvvuh-25999falseobjnulltrue2022-09-13T07:44:10.212Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:57.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:57.617 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.618 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.618 DEBUG:boto:path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.619 DEBUG:boto:auth_path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.619 DEBUG:boto:Method: GET 2022-09-13T07:44:57.620 DEBUG:boto:Path: /yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.620 DEBUG:boto:Data: 2022-09-13T07:44:57.621 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.622 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:57.622 DEBUG:boto:Port: 8001 2022-09-13T07:44:57.623 DEBUG:boto:Params: {} 2022-09-13T07:44:57.624 DEBUG:boto:Token: None 2022-09-13T07:44:57.625 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/?versions 2022-09-13T07:44:57.626 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s= 2022-09-13T07:44:57.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s=', 'Content-Length': '0'} 2022-09-13T07:44:57.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dab08fe210adae22-00632034f9-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.631 DEBUG:boto:b'yxvvuh-25999falseobjnulltrue2022-09-13T07:44:10.212Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:57.633 DEBUG:boto:path=/yxvvuh-25/ 2022-09-13T07:44:57.634 DEBUG:boto:auth_path=/yxvvuh-25/ 2022-09-13T07:44:57.634 DEBUG:boto:path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.635 DEBUG:boto:auth_path=/yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.636 DEBUG:boto:Method: GET 2022-09-13T07:44:57.637 DEBUG:boto:Path: /yxvvuh-25/?versions&max-keys=999 2022-09-13T07:44:57.637 DEBUG:boto:Data: 2022-09-13T07:44:57.638 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.638 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:57.639 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.640 DEBUG:boto:Params: {} 2022-09-13T07:44:57.641 DEBUG:boto:Token: None 2022-09-13T07:44:57.642 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/?versions 2022-09-13T07:44:57.643 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s= 2022-09-13T07:44:57.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:52nlYT2vHYKFHE6lGt65AMONv/s=', 'Content-Length': '0'} 2022-09-13T07:44:57.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008479ac758e6e4de4-00632034f9-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.697 DEBUG:boto:b'yxvvuh-25999falseobjnulltrue2022-09-13T07:44:10.212Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:57.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:44:57.699 DEBUG:boto:path=/yxvvuh-25/obj 2022-09-13T07:44:57.700 DEBUG:boto:auth_path=/yxvvuh-25/obj 2022-09-13T07:44:57.700 DEBUG:boto:Method: HEAD 2022-09-13T07:44:57.701 DEBUG:boto:Path: /yxvvuh-25/obj 2022-09-13T07:44:57.701 DEBUG:boto:Data: 2022-09-13T07:44:57.702 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.702 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:57.703 DEBUG:boto:Port: 8001 2022-09-13T07:44:57.703 DEBUG:boto:Params: {} 2022-09-13T07:44:57.704 DEBUG:boto:Token: None 2022-09-13T07:44:57.705 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/obj 2022-09-13T07:44:57.705 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:v1LhAG4mMITNaLImNxwFRNwanVQ= 2022-09-13T07:44:57.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:v1LhAG4mMITNaLImNxwFRNwanVQ=', 'Content-Length': '0'} 2022-09-13T07:44:57.708 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000b3416ad44a9663fc-00632034f9-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.709 DEBUG:boto:path=/yxvvuh-25/obj 2022-09-13T07:44:57.709 DEBUG:boto:auth_path=/yxvvuh-25/obj 2022-09-13T07:44:57.710 DEBUG:boto:Method: HEAD 2022-09-13T07:44:57.710 DEBUG:boto:Path: /yxvvuh-25/obj 2022-09-13T07:44:57.711 DEBUG:boto:Data: 2022-09-13T07:44:57.711 DEBUG:boto:Headers: {} 2022-09-13T07:44:57.712 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:57.712 DEBUG:boto:Port: 8000 2022-09-13T07:44:57.713 DEBUG:boto:Params: {} 2022-09-13T07:44:57.713 DEBUG:boto:Token: None 2022-09-13T07:44:57.714 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:57 GMT /yxvvuh-25/obj 2022-09-13T07:44:57.715 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:v1LhAG4mMITNaLImNxwFRNwanVQ= 2022-09-13T07:44:57.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:57 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:v1LhAG4mMITNaLImNxwFRNwanVQ=', 'Content-Length': '0'} 2022-09-13T07:44:57.717 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000009642c4c419ff9253-00632034f9-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:57.718 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-25 zones={test-zone3, test-zone2} 2022-09-13T07:44:57.719 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-26 2022-09-13T07:44:57.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:57.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:57.722 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:57.781 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:57.781 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:57.816 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:57.812+0000 7f72ee812600 1 bucket sync caught up with source: 2022-09-13T07:44:57.817 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000001.50.12, , ] 2022-09-13T07:44:57.817 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000001.50.12, , ] 2022-09-13T07:44:57.818 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:57.812+0000 7f72ee812600 0 bucket checkpoint complete 2022-09-13T07:44:57.830 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-26 2022-09-13T07:44:57.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:57.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:57.833 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:57.893 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:57.894 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:57.930 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:57.926+0000 7f100f52a600 1 bucket sync caught up with source: 2022-09-13T07:44:57.930 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000001.50.12, , ] 2022-09-13T07:44:57.931 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000001.50.12, , ] 2022-09-13T07:44:57.931 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:57.926+0000 7f100f52a600 0 bucket checkpoint complete 2022-09-13T07:44:57.943 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-26 2022-09-13T07:44:57.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:57.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:57.945 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:58.003 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:58.003 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:58.039 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:58.035+0000 7fcc11577600 1 bucket sync caught up with source: 2022-09-13T07:44:58.040 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000001.62.12, , ] 2022-09-13T07:44:58.040 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000001.62.12, , ] 2022-09-13T07:44:58.040 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:58.035+0000 7fcc11577600 0 bucket checkpoint complete 2022-09-13T07:44:58.061 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-26 2022-09-13T07:44:58.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:58.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:58.062 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:58.120 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:58.120 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:58.165 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:58.161+0000 7f8f610e2600 1 bucket sync caught up with source: 2022-09-13T07:44:58.165 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000001.62.12, , ] 2022-09-13T07:44:58.165 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000001.62.12, , ] 2022-09-13T07:44:58.166 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:58.161+0000 7f8f610e2600 0 bucket checkpoint complete 2022-09-13T07:44:58.186 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-26 2022-09-13T07:44:58.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:58.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:58.188 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:58.247 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:58.247 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:58.291 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:58.287+0000 7fb138150600 1 bucket sync caught up with source: 2022-09-13T07:44:58.292 INFO:teuthology.orchestra.run.smithi036.stderr: local status: [, , , , , , , , 00000000003.224.6, , ] 2022-09-13T07:44:58.293 INFO:teuthology.orchestra.run.smithi036.stderr: remote markers: [, , , , , , , , 00000000003.224.6, , ] 2022-09-13T07:44:58.293 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T07:44:58.287+0000 7fb138150600 0 bucket checkpoint complete 2022-09-13T07:44:58.306 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-26 2022-09-13T07:44:58.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:58.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T07:44:58.309 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T07:44:58.369 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:44:58.369 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:44:58.426 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:58.421+0000 7fc06f5af600 1 bucket sync caught up with source: 2022-09-13T07:44:58.426 INFO:teuthology.orchestra.run.smithi083.stderr: local status: [, , , , , , , , 00000000003.224.6, , ] 2022-09-13T07:44:58.427 INFO:teuthology.orchestra.run.smithi083.stderr: remote markers: [, , , , , , , , 00000000003.224.6, , ] 2022-09-13T07:44:58.427 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:44:58.421+0000 7fc06f5af600 0 bucket checkpoint complete 2022-09-13T07:44:58.449 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-26 zones={test-zone2, test-zone1} 2022-09-13T07:44:58.451 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.451 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.452 DEBUG:boto:Method: HEAD 2022-09-13T07:44:58.453 DEBUG:boto:Path: /yxvvuh-26/ 2022-09-13T07:44:58.455 DEBUG:boto:Data: 2022-09-13T07:44:58.456 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.457 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:58.458 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.459 DEBUG:boto:Params: {} 2022-09-13T07:44:58.460 DEBUG:boto:Token: None 2022-09-13T07:44:58.461 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/ 2022-09-13T07:44:58.462 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dEpphfLQMsjU7fZUc1/x8BYgKVM= 2022-09-13T07:44:58.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dEpphfLQMsjU7fZUc1/x8BYgKVM=', 'Content-Length': '0'} 2022-09-13T07:44:58.466 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad146d887bf741ae-00632034fa-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.467 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.467 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.468 DEBUG:boto:Method: HEAD 2022-09-13T07:44:58.468 DEBUG:boto:Path: /yxvvuh-26/ 2022-09-13T07:44:58.469 DEBUG:boto:Data: 2022-09-13T07:44:58.470 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.471 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:58.472 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.473 DEBUG:boto:Params: {} 2022-09-13T07:44:58.474 DEBUG:boto:Token: None 2022-09-13T07:44:58.474 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/ 2022-09-13T07:44:58.475 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dEpphfLQMsjU7fZUc1/x8BYgKVM= 2022-09-13T07:44:58.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dEpphfLQMsjU7fZUc1/x8BYgKVM=', 'Content-Length': '0'} 2022-09-13T07:44:58.478 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb5365d6733b948d-00632034fa-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.479 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:44:58.479 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.480 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.480 DEBUG:boto:path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.481 DEBUG:boto:auth_path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.481 DEBUG:boto:Method: GET 2022-09-13T07:44:58.482 DEBUG:boto:Path: /yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.482 DEBUG:boto:Data: 2022-09-13T07:44:58.483 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.483 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:58.484 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.484 DEBUG:boto:Params: {} 2022-09-13T07:44:58.485 DEBUG:boto:Token: None 2022-09-13T07:44:58.485 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/?versions 2022-09-13T07:44:58.486 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk= 2022-09-13T07:44:58.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk=', 'Content-Length': '0'} 2022-09-13T07:44:58.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a451fe7174f1c6d-00632034fa-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.490 DEBUG:boto:b'yxvvuh-26999falseobjnulltrue2022-09-13T07:44:10.292Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:58.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:58.491 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:44:58.492 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.493 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.493 DEBUG:boto:path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.494 DEBUG:boto:auth_path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.494 DEBUG:boto:Method: GET 2022-09-13T07:44:58.495 DEBUG:boto:Path: /yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.495 DEBUG:boto:Data: 2022-09-13T07:44:58.496 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.496 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:58.497 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.497 DEBUG:boto:Params: {} 2022-09-13T07:44:58.498 DEBUG:boto:Token: None 2022-09-13T07:44:58.499 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/?versions 2022-09-13T07:44:58.499 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk= 2022-09-13T07:44:58.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk=', 'Content-Length': '0'} 2022-09-13T07:44:58.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000213ab9a027f72da1-00632034fa-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.503 DEBUG:boto:b'yxvvuh-26999falseobjnulltrue2022-09-13T07:44:10.292Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:58.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:58.504 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.505 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.505 DEBUG:boto:path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.506 DEBUG:boto:auth_path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.506 DEBUG:boto:Method: GET 2022-09-13T07:44:58.507 DEBUG:boto:Path: /yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.508 DEBUG:boto:Data: 2022-09-13T07:44:58.508 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.509 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:58.509 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.510 DEBUG:boto:Params: {} 2022-09-13T07:44:58.510 DEBUG:boto:Token: None 2022-09-13T07:44:58.511 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/?versions 2022-09-13T07:44:58.511 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk= 2022-09-13T07:44:58.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk=', 'Content-Length': '0'} 2022-09-13T07:44:58.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018e194452014c7c3-00632034fa-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.515 DEBUG:boto:b'yxvvuh-26999falseobjnulltrue2022-09-13T07:44:10.292Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:58.516 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.517 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.517 DEBUG:boto:path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.518 DEBUG:boto:auth_path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.518 DEBUG:boto:Method: GET 2022-09-13T07:44:58.519 DEBUG:boto:Path: /yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.519 DEBUG:boto:Data: 2022-09-13T07:44:58.520 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.520 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:58.521 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.521 DEBUG:boto:Params: {} 2022-09-13T07:44:58.522 DEBUG:boto:Token: None 2022-09-13T07:44:58.522 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/?versions 2022-09-13T07:44:58.523 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk= 2022-09-13T07:44:58.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk=', 'Content-Length': '0'} 2022-09-13T07:44:58.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef7aebec2045bdff-00632034fa-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.527 DEBUG:boto:b'yxvvuh-26999falseobjnulltrue2022-09-13T07:44:10.292Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:58.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:44:58.528 DEBUG:boto:path=/yxvvuh-26/obj 2022-09-13T07:44:58.528 DEBUG:boto:auth_path=/yxvvuh-26/obj 2022-09-13T07:44:58.529 DEBUG:boto:Method: HEAD 2022-09-13T07:44:58.529 DEBUG:boto:Path: /yxvvuh-26/obj 2022-09-13T07:44:58.530 DEBUG:boto:Data: 2022-09-13T07:44:58.530 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.531 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:58.532 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.532 DEBUG:boto:Params: {} 2022-09-13T07:44:58.533 DEBUG:boto:Token: None 2022-09-13T07:44:58.533 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/obj 2022-09-13T07:44:58.534 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rd1MplOT6neK41eX5ksLNcZYMvk= 2022-09-13T07:44:58.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rd1MplOT6neK41eX5ksLNcZYMvk=', 'Content-Length': '0'} 2022-09-13T07:44:58.536 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000007b111d43d4ee0822-00632034fa-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.537 DEBUG:boto:path=/yxvvuh-26/obj 2022-09-13T07:44:58.537 DEBUG:boto:auth_path=/yxvvuh-26/obj 2022-09-13T07:44:58.538 DEBUG:boto:Method: HEAD 2022-09-13T07:44:58.538 DEBUG:boto:Path: /yxvvuh-26/obj 2022-09-13T07:44:58.539 DEBUG:boto:Data: 2022-09-13T07:44:58.539 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.540 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:58.540 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.541 DEBUG:boto:Params: {} 2022-09-13T07:44:58.541 DEBUG:boto:Token: None 2022-09-13T07:44:58.542 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/obj 2022-09-13T07:44:58.542 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rd1MplOT6neK41eX5ksLNcZYMvk= 2022-09-13T07:44:58.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rd1MplOT6neK41eX5ksLNcZYMvk=', 'Content-Length': '0'} 2022-09-13T07:44:58.545 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000397c6a3f4855f31e-00632034fa-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.545 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-26 zones={test-zone2, test-zone1} 2022-09-13T07:44:58.546 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-26 zones={test-zone3, test-zone1} 2022-09-13T07:44:58.546 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.547 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.547 DEBUG:boto:Method: HEAD 2022-09-13T07:44:58.548 DEBUG:boto:Path: /yxvvuh-26/ 2022-09-13T07:44:58.548 DEBUG:boto:Data: 2022-09-13T07:44:58.549 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.549 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:58.550 DEBUG:boto:Port: 8001 2022-09-13T07:44:58.550 DEBUG:boto:Params: {} 2022-09-13T07:44:58.551 DEBUG:boto:Token: None 2022-09-13T07:44:58.551 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/ 2022-09-13T07:44:58.552 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dEpphfLQMsjU7fZUc1/x8BYgKVM= 2022-09-13T07:44:58.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dEpphfLQMsjU7fZUc1/x8BYgKVM=', 'Content-Length': '0'} 2022-09-13T07:44:58.555 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045155e04e74ac841-00632034fa-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.555 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.556 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.556 DEBUG:boto:Method: HEAD 2022-09-13T07:44:58.557 DEBUG:boto:Path: /yxvvuh-26/ 2022-09-13T07:44:58.558 DEBUG:boto:Data: 2022-09-13T07:44:58.558 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.559 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:58.559 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.560 DEBUG:boto:Params: {} 2022-09-13T07:44:58.561 DEBUG:boto:Token: None 2022-09-13T07:44:58.562 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/ 2022-09-13T07:44:58.563 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dEpphfLQMsjU7fZUc1/x8BYgKVM= 2022-09-13T07:44:58.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dEpphfLQMsjU7fZUc1/x8BYgKVM=', 'Content-Length': '0'} 2022-09-13T07:44:58.567 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a155731d5c9d4cc-00632034fa-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.568 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:44:58.569 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.570 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.570 DEBUG:boto:path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.571 DEBUG:boto:auth_path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.572 DEBUG:boto:Method: GET 2022-09-13T07:44:58.572 DEBUG:boto:Path: /yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.573 DEBUG:boto:Data: 2022-09-13T07:44:58.573 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.574 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:58.574 DEBUG:boto:Port: 8001 2022-09-13T07:44:58.575 DEBUG:boto:Params: {} 2022-09-13T07:44:58.575 DEBUG:boto:Token: None 2022-09-13T07:44:58.576 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/?versions 2022-09-13T07:44:58.577 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk= 2022-09-13T07:44:58.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk=', 'Content-Length': '0'} 2022-09-13T07:44:58.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096388298d72c70db-00632034fa-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.582 DEBUG:boto:b'yxvvuh-26999falseobjnulltrue2022-09-13T07:44:10.292Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:58.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:58.584 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:44:58.585 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.586 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.586 DEBUG:boto:path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.587 DEBUG:boto:auth_path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.588 DEBUG:boto:Method: GET 2022-09-13T07:44:58.589 DEBUG:boto:Path: /yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.589 DEBUG:boto:Data: 2022-09-13T07:44:58.590 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.590 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:58.591 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.591 DEBUG:boto:Params: {} 2022-09-13T07:44:58.592 DEBUG:boto:Token: None 2022-09-13T07:44:58.592 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/?versions 2022-09-13T07:44:58.593 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk= 2022-09-13T07:44:58.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk=', 'Content-Length': '0'} 2022-09-13T07:44:58.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079f4d37a08fe549a-00632034fa-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.641 DEBUG:boto:b'yxvvuh-26999falseobjnulltrue2022-09-13T07:44:10.292Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:58.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:58.642 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.643 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.643 DEBUG:boto:path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.644 DEBUG:boto:auth_path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.644 DEBUG:boto:Method: GET 2022-09-13T07:44:58.645 DEBUG:boto:Path: /yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.645 DEBUG:boto:Data: 2022-09-13T07:44:58.646 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.646 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:58.647 DEBUG:boto:Port: 8001 2022-09-13T07:44:58.648 DEBUG:boto:Params: {} 2022-09-13T07:44:58.648 DEBUG:boto:Token: None 2022-09-13T07:44:58.649 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/?versions 2022-09-13T07:44:58.649 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk= 2022-09-13T07:44:58.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk=', 'Content-Length': '0'} 2022-09-13T07:44:58.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cd2da0e5a8a2b8f-00632034fa-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.653 DEBUG:boto:b'yxvvuh-26999falseobjnulltrue2022-09-13T07:44:10.292Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:58.654 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.654 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.655 DEBUG:boto:path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.655 DEBUG:boto:auth_path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.656 DEBUG:boto:Method: GET 2022-09-13T07:44:58.656 DEBUG:boto:Path: /yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.657 DEBUG:boto:Data: 2022-09-13T07:44:58.657 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.658 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:58.658 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.659 DEBUG:boto:Params: {} 2022-09-13T07:44:58.659 DEBUG:boto:Token: None 2022-09-13T07:44:58.660 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/?versions 2022-09-13T07:44:58.660 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk= 2022-09-13T07:44:58.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk=', 'Content-Length': '0'} 2022-09-13T07:44:58.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000720bb8c8f6f40253-00632034fa-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.664 DEBUG:boto:b'yxvvuh-26999falseobjnulltrue2022-09-13T07:44:10.292Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:58.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:44:58.665 DEBUG:boto:path=/yxvvuh-26/obj 2022-09-13T07:44:58.666 DEBUG:boto:auth_path=/yxvvuh-26/obj 2022-09-13T07:44:58.666 DEBUG:boto:Method: HEAD 2022-09-13T07:44:58.667 DEBUG:boto:Path: /yxvvuh-26/obj 2022-09-13T07:44:58.667 DEBUG:boto:Data: 2022-09-13T07:44:58.668 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.668 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:58.669 DEBUG:boto:Port: 8001 2022-09-13T07:44:58.669 DEBUG:boto:Params: {} 2022-09-13T07:44:58.670 DEBUG:boto:Token: None 2022-09-13T07:44:58.670 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/obj 2022-09-13T07:44:58.671 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rd1MplOT6neK41eX5ksLNcZYMvk= 2022-09-13T07:44:58.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rd1MplOT6neK41eX5ksLNcZYMvk=', 'Content-Length': '0'} 2022-09-13T07:44:58.673 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000003a2042284f2e9820-00632034fa-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.674 DEBUG:boto:path=/yxvvuh-26/obj 2022-09-13T07:44:58.674 DEBUG:boto:auth_path=/yxvvuh-26/obj 2022-09-13T07:44:58.675 DEBUG:boto:Method: HEAD 2022-09-13T07:44:58.675 DEBUG:boto:Path: /yxvvuh-26/obj 2022-09-13T07:44:58.676 DEBUG:boto:Data: 2022-09-13T07:44:58.676 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.677 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:58.677 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.678 DEBUG:boto:Params: {} 2022-09-13T07:44:58.678 DEBUG:boto:Token: None 2022-09-13T07:44:58.679 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/obj 2022-09-13T07:44:58.679 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rd1MplOT6neK41eX5ksLNcZYMvk= 2022-09-13T07:44:58.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rd1MplOT6neK41eX5ksLNcZYMvk=', 'Content-Length': '0'} 2022-09-13T07:44:58.682 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000006f8ed8a9582b182b-00632034fa-10ed-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.682 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-26 zones={test-zone3, test-zone1} 2022-09-13T07:44:58.683 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yxvvuh-26 zones={test-zone3, test-zone2} 2022-09-13T07:44:58.683 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.684 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.684 DEBUG:boto:Method: HEAD 2022-09-13T07:44:58.685 DEBUG:boto:Path: /yxvvuh-26/ 2022-09-13T07:44:58.685 DEBUG:boto:Data: 2022-09-13T07:44:58.686 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.686 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:58.687 DEBUG:boto:Port: 8001 2022-09-13T07:44:58.687 DEBUG:boto:Params: {} 2022-09-13T07:44:58.688 DEBUG:boto:Token: None 2022-09-13T07:44:58.688 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/ 2022-09-13T07:44:58.689 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dEpphfLQMsjU7fZUc1/x8BYgKVM= 2022-09-13T07:44:58.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dEpphfLQMsjU7fZUc1/x8BYgKVM=', 'Content-Length': '0'} 2022-09-13T07:44:58.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd920bf0885654a8-00632034fa-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.692 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.692 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.693 DEBUG:boto:Method: HEAD 2022-09-13T07:44:58.693 DEBUG:boto:Path: /yxvvuh-26/ 2022-09-13T07:44:58.694 DEBUG:boto:Data: 2022-09-13T07:44:58.694 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.695 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:58.695 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.696 DEBUG:boto:Params: {} 2022-09-13T07:44:58.696 DEBUG:boto:Token: None 2022-09-13T07:44:58.697 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/ 2022-09-13T07:44:58.697 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dEpphfLQMsjU7fZUc1/x8BYgKVM= 2022-09-13T07:44:58.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dEpphfLQMsjU7fZUc1/x8BYgKVM=', 'Content-Length': '0'} 2022-09-13T07:44:58.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041f6d8a26236cf15-00632034fa-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.701 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-13T07:44:58.701 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.702 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.702 DEBUG:boto:path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.703 DEBUG:boto:auth_path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.703 DEBUG:boto:Method: GET 2022-09-13T07:44:58.704 DEBUG:boto:Path: /yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.704 DEBUG:boto:Data: 2022-09-13T07:44:58.705 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.705 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:58.706 DEBUG:boto:Port: 8001 2022-09-13T07:44:58.706 DEBUG:boto:Params: {} 2022-09-13T07:44:58.707 DEBUG:boto:Token: None 2022-09-13T07:44:58.707 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/?versions 2022-09-13T07:44:58.708 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk= 2022-09-13T07:44:58.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk=', 'Content-Length': '0'} 2022-09-13T07:44:58.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028870dedd32b3c04-00632034fa-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.711 DEBUG:boto:b'yxvvuh-26999falseobjnulltrue2022-09-13T07:44:10.292Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:58.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:58.713 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-13T07:44:58.713 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.714 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.714 DEBUG:boto:path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.715 DEBUG:boto:auth_path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.715 DEBUG:boto:Method: GET 2022-09-13T07:44:58.716 DEBUG:boto:Path: /yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.716 DEBUG:boto:Data: 2022-09-13T07:44:58.717 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.717 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:58.718 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.718 DEBUG:boto:Params: {} 2022-09-13T07:44:58.719 DEBUG:boto:Token: None 2022-09-13T07:44:58.719 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/?versions 2022-09-13T07:44:58.720 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk= 2022-09-13T07:44:58.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk=', 'Content-Length': '0'} 2022-09-13T07:44:58.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024eb6815a24fbfe1-00632034fa-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.724 DEBUG:boto:b'yxvvuh-26999falseobjnulltrue2022-09-13T07:44:10.292Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:58.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-13T07:44:58.725 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.725 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.726 DEBUG:boto:path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.726 DEBUG:boto:auth_path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.727 DEBUG:boto:Method: GET 2022-09-13T07:44:58.727 DEBUG:boto:Path: /yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.728 DEBUG:boto:Data: 2022-09-13T07:44:58.729 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.730 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:58.730 DEBUG:boto:Port: 8001 2022-09-13T07:44:58.731 DEBUG:boto:Params: {} 2022-09-13T07:44:58.731 DEBUG:boto:Token: None 2022-09-13T07:44:58.732 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/?versions 2022-09-13T07:44:58.732 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk= 2022-09-13T07:44:58.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk=', 'Content-Length': '0'} 2022-09-13T07:44:58.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d112744ae0db659f-00632034fa-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.782 DEBUG:boto:b'yxvvuh-26999falseobjnulltrue2022-09-13T07:44:10.292Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:58.783 DEBUG:boto:path=/yxvvuh-26/ 2022-09-13T07:44:58.784 DEBUG:boto:auth_path=/yxvvuh-26/ 2022-09-13T07:44:58.784 DEBUG:boto:path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.785 DEBUG:boto:auth_path=/yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.785 DEBUG:boto:Method: GET 2022-09-13T07:44:58.786 DEBUG:boto:Path: /yxvvuh-26/?versions&max-keys=999 2022-09-13T07:44:58.786 DEBUG:boto:Data: 2022-09-13T07:44:58.787 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.787 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:58.788 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.788 DEBUG:boto:Params: {} 2022-09-13T07:44:58.789 DEBUG:boto:Token: None 2022-09-13T07:44:58.789 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/?versions 2022-09-13T07:44:58.790 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk= 2022-09-13T07:44:58.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FLXgDjLoH/yYXZdtouWxFzk4zqk=', 'Content-Length': '0'} 2022-09-13T07:44:58.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000990bb2e9b9f1de2b-00632034fa-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.794 DEBUG:boto:b'yxvvuh-26999falseobjnulltrue2022-09-13T07:44:10.292Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T07:44:58.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-13T07:44:58.795 DEBUG:boto:path=/yxvvuh-26/obj 2022-09-13T07:44:58.795 DEBUG:boto:auth_path=/yxvvuh-26/obj 2022-09-13T07:44:58.796 DEBUG:boto:Method: HEAD 2022-09-13T07:44:58.796 DEBUG:boto:Path: /yxvvuh-26/obj 2022-09-13T07:44:58.797 DEBUG:boto:Data: 2022-09-13T07:44:58.797 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.798 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:58.799 DEBUG:boto:Port: 8001 2022-09-13T07:44:58.799 DEBUG:boto:Params: {} 2022-09-13T07:44:58.800 DEBUG:boto:Token: None 2022-09-13T07:44:58.800 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/obj 2022-09-13T07:44:58.801 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rd1MplOT6neK41eX5ksLNcZYMvk= 2022-09-13T07:44:58.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rd1MplOT6neK41eX5ksLNcZYMvk=', 'Content-Length': '0'} 2022-09-13T07:44:58.803 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000827d34f9ca4f4e02-00632034fa-10f3-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.804 DEBUG:boto:path=/yxvvuh-26/obj 2022-09-13T07:44:58.804 DEBUG:boto:auth_path=/yxvvuh-26/obj 2022-09-13T07:44:58.805 DEBUG:boto:Method: HEAD 2022-09-13T07:44:58.806 DEBUG:boto:Path: /yxvvuh-26/obj 2022-09-13T07:44:58.806 DEBUG:boto:Data: 2022-09-13T07:44:58.807 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.807 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:58.808 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.808 DEBUG:boto:Params: {} 2022-09-13T07:44:58.809 DEBUG:boto:Token: None 2022-09-13T07:44:58.809 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-26/obj 2022-09-13T07:44:58.810 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rd1MplOT6neK41eX5ksLNcZYMvk= 2022-09-13T07:44:58.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rd1MplOT6neK41eX5ksLNcZYMvk=', 'Content-Length': '0'} 2022-09-13T07:44:58.812 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000092f9d6c1551f7abc-00632034fa-10c9-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.813 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yxvvuh-26 zones={test-zone3, test-zone2} 2022-09-13T07:44:58.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2022-09-13T07:44:58.815 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-27 2022-09-13T07:44:58.816 DEBUG:boto:path=/yxvvuh-27/ 2022-09-13T07:44:58.816 DEBUG:boto:auth_path=/yxvvuh-27/ 2022-09-13T07:44:58.816 DEBUG:boto:Method: PUT 2022-09-13T07:44:58.817 DEBUG:boto:Path: /yxvvuh-27/ 2022-09-13T07:44:58.817 DEBUG:boto:Data: 2022-09-13T07:44:58.818 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.819 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:58.819 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.819 DEBUG:boto:Params: {} 2022-09-13T07:44:58.820 DEBUG:boto:Token: None 2022-09-13T07:44:58.821 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-27/ 2022-09-13T07:44:58.821 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3Uklbu7cqoKepdLSj9RZQhjoEuw= 2022-09-13T07:44:58.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3Uklbu7cqoKepdLSj9RZQhjoEuw=', 'Content-Length': '0'} 2022-09-13T07:44:58.838 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000abeae104e30d57fa-00632034fa-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.839 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-28 2022-09-13T07:44:58.839 DEBUG:boto:path=/yxvvuh-28/ 2022-09-13T07:44:58.840 DEBUG:boto:auth_path=/yxvvuh-28/ 2022-09-13T07:44:58.841 DEBUG:boto:Method: PUT 2022-09-13T07:44:58.841 DEBUG:boto:Path: /yxvvuh-28/ 2022-09-13T07:44:58.842 DEBUG:boto:Data: 2022-09-13T07:44:58.842 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.843 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:58.844 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.844 DEBUG:boto:Params: {} 2022-09-13T07:44:58.845 DEBUG:boto:Token: None 2022-09-13T07:44:58.845 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-28/ 2022-09-13T07:44:58.846 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vym/eTggNKFkPUeuxZ2w4+zN48E= 2022-09-13T07:44:58.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vym/eTggNKFkPUeuxZ2w4+zN48E=', 'Content-Length': '0'} 2022-09-13T07:44:58.879 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d03c90842f78b83d-00632034fa-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.881 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-29 2022-09-13T07:44:58.881 DEBUG:boto:path=/yxvvuh-29/ 2022-09-13T07:44:58.882 DEBUG:boto:auth_path=/yxvvuh-29/ 2022-09-13T07:44:58.883 DEBUG:boto:Method: PUT 2022-09-13T07:44:58.884 DEBUG:boto:Path: /yxvvuh-29/ 2022-09-13T07:44:58.885 DEBUG:boto:Data: 2022-09-13T07:44:58.886 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.887 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:58.888 DEBUG:boto:Port: 8001 2022-09-13T07:44:58.889 DEBUG:boto:Params: {} 2022-09-13T07:44:58.889 DEBUG:boto:Token: None 2022-09-13T07:44:58.890 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-29/ 2022-09-13T07:44:58.891 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:2V5J3ypP86WZ3FlEeWsUbgMmQvI= 2022-09-13T07:44:58.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:2V5J3ypP86WZ3FlEeWsUbgMmQvI=', 'Content-Length': '0'} 2022-09-13T07:44:58.935 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002a3b2ec07e08b3c2-00632034fa-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.936 DEBUG:boto:path=/yxvvuh-27/ 2022-09-13T07:44:58.937 DEBUG:boto:auth_path=/yxvvuh-27/ 2022-09-13T07:44:58.938 DEBUG:boto:path=/yxvvuh-27/?versioning 2022-09-13T07:44:58.939 DEBUG:boto:auth_path=/yxvvuh-27/?versioning 2022-09-13T07:44:58.940 DEBUG:boto:Method: PUT 2022-09-13T07:44:58.941 DEBUG:boto:Path: /yxvvuh-27/?versioning 2022-09-13T07:44:58.942 DEBUG:boto:Data: Enabled Disabled 2022-09-13T07:44:58.942 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.943 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:58.944 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.945 DEBUG:boto:Params: {} 2022-09-13T07:44:58.946 DEBUG:boto:Token: None 2022-09-13T07:44:58.947 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-27/?versioning 2022-09-13T07:44:58.948 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Xrns2zwTWuWybzLAZ++Ao+Ge3LM= 2022-09-13T07:44:58.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Xrns2zwTWuWybzLAZ++Ao+Ge3LM=', 'Content-Length': '228'} 2022-09-13T07:44:58.957 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cf5aa34fd1c88a4e-00632034fa-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.959 DEBUG:boto:path=/yxvvuh-27/ 2022-09-13T07:44:58.959 DEBUG:boto:auth_path=/yxvvuh-27/ 2022-09-13T07:44:58.960 DEBUG:boto:path=/yxvvuh-27/?versioning 2022-09-13T07:44:58.961 DEBUG:boto:auth_path=/yxvvuh-27/?versioning 2022-09-13T07:44:58.962 DEBUG:boto:Method: GET 2022-09-13T07:44:58.963 DEBUG:boto:Path: /yxvvuh-27/?versioning 2022-09-13T07:44:58.964 DEBUG:boto:Data: 2022-09-13T07:44:58.965 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.966 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:58.967 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.968 DEBUG:boto:Params: {} 2022-09-13T07:44:58.969 DEBUG:boto:Token: None 2022-09-13T07:44:58.970 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-27/?versioning 2022-09-13T07:44:58.971 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lCNnUNUa+OB2smGSI/qG5YP5mPc= 2022-09-13T07:44:58.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lCNnUNUa+OB2smGSI/qG5YP5mPc=', 'Content-Length': '0'} 2022-09-13T07:44:58.974 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000009e7a5a266b85448-00632034fa-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:58.975 DEBUG:boto:EnabledDisabled 2022-09-13T07:44:58.977 DEBUG:boto:path=/yxvvuh-28/ 2022-09-13T07:44:58.978 DEBUG:boto:auth_path=/yxvvuh-28/ 2022-09-13T07:44:58.979 DEBUG:boto:path=/yxvvuh-28/?versioning 2022-09-13T07:44:58.980 DEBUG:boto:auth_path=/yxvvuh-28/?versioning 2022-09-13T07:44:58.981 DEBUG:boto:Method: PUT 2022-09-13T07:44:58.981 DEBUG:boto:Path: /yxvvuh-28/?versioning 2022-09-13T07:44:58.982 DEBUG:boto:Data: Enabled Disabled 2022-09-13T07:44:58.983 DEBUG:boto:Headers: {} 2022-09-13T07:44:58.984 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:58.985 DEBUG:boto:Port: 8000 2022-09-13T07:44:58.986 DEBUG:boto:Params: {} 2022-09-13T07:44:58.986 DEBUG:boto:Token: None 2022-09-13T07:44:58.987 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:44:58 GMT /yxvvuh-28/?versioning 2022-09-13T07:44:58.989 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:SgwyMDq0nf4qp4kAnhUNQ0mAWiY= 2022-09-13T07:44:58.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:58 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:SgwyMDq0nf4qp4kAnhUNQ0mAWiY=', 'Content-Length': '228'} 2022-09-13T07:44:58.999 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d8d9a7650f96811d-00632034fa-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.001 DEBUG:boto:path=/yxvvuh-28/ 2022-09-13T07:44:59.002 DEBUG:boto:auth_path=/yxvvuh-28/ 2022-09-13T07:44:59.002 DEBUG:boto:path=/yxvvuh-28/?versioning 2022-09-13T07:44:59.003 DEBUG:boto:auth_path=/yxvvuh-28/?versioning 2022-09-13T07:44:59.004 DEBUG:boto:Method: GET 2022-09-13T07:44:59.005 DEBUG:boto:Path: /yxvvuh-28/?versioning 2022-09-13T07:44:59.006 DEBUG:boto:Data: 2022-09-13T07:44:59.007 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.008 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:59.009 DEBUG:boto:Port: 8000 2022-09-13T07:44:59.009 DEBUG:boto:Params: {} 2022-09-13T07:44:59.010 DEBUG:boto:Token: None 2022-09-13T07:44:59.011 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-28/?versioning 2022-09-13T07:44:59.012 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:aZgH7pqvdelQIJy9+PT4qocUEHc= 2022-09-13T07:44:59.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aZgH7pqvdelQIJy9+PT4qocUEHc=', 'Content-Length': '0'} 2022-09-13T07:44:59.015 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000013a32f8d2e813b8c-00632034fb-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.058 DEBUG:boto:EnabledDisabled 2022-09-13T07:44:59.059 DEBUG:boto:path=/yxvvuh-29/ 2022-09-13T07:44:59.059 DEBUG:boto:auth_path=/yxvvuh-29/ 2022-09-13T07:44:59.060 DEBUG:boto:path=/yxvvuh-29/?versioning 2022-09-13T07:44:59.061 DEBUG:boto:auth_path=/yxvvuh-29/?versioning 2022-09-13T07:44:59.062 DEBUG:boto:Method: PUT 2022-09-13T07:44:59.063 DEBUG:boto:Path: /yxvvuh-29/?versioning 2022-09-13T07:44:59.064 DEBUG:boto:Data: Enabled Disabled 2022-09-13T07:44:59.065 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.065 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:59.066 DEBUG:boto:Port: 8001 2022-09-13T07:44:59.067 DEBUG:boto:Params: {} 2022-09-13T07:44:59.067 DEBUG:boto:Token: None 2022-09-13T07:44:59.068 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-29/?versioning 2022-09-13T07:44:59.069 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tff18fW/g6qVPezR6Qfd+xceMSw= 2022-09-13T07:44:59.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tff18fW/g6qVPezR6Qfd+xceMSw=', 'Content-Length': '228'} 2022-09-13T07:44:59.078 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000081502986e66e248f-00632034fb-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.080 DEBUG:boto:path=/yxvvuh-29/ 2022-09-13T07:44:59.080 DEBUG:boto:auth_path=/yxvvuh-29/ 2022-09-13T07:44:59.081 DEBUG:boto:path=/yxvvuh-29/?versioning 2022-09-13T07:44:59.081 DEBUG:boto:auth_path=/yxvvuh-29/?versioning 2022-09-13T07:44:59.082 DEBUG:boto:Method: GET 2022-09-13T07:44:59.082 DEBUG:boto:Path: /yxvvuh-29/?versioning 2022-09-13T07:44:59.083 DEBUG:boto:Data: 2022-09-13T07:44:59.084 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.085 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:59.086 DEBUG:boto:Port: 8001 2022-09-13T07:44:59.086 DEBUG:boto:Params: {} 2022-09-13T07:44:59.087 DEBUG:boto:Token: None 2022-09-13T07:44:59.088 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-29/?versioning 2022-09-13T07:44:59.089 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IFV+EwMM4WoJGomP9j833qPOSB8= 2022-09-13T07:44:59.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IFV+EwMM4WoJGomP9j833qPOSB8=', 'Content-Length': '0'} 2022-09-13T07:44:59.091 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000838e83bb749d592f-00632034fb-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.092 DEBUG:boto:EnabledDisabled 2022-09-13T07:44:59.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2022-09-13T07:44:59.093 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-30 2022-09-13T07:44:59.094 DEBUG:boto:path=/yxvvuh-30/ 2022-09-13T07:44:59.094 DEBUG:boto:auth_path=/yxvvuh-30/ 2022-09-13T07:44:59.095 DEBUG:boto:Method: PUT 2022-09-13T07:44:59.095 DEBUG:boto:Path: /yxvvuh-30/ 2022-09-13T07:44:59.096 DEBUG:boto:Data: 2022-09-13T07:44:59.097 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.097 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:59.098 DEBUG:boto:Port: 8000 2022-09-13T07:44:59.098 DEBUG:boto:Params: {} 2022-09-13T07:44:59.099 DEBUG:boto:Token: None 2022-09-13T07:44:59.100 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-30/ 2022-09-13T07:44:59.100 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:vHAju673UFxswvB5BOskz3VMots= 2022-09-13T07:44:59.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vHAju673UFxswvB5BOskz3VMots=', 'Content-Length': '0'} 2022-09-13T07:44:59.115 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000070ffa7167c688c27-00632034fb-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.116 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-31 2022-09-13T07:44:59.117 DEBUG:boto:path=/yxvvuh-31/ 2022-09-13T07:44:59.118 DEBUG:boto:auth_path=/yxvvuh-31/ 2022-09-13T07:44:59.119 DEBUG:boto:Method: PUT 2022-09-13T07:44:59.120 DEBUG:boto:Path: /yxvvuh-31/ 2022-09-13T07:44:59.121 DEBUG:boto:Data: 2022-09-13T07:44:59.122 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.123 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:59.124 DEBUG:boto:Port: 8000 2022-09-13T07:44:59.124 DEBUG:boto:Params: {} 2022-09-13T07:44:59.125 DEBUG:boto:Token: None 2022-09-13T07:44:59.126 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-31/ 2022-09-13T07:44:59.126 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ABIcCseGBwFmtCLytJDkfJMQP2c= 2022-09-13T07:44:59.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ABIcCseGBwFmtCLytJDkfJMQP2c=', 'Content-Length': '0'} 2022-09-13T07:44:59.163 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000054896004bbd9adc1-00632034fb-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.165 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-32 2022-09-13T07:44:59.165 DEBUG:boto:path=/yxvvuh-32/ 2022-09-13T07:44:59.166 DEBUG:boto:auth_path=/yxvvuh-32/ 2022-09-13T07:44:59.167 DEBUG:boto:Method: PUT 2022-09-13T07:44:59.168 DEBUG:boto:Path: /yxvvuh-32/ 2022-09-13T07:44:59.168 DEBUG:boto:Data: 2022-09-13T07:44:59.169 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.169 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:59.170 DEBUG:boto:Port: 8001 2022-09-13T07:44:59.171 DEBUG:boto:Params: {} 2022-09-13T07:44:59.172 DEBUG:boto:Token: None 2022-09-13T07:44:59.172 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-32/ 2022-09-13T07:44:59.173 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:+dPJbNdLjhLEpDbvwlQzAC5DLtE= 2022-09-13T07:44:59.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:+dPJbNdLjhLEpDbvwlQzAC5DLtE=', 'Content-Length': '0'} 2022-09-13T07:44:59.204 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c9f764f07a2d9eac-00632034fb-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.205 DEBUG:boto:path=/yxvvuh-30/ 2022-09-13T07:44:59.205 DEBUG:boto:auth_path=/yxvvuh-30/ 2022-09-13T07:44:59.206 DEBUG:boto:path=/yxvvuh-30/?acl 2022-09-13T07:44:59.206 DEBUG:boto:auth_path=/yxvvuh-30/?acl 2022-09-13T07:44:59.207 DEBUG:boto:Method: GET 2022-09-13T07:44:59.207 DEBUG:boto:Path: /yxvvuh-30/?acl 2022-09-13T07:44:59.208 DEBUG:boto:Data: 2022-09-13T07:44:59.208 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.209 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:59.209 DEBUG:boto:Port: 8000 2022-09-13T07:44:59.210 DEBUG:boto:Params: {} 2022-09-13T07:44:59.210 DEBUG:boto:Token: None 2022-09-13T07:44:59.211 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-30/?acl 2022-09-13T07:44:59.211 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:JPJadtkZek0/5GjoaR0IjzIJReg= 2022-09-13T07:44:59.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JPJadtkZek0/5GjoaR0IjzIJReg=', 'Content-Length': '0'} 2022-09-13T07:44:59.213 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003b71117f0aa3cbfa-00632034fb-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.214 DEBUG:boto:path=/yxvvuh-30/ 2022-09-13T07:44:59.215 DEBUG:boto:auth_path=/yxvvuh-30/ 2022-09-13T07:44:59.215 DEBUG:boto:path=/yxvvuh-30/?acl 2022-09-13T07:44:59.216 DEBUG:boto:auth_path=/yxvvuh-30/?acl 2022-09-13T07:44:59.216 DEBUG:boto:Method: PUT 2022-09-13T07:44:59.217 DEBUG:boto:Path: /yxvvuh-30/?acl 2022-09-13T07:44:59.217 DEBUG:boto:Data: 2022-09-13T07:44:59.218 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-13T07:44:59.218 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:59.219 DEBUG:boto:Port: 8000 2022-09-13T07:44:59.219 DEBUG:boto:Params: {} 2022-09-13T07:44:59.220 DEBUG:boto:Token: None 2022-09-13T07:44:59.220 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:44:59 GMT x-amz-acl:public-read /yxvvuh-30/?acl 2022-09-13T07:44:59.221 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9YtjHRl1etA69s1n7Ndihqbh2k4= 2022-09-13T07:44:59.221 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9YtjHRl1etA69s1n7Ndihqbh2k4=', 'Content-Length': '0'} 2022-09-13T07:44:59.226 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009ad442615f557f0a-00632034fb-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.227 DEBUG:boto:path=/yxvvuh-30/ 2022-09-13T07:44:59.227 DEBUG:boto:auth_path=/yxvvuh-30/ 2022-09-13T07:44:59.228 DEBUG:boto:path=/yxvvuh-30/?acl 2022-09-13T07:44:59.228 DEBUG:boto:auth_path=/yxvvuh-30/?acl 2022-09-13T07:44:59.229 DEBUG:boto:Method: GET 2022-09-13T07:44:59.229 DEBUG:boto:Path: /yxvvuh-30/?acl 2022-09-13T07:44:59.230 DEBUG:boto:Data: 2022-09-13T07:44:59.231 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.231 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:59.232 DEBUG:boto:Port: 8000 2022-09-13T07:44:59.232 DEBUG:boto:Params: {} 2022-09-13T07:44:59.233 DEBUG:boto:Token: None 2022-09-13T07:44:59.234 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-30/?acl 2022-09-13T07:44:59.234 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:JPJadtkZek0/5GjoaR0IjzIJReg= 2022-09-13T07:44:59.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JPJadtkZek0/5GjoaR0IjzIJReg=', 'Content-Length': '0'} 2022-09-13T07:44:59.236 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000094ed1497de14203d-00632034fb-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.278 DEBUG:boto:path=/yxvvuh-31/ 2022-09-13T07:44:59.279 DEBUG:boto:auth_path=/yxvvuh-31/ 2022-09-13T07:44:59.279 DEBUG:boto:path=/yxvvuh-31/?acl 2022-09-13T07:44:59.280 DEBUG:boto:auth_path=/yxvvuh-31/?acl 2022-09-13T07:44:59.280 DEBUG:boto:Method: GET 2022-09-13T07:44:59.281 DEBUG:boto:Path: /yxvvuh-31/?acl 2022-09-13T07:44:59.281 DEBUG:boto:Data: 2022-09-13T07:44:59.282 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.282 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:59.283 DEBUG:boto:Port: 8000 2022-09-13T07:44:59.283 DEBUG:boto:Params: {} 2022-09-13T07:44:59.285 DEBUG:boto:Token: None 2022-09-13T07:44:59.285 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-31/?acl 2022-09-13T07:44:59.286 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3pvD2BccdUw+z6deHJZOwdw7WwA= 2022-09-13T07:44:59.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3pvD2BccdUw+z6deHJZOwdw7WwA=', 'Content-Length': '0'} 2022-09-13T07:44:59.288 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000a3369a541ec44ce-00632034fb-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.289 DEBUG:boto:path=/yxvvuh-31/ 2022-09-13T07:44:59.289 DEBUG:boto:auth_path=/yxvvuh-31/ 2022-09-13T07:44:59.290 DEBUG:boto:path=/yxvvuh-31/?acl 2022-09-13T07:44:59.290 DEBUG:boto:auth_path=/yxvvuh-31/?acl 2022-09-13T07:44:59.291 DEBUG:boto:Method: PUT 2022-09-13T07:44:59.291 DEBUG:boto:Path: /yxvvuh-31/?acl 2022-09-13T07:44:59.292 DEBUG:boto:Data: 2022-09-13T07:44:59.292 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-13T07:44:59.293 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:59.293 DEBUG:boto:Port: 8000 2022-09-13T07:44:59.294 DEBUG:boto:Params: {} 2022-09-13T07:44:59.294 DEBUG:boto:Token: None 2022-09-13T07:44:59.295 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:44:59 GMT x-amz-acl:public-read /yxvvuh-31/?acl 2022-09-13T07:44:59.296 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:M6xH6+hG29bI8gvPUcCQyy6txaw= 2022-09-13T07:44:59.296 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:M6xH6+hG29bI8gvPUcCQyy6txaw=', 'Content-Length': '0'} 2022-09-13T07:44:59.310 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000076c6f6e225dc90f-00632034fb-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.311 DEBUG:boto:path=/yxvvuh-31/ 2022-09-13T07:44:59.311 DEBUG:boto:auth_path=/yxvvuh-31/ 2022-09-13T07:44:59.312 DEBUG:boto:path=/yxvvuh-31/?acl 2022-09-13T07:44:59.312 DEBUG:boto:auth_path=/yxvvuh-31/?acl 2022-09-13T07:44:59.313 DEBUG:boto:Method: GET 2022-09-13T07:44:59.313 DEBUG:boto:Path: /yxvvuh-31/?acl 2022-09-13T07:44:59.313 DEBUG:boto:Data: 2022-09-13T07:44:59.314 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.314 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:59.315 DEBUG:boto:Port: 8000 2022-09-13T07:44:59.315 DEBUG:boto:Params: {} 2022-09-13T07:44:59.316 DEBUG:boto:Token: None 2022-09-13T07:44:59.316 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-31/?acl 2022-09-13T07:44:59.317 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:3pvD2BccdUw+z6deHJZOwdw7WwA= 2022-09-13T07:44:59.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:3pvD2BccdUw+z6deHJZOwdw7WwA=', 'Content-Length': '0'} 2022-09-13T07:44:59.319 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000118b5b7e621a81b6-00632034fb-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.321 DEBUG:boto:path=/yxvvuh-32/ 2022-09-13T07:44:59.321 DEBUG:boto:auth_path=/yxvvuh-32/ 2022-09-13T07:44:59.322 DEBUG:boto:path=/yxvvuh-32/?acl 2022-09-13T07:44:59.322 DEBUG:boto:auth_path=/yxvvuh-32/?acl 2022-09-13T07:44:59.323 DEBUG:boto:Method: GET 2022-09-13T07:44:59.323 DEBUG:boto:Path: /yxvvuh-32/?acl 2022-09-13T07:44:59.323 DEBUG:boto:Data: 2022-09-13T07:44:59.324 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.324 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:59.325 DEBUG:boto:Port: 8001 2022-09-13T07:44:59.325 DEBUG:boto:Params: {} 2022-09-13T07:44:59.326 DEBUG:boto:Token: None 2022-09-13T07:44:59.326 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-32/?acl 2022-09-13T07:44:59.327 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dTP30c7n/ktf2neFschU3PDKVwQ= 2022-09-13T07:44:59.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dTP30c7n/ktf2neFschU3PDKVwQ=', 'Content-Length': '0'} 2022-09-13T07:44:59.329 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b908c8a77b3a81e7-00632034fb-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.330 DEBUG:boto:path=/yxvvuh-32/ 2022-09-13T07:44:59.331 DEBUG:boto:auth_path=/yxvvuh-32/ 2022-09-13T07:44:59.332 DEBUG:boto:path=/yxvvuh-32/?acl 2022-09-13T07:44:59.332 DEBUG:boto:auth_path=/yxvvuh-32/?acl 2022-09-13T07:44:59.333 DEBUG:boto:Method: PUT 2022-09-13T07:44:59.334 DEBUG:boto:Path: /yxvvuh-32/?acl 2022-09-13T07:44:59.334 DEBUG:boto:Data: 2022-09-13T07:44:59.335 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-13T07:44:59.336 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:59.336 DEBUG:boto:Port: 8001 2022-09-13T07:44:59.337 DEBUG:boto:Params: {} 2022-09-13T07:44:59.338 DEBUG:boto:Token: None 2022-09-13T07:44:59.339 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:44:59 GMT x-amz-acl:public-read /yxvvuh-32/?acl 2022-09-13T07:44:59.339 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:L8/eYQnypRy95JB6cBY+fXF9O54= 2022-09-13T07:44:59.340 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:L8/eYQnypRy95JB6cBY+fXF9O54=', 'Content-Length': '0'} 2022-09-13T07:44:59.356 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000001189feaeaf37c0e-00632034fb-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.357 DEBUG:boto:path=/yxvvuh-32/ 2022-09-13T07:44:59.358 DEBUG:boto:auth_path=/yxvvuh-32/ 2022-09-13T07:44:59.359 DEBUG:boto:path=/yxvvuh-32/?acl 2022-09-13T07:44:59.360 DEBUG:boto:auth_path=/yxvvuh-32/?acl 2022-09-13T07:44:59.360 DEBUG:boto:Method: GET 2022-09-13T07:44:59.361 DEBUG:boto:Path: /yxvvuh-32/?acl 2022-09-13T07:44:59.362 DEBUG:boto:Data: 2022-09-13T07:44:59.363 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.363 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:59.364 DEBUG:boto:Port: 8001 2022-09-13T07:44:59.365 DEBUG:boto:Params: {} 2022-09-13T07:44:59.366 DEBUG:boto:Token: None 2022-09-13T07:44:59.367 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-32/?acl 2022-09-13T07:44:59.368 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dTP30c7n/ktf2neFschU3PDKVwQ= 2022-09-13T07:44:59.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dTP30c7n/ktf2neFschU3PDKVwQ=', 'Content-Length': '0'} 2022-09-13T07:44:59.370 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002a15e86d6045e69e-00632034fb-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2022-09-13T07:44:59.419 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-33 2022-09-13T07:44:59.420 DEBUG:boto:path=/yxvvuh-33/ 2022-09-13T07:44:59.420 DEBUG:boto:auth_path=/yxvvuh-33/ 2022-09-13T07:44:59.421 DEBUG:boto:Method: PUT 2022-09-13T07:44:59.422 DEBUG:boto:Path: /yxvvuh-33/ 2022-09-13T07:44:59.422 DEBUG:boto:Data: 2022-09-13T07:44:59.423 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.423 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:59.424 DEBUG:boto:Port: 8000 2022-09-13T07:44:59.424 DEBUG:boto:Params: {} 2022-09-13T07:44:59.425 DEBUG:boto:Token: None 2022-09-13T07:44:59.425 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-33/ 2022-09-13T07:44:59.426 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:e2dGvxqL2jro81nhWu5s+869BbQ= 2022-09-13T07:44:59.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:e2dGvxqL2jro81nhWu5s+869BbQ=', 'Content-Length': '0'} 2022-09-13T07:44:59.442 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b1b0c44c518ea711-00632034fb-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.443 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-34 2022-09-13T07:44:59.443 DEBUG:boto:path=/yxvvuh-34/ 2022-09-13T07:44:59.444 DEBUG:boto:auth_path=/yxvvuh-34/ 2022-09-13T07:44:59.445 DEBUG:boto:Method: PUT 2022-09-13T07:44:59.445 DEBUG:boto:Path: /yxvvuh-34/ 2022-09-13T07:44:59.446 DEBUG:boto:Data: 2022-09-13T07:44:59.446 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.447 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:59.448 DEBUG:boto:Port: 8000 2022-09-13T07:44:59.448 DEBUG:boto:Params: {} 2022-09-13T07:44:59.449 DEBUG:boto:Token: None 2022-09-13T07:44:59.449 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-34/ 2022-09-13T07:44:59.450 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lX1VraoMUpS8b7MwsQLi2MeIORg= 2022-09-13T07:44:59.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lX1VraoMUpS8b7MwsQLi2MeIORg=', 'Content-Length': '0'} 2022-09-13T07:44:59.487 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000066ad8af37c467673-00632034fb-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.490 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-35 2022-09-13T07:44:59.491 DEBUG:boto:path=/yxvvuh-35/ 2022-09-13T07:44:59.492 DEBUG:boto:auth_path=/yxvvuh-35/ 2022-09-13T07:44:59.492 DEBUG:boto:Method: PUT 2022-09-13T07:44:59.493 DEBUG:boto:Path: /yxvvuh-35/ 2022-09-13T07:44:59.494 DEBUG:boto:Data: 2022-09-13T07:44:59.495 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.496 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:44:59.497 DEBUG:boto:Port: 8001 2022-09-13T07:44:59.498 DEBUG:boto:Params: {} 2022-09-13T07:44:59.499 DEBUG:boto:Token: None 2022-09-13T07:44:59.500 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-35/ 2022-09-13T07:44:59.501 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1/8EAZUKpnI1Pc3QNLw9FPQ4Z1Q= 2022-09-13T07:44:59.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1/8EAZUKpnI1Pc3QNLw9FPQ4Z1Q=', 'Content-Length': '0'} 2022-09-13T07:44:59.539 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ab89634ee38adefa-00632034fb-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.540 DEBUG:boto:path=/yxvvuh-33/ 2022-09-13T07:44:59.540 DEBUG:boto:auth_path=/yxvvuh-33/ 2022-09-13T07:44:59.541 DEBUG:boto:path=/yxvvuh-33/?cors 2022-09-13T07:44:59.542 DEBUG:boto:auth_path=/yxvvuh-33/?cors 2022-09-13T07:44:59.542 DEBUG:boto:Method: PUT 2022-09-13T07:44:59.543 DEBUG:boto:Path: /yxvvuh-33/?cors 2022-09-13T07:44:59.543 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2022-09-13T07:44:59.543 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2022-09-13T07:44:59.544 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:59.544 DEBUG:boto:Port: 8000 2022-09-13T07:44:59.545 DEBUG:boto:Params: {} 2022-09-13T07:44:59.545 DEBUG:boto:Token: None 2022-09-13T07:44:59.546 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-33/?cors 2022-09-13T07:44:59.547 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uYYQZrBhlj28pWX+j957sm/FTmw= 2022-09-13T07:44:59.547 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uYYQZrBhlj28pWX+j957sm/FTmw=', 'Content-Length': '218'} 2022-09-13T07:44:59.570 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002fc87d87e831959f-00632034fb-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.571 DEBUG:boto:path=/yxvvuh-33/ 2022-09-13T07:44:59.571 DEBUG:boto:auth_path=/yxvvuh-33/ 2022-09-13T07:44:59.572 DEBUG:boto:path=/yxvvuh-33/?cors 2022-09-13T07:44:59.572 DEBUG:boto:auth_path=/yxvvuh-33/?cors 2022-09-13T07:44:59.573 DEBUG:boto:Method: GET 2022-09-13T07:44:59.573 DEBUG:boto:Path: /yxvvuh-33/?cors 2022-09-13T07:44:59.574 DEBUG:boto:Data: 2022-09-13T07:44:59.575 DEBUG:boto:Headers: {} 2022-09-13T07:44:59.575 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:44:59.576 DEBUG:boto:Port: 8000 2022-09-13T07:44:59.576 DEBUG:boto:Params: {} 2022-09-13T07:44:59.577 DEBUG:boto:Token: None 2022-09-13T07:44:59.577 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-33/?cors 2022-09-13T07:44:59.578 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:5BfTAiRhs9ljD6p1osvywUyQJa4= 2022-09-13T07:44:59.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5BfTAiRhs9ljD6p1osvywUyQJa4=', 'Content-Length': '0'} 2022-09-13T07:44:59.580 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d1bfdae5ce7c601f-00632034fb-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Tue, 13 Sep 2022 07:44:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:44:59.581 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2022-09-13T07:44:59.581 DEBUG:boto:path=/yxvvuh-34/ 2022-09-13T07:44:59.582 DEBUG:boto:auth_path=/yxvvuh-34/ 2022-09-13T07:44:59.582 DEBUG:boto:path=/yxvvuh-34/?cors 2022-09-13T07:44:59.583 DEBUG:boto:auth_path=/yxvvuh-34/?cors 2022-09-13T07:44:59.583 DEBUG:boto:Method: PUT 2022-09-13T07:44:59.584 DEBUG:boto:Path: /yxvvuh-34/?cors 2022-09-13T07:44:59.584 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2022-09-13T07:44:59.585 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2022-09-13T07:44:59.585 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:44:59.586 DEBUG:boto:Port: 8000 2022-09-13T07:44:59.586 DEBUG:boto:Params: {} 2022-09-13T07:44:59.587 DEBUG:boto:Token: None 2022-09-13T07:44:59.587 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Tue, 13 Sep 2022 07:44:59 GMT /yxvvuh-34/?cors 2022-09-13T07:44:59.588 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:uYpu1GGOqFbFMhc72+hUWWCQJH8= 2022-09-13T07:44:59.588 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:44:59 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uYpu1GGOqFbFMhc72+hUWWCQJH8=', 'Content-Length': '218'} 2022-09-13T07:45:00.598 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000424d2eb8a939356f-00632034fb-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:45:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:45:00.598 DEBUG:boto:path=/yxvvuh-34/ 2022-09-13T07:45:00.599 DEBUG:boto:auth_path=/yxvvuh-34/ 2022-09-13T07:45:00.599 DEBUG:boto:path=/yxvvuh-34/?cors 2022-09-13T07:45:00.601 DEBUG:boto:auth_path=/yxvvuh-34/?cors 2022-09-13T07:45:00.601 DEBUG:boto:Method: GET 2022-09-13T07:45:00.602 DEBUG:boto:Path: /yxvvuh-34/?cors 2022-09-13T07:45:00.602 DEBUG:boto:Data: 2022-09-13T07:45:00.603 DEBUG:boto:Headers: {} 2022-09-13T07:45:00.604 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:45:00.604 DEBUG:boto:Port: 8000 2022-09-13T07:45:00.605 DEBUG:boto:Params: {} 2022-09-13T07:45:00.607 DEBUG:boto:Token: None 2022-09-13T07:45:00.608 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:45:00 GMT /yxvvuh-34/?cors 2022-09-13T07:45:00.609 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:i3w2nv3PM0t9OKNYheDtBIDSmlQ= 2022-09-13T07:45:00.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:45:00 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:i3w2nv3PM0t9OKNYheDtBIDSmlQ=', 'Content-Length': '0'} 2022-09-13T07:45:00.611 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cffc490a3850a0b7-00632034fc-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Tue, 13 Sep 2022 07:45:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:45:00.612 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2022-09-13T07:45:00.613 DEBUG:boto:path=/yxvvuh-35/ 2022-09-13T07:45:00.613 DEBUG:boto:auth_path=/yxvvuh-35/ 2022-09-13T07:45:00.614 DEBUG:boto:path=/yxvvuh-35/?cors 2022-09-13T07:45:00.614 DEBUG:boto:auth_path=/yxvvuh-35/?cors 2022-09-13T07:45:00.615 DEBUG:boto:Method: PUT 2022-09-13T07:45:00.615 DEBUG:boto:Path: /yxvvuh-35/?cors 2022-09-13T07:45:00.616 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2022-09-13T07:45:00.616 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2022-09-13T07:45:00.617 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:45:00.617 DEBUG:boto:Port: 8001 2022-09-13T07:45:00.618 DEBUG:boto:Params: {} 2022-09-13T07:45:00.619 DEBUG:boto:Token: None 2022-09-13T07:45:00.619 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Tue, 13 Sep 2022 07:45:00 GMT /yxvvuh-35/?cors 2022-09-13T07:45:00.620 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:XNMPoAsZoYXYkXb8fymnv/JeEWY= 2022-09-13T07:45:00.622 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:45:00 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:XNMPoAsZoYXYkXb8fymnv/JeEWY=', 'Content-Length': '218'} 2022-09-13T07:45:01.636 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001bdbac95f6d10395-00632034fc-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:45:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:45:01.637 DEBUG:boto:path=/yxvvuh-35/ 2022-09-13T07:45:01.638 DEBUG:boto:auth_path=/yxvvuh-35/ 2022-09-13T07:45:01.638 DEBUG:boto:path=/yxvvuh-35/?cors 2022-09-13T07:45:01.639 DEBUG:boto:auth_path=/yxvvuh-35/?cors 2022-09-13T07:45:01.639 DEBUG:boto:Method: GET 2022-09-13T07:45:01.640 DEBUG:boto:Path: /yxvvuh-35/?cors 2022-09-13T07:45:01.640 DEBUG:boto:Data: 2022-09-13T07:45:01.641 DEBUG:boto:Headers: {} 2022-09-13T07:45:01.641 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:45:01.642 DEBUG:boto:Port: 8001 2022-09-13T07:45:01.642 DEBUG:boto:Params: {} 2022-09-13T07:45:01.643 DEBUG:boto:Token: None 2022-09-13T07:45:01.644 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 07:45:01 GMT /yxvvuh-35/?cors 2022-09-13T07:45:01.645 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:QcVTwoU6Xb9dGBMldxX5ds9Nr4U= 2022-09-13T07:45:01.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:45:01 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QcVTwoU6Xb9dGBMldxX5ds9Nr4U=', 'Content-Length': '0'} 2022-09-13T07:45:01.647 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000011b639fdde5a113d-00632034fd-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Tue, 13 Sep 2022 07:45:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:45:01.648 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2022-09-13T07:45:01.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2022-09-13T07:45:01.650 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-36 2022-09-13T07:45:01.650 DEBUG:boto:path=/yxvvuh-36/ 2022-09-13T07:45:01.651 DEBUG:boto:auth_path=/yxvvuh-36/ 2022-09-13T07:45:01.651 DEBUG:boto:Method: PUT 2022-09-13T07:45:01.652 DEBUG:boto:Path: /yxvvuh-36/ 2022-09-13T07:45:01.652 DEBUG:boto:Data: 2022-09-13T07:45:01.653 DEBUG:boto:Headers: {} 2022-09-13T07:45:01.653 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:45:01.654 DEBUG:boto:Port: 8000 2022-09-13T07:45:01.654 DEBUG:boto:Params: {} 2022-09-13T07:45:01.655 DEBUG:boto:Token: None 2022-09-13T07:45:01.655 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:45:01 GMT /yxvvuh-36/ 2022-09-13T07:45:01.656 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1PAGcd/B9huMFYnrzo7LV2TerNs= 2022-09-13T07:45:01.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:45:01 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1PAGcd/B9huMFYnrzo7LV2TerNs=', 'Content-Length': '0'} 2022-09-13T07:45:01.693 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b1f299f414eb203f-00632034fd-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:45:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:45:01.694 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-37 2022-09-13T07:45:01.695 DEBUG:boto:path=/yxvvuh-37/ 2022-09-13T07:45:01.697 DEBUG:boto:auth_path=/yxvvuh-37/ 2022-09-13T07:45:01.698 DEBUG:boto:Method: PUT 2022-09-13T07:45:01.699 DEBUG:boto:Path: /yxvvuh-37/ 2022-09-13T07:45:01.699 DEBUG:boto:Data: 2022-09-13T07:45:01.700 DEBUG:boto:Headers: {} 2022-09-13T07:45:01.701 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:45:01.701 DEBUG:boto:Port: 8000 2022-09-13T07:45:01.702 DEBUG:boto:Params: {} 2022-09-13T07:45:01.703 DEBUG:boto:Token: None 2022-09-13T07:45:01.704 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:45:01 GMT /yxvvuh-37/ 2022-09-13T07:45:01.705 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:y8aFfrd7Ow+XZDeXWjkLko2PA7Y= 2022-09-13T07:45:01.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:45:01 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:y8aFfrd7Ow+XZDeXWjkLko2PA7Y=', 'Content-Length': '0'} 2022-09-13T07:45:01.758 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ca5d6d20cddc8cf4-00632034fd-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:45:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:45:01.759 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-38 2022-09-13T07:45:01.760 DEBUG:boto:path=/yxvvuh-38/ 2022-09-13T07:45:01.760 DEBUG:boto:auth_path=/yxvvuh-38/ 2022-09-13T07:45:01.761 DEBUG:boto:Method: PUT 2022-09-13T07:45:01.762 DEBUG:boto:Path: /yxvvuh-38/ 2022-09-13T07:45:01.762 DEBUG:boto:Data: 2022-09-13T07:45:01.763 DEBUG:boto:Headers: {} 2022-09-13T07:45:01.764 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:45:01.764 DEBUG:boto:Port: 8001 2022-09-13T07:45:01.765 DEBUG:boto:Params: {} 2022-09-13T07:45:01.766 DEBUG:boto:Token: None 2022-09-13T07:45:01.767 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:45:01 GMT /yxvvuh-38/ 2022-09-13T07:45:01.768 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lS5NYml0lKg3RVXjB+2L1BP7Ho0= 2022-09-13T07:45:01.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:45:01 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lS5NYml0lKg3RVXjB+2L1BP7Ho0=', 'Content-Length': '0'} 2022-09-13T07:45:01.798 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000090e745dc3b9c25c4-00632034fd-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:45:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:45:01.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:01.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:01.801 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:45:01.853 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:45:01.854 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:45:01.909 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:45:01.966 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:45:01.967 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:45:01.967 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:45:01.968 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:45:01.968 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:45:01.969 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:45:01.969 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:45:01.970 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:45:01.970 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:45:01.970 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:45:01.971 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663055101.786343_268.1", 2022-09-13T07:45:01.971 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:45:01.786343Z" 2022-09-13T07:45:01.972 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:45:01.972 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:45:01.972 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:45:01.973 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T07:45:01.973 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:45:01.974 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:45:01.975 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:01.975 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:45:01.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:01.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:01.978 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:45:02.027 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:45:02.028 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:45:02.056 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:02.051+0000 7fe090871600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:02.056 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:02.052+0000 7fe090871600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:02.068 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:45:02.069 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:45:02.069 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:45:02.070 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:45:02.070 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:45:02.070 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:45:02.071 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:02.071 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:02.071 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:45:02.071 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:02.072 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:45:02.072 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:02.072 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:02.072 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:45:02.073 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:02.073 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:45:02.073 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:02.074 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:45:02.074 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:02.074 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:02.075 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:02.075 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:02.075 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:45:02.075 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:02.076 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:02.076 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:45:02.076 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:02.077 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:02.077 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:02.077 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:45:02.077 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:02.078 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:02.078 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:02.078 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:02.078 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:45:02.079 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:02.079 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:02.080 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.858000_209.1", 2022-09-13T07:45:02.080 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:02.081 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:02.081 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:02.081 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T07:45:02.082 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:02.082 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:02.083 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:02.083 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:02.084 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:45:02.084 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:02.085 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:02.085 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:45:02.086 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:02.086 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:02.087 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:02.087 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:45:02.088 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:02.088 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:02.089 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:02.089 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:45:02.090 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:02.090 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:45:02.091 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:45:02.091 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:45:02.092 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:02.092 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:45:02Z" 2022-09-13T07:45:02.093 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:45:02.095 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.858000_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.858000Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:45:02Z" } 2022-09-13T07:45:02.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:45:02.097 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:02.098 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:02.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.858000_209.1 2022-09-13T07:45:07.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:07.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:07.102 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:45:07.166 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:45:07.166 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:45:07.247 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:07.242+0000 7f2ca377e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:07.248 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:07.243+0000 7f2ca377e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:07.273 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:45:07.273 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:45:07.274 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:45:07.274 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:45:07.275 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:45:07.275 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:45:07.275 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:07.275 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:07.276 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:45:07.276 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:07.276 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:45:07.276 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:07.277 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:07.277 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:45:07.277 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:07.277 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:45:07.278 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:07.278 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:45:07.278 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:07.279 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:07.279 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:07.279 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:07.279 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:45:07.280 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:07.280 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:07.280 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:45:07.280 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:07.281 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:07.282 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:07.283 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:45:07.283 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:07.284 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:07.284 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:07.284 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:07.285 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:45:07.285 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:07.286 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:07.286 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.900410_211.1", 2022-09-13T07:45:07.287 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:07.288 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:07.288 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:07.288 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T07:45:07.289 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:07.289 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:07.290 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:07.290 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:07.291 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:45:07.291 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:07.292 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:07.292 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:45:07.293 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:07.293 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:07.294 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:07.294 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:45:07.294 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:07.294 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:07.295 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:07.295 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:45:07.295 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:07.296 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:45:07.296 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:45:07.296 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:45:07.296 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:07.297 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:45:07Z" 2022-09-13T07:45:07.297 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:45:07.298 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.900410_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.900410Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:45:07Z" } 2022-09-13T07:45:07.299 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:45:07.300 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:07.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:07.301 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.900410_211.1 2022-09-13T07:45:12.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:12.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:12.303 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:45:12.353 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:45:12.354 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:45:12.380 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:12.376+0000 7fee8d2bd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:12.381 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:12.378+0000 7fee8d2bd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:12.393 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:45:12.393 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:45:12.393 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:45:12.394 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:45:12.394 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:45:12.394 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:45:12.394 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:12.395 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:12.395 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:45:12.395 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:12.396 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:45:12.396 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:12.397 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:12.397 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:45:12.398 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:12.398 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:45:12.399 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:12.399 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:45:12.400 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:12.400 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:12.400 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:12.401 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:12.401 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:45:12.402 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:12.402 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:12.402 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:45:12.403 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:12.403 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:12.404 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:12.404 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:45:12.404 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:12.405 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:12.405 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:12.405 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:12.405 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:45:12.406 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:12.406 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:12.406 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T07:45:12.407 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:12.407 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:12.407 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:12.407 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T07:45:12.408 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:12.408 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:12.408 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:12.408 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:12.409 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:45:12.409 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:12.409 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:12.409 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:45:12.410 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:12.410 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:12.410 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:12.410 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:45:12.411 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:12.411 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:12.411 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:12.411 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:45:12.412 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:12.412 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:45:12.413 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:45:12.413 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:45:12.414 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:12.414 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:45:12Z" 2022-09-13T07:45:12.415 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:45:12.417 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:45:12Z" } 2022-09-13T07:45:12.418 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:45:12.419 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:12.420 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:12.420 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.832045_205.1 2022-09-13T07:45:17.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:17.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:17.424 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:45:17.510 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:45:17.511 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:45:17.555 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:17.551+0000 7fe1fcacb600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:17.559 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:17.552+0000 7fe1fcacb600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:17.575 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:45:17.575 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:45:17.576 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:45:17.576 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:45:17.576 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:45:17.577 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:45:17.577 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:17.577 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:17.578 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:45:17.578 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:17.578 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:45:17.579 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:17.579 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:17.579 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:45:17.580 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:17.580 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:45:17.580 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:17.580 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:45:17.581 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:17.581 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:17.581 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:17.581 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:17.582 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:45:17.582 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:17.582 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:17.582 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:45:17.583 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:17.583 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:17.583 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:17.583 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:45:17.584 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:17.584 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:17.585 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:17.585 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:17.586 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:45:17.586 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:17.587 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:17.587 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.861945_210.1", 2022-09-13T07:45:17.588 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:17.588 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:17.588 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:17.589 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T07:45:17.589 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:17.589 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:17.590 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:17.590 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:17.590 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:45:17.590 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:17.591 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:17.591 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:45:17.591 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:17.592 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:17.592 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:17.592 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:45:17.592 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:17.593 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:17.593 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:17.594 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:45:17.594 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:17.595 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:45:17.596 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:45:17.596 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:45:17.596 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:17.597 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:45:17Z" 2022-09-13T07:45:17.597 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:45:17.599 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.861945_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.861945Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:45:17Z" } 2022-09-13T07:45:17.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:45:17.600 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:17.601 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:17.601 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.861945_210.1 2022-09-13T07:45:22.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:22.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:22.605 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:45:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:45:22.656 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:45:22.686 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:22.683+0000 7fee813dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:22.687 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:22.684+0000 7fee813dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:22.699 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:45:22.700 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:45:22.700 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:45:22.701 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:45:22.701 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:45:22.702 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:45:22.702 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:22.703 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:22.703 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:45:22.704 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:22.704 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:45:22.704 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:22.704 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:22.705 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:45:22.706 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:22.706 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:45:22.706 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:22.707 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:45:22.707 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:22.707 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:22.707 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:22.708 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:22.708 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:45:22.708 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:22.709 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:22.709 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:45:22.709 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:22.709 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:22.710 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:22.710 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:45:22.711 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:22.711 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:22.711 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:22.711 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:22.712 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:45:22.712 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:22.712 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:22.713 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.906479_212.1", 2022-09-13T07:45:22.713 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:22.713 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:22.713 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:22.714 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.906479Z", 2022-09-13T07:45:22.714 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:22.714 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:22.714 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:22.715 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:22.715 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:45:22.715 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:22.715 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:22.716 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:45:22.716 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:22.716 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:22.716 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:22.717 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:45:22.717 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:22.717 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:22.717 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:22.718 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:45:22.718 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:22.718 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:45:22.719 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:45:22.719 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:45:22.719 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:22.719 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:45:22Z" 2022-09-13T07:45:22.720 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:45:22.721 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.906479_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.906479Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:45:22Z" } 2022-09-13T07:45:22.721 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.906479_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.906479Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:45:22.722 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:22.723 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.906479_212.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:22.723 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.906479_212.1 2022-09-13T07:45:27.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:27.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:27.726 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:45:27.774 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:45:27.775 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:45:27.800 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:27.796+0000 7ff2c5bac600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:27.801 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:27.797+0000 7ff2c5bac600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:27.813 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:45:27.813 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:45:27.813 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:45:27.814 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:45:27.814 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:45:27.814 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:45:27.815 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:27.815 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:27.815 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:45:27.815 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:27.816 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:45:27.816 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:27.816 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:27.816 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:45:27.817 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:27.817 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:45:27.817 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:27.818 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:45:27.818 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:27.818 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:27.818 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:27.819 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:27.819 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:45:27.819 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:27.820 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:27.820 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:45:27.820 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:27.820 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:27.821 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:27.821 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:45:27.821 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:27.821 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:27.822 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:27.822 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:27.822 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:45:27.822 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:27.823 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:27.823 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:45:27.823 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:27.823 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:27.824 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:27.824 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:45:27.824 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:27.824 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:27.825 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:27.825 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:27.825 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:45:27.826 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:27.826 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:27.826 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:45:27.827 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:27.827 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:27.827 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:27.827 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:45:27.828 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:27.828 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:27.828 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:27.828 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:45:27.829 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:27.829 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:45:27.829 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:45:27.830 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:45:27.830 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:27.830 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:45:27Z" 2022-09-13T07:45:27.831 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:45:27.832 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:45:27Z" } 2022-09-13T07:45:27.833 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:45:27.834 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:27.834 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:27.835 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.831134_204.1 2022-09-13T07:45:32.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:32.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:32.837 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:45:32.888 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:45:32.888 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:45:32.918 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:32.913+0000 7f56e2cb0600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:32.919 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:32.914+0000 7f56e2cb0600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:32.930 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:45:32.930 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:45:32.931 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:45:32.931 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:45:32.931 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:45:32.931 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:45:32.932 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:32.932 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:32.932 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:45:32.932 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:32.933 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:45:32.933 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:32.933 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:32.933 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:45:32.934 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:32.934 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:45:32.934 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:32.935 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:45:32.935 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:32.935 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:32.936 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:32.936 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:32.937 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:45:32.937 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:32.937 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:32.938 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:45:32.938 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:32.938 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:32.938 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:32.939 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:45:32.939 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:32.939 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:32.939 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:32.940 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:32.940 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:45:32.940 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:32.941 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:32.941 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.854520_207.1", 2022-09-13T07:45:32.941 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:32.941 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:32.942 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:32.942 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T07:45:32.943 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:32.943 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:32.944 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:32.944 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:32.944 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:45:32.944 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:32.945 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:32.945 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:45:32.946 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:32.946 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:32.948 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:32.948 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:45:32.949 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:32.950 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:32.950 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:32.951 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:45:32.951 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:32.952 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:45:32.953 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:45:32.954 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:45:32.954 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:32.955 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:45:32Z" 2022-09-13T07:45:32.956 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:45:32.958 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.854520_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.854520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:45:32Z" } 2022-09-13T07:45:32.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:45:32.960 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:32.961 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:32.962 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.854520_207.1 2022-09-13T07:45:37.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:37.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:37.965 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:45:38.016 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:45:38.016 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:45:38.042 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:38.037+0000 7fa9fe00d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:38.043 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:38.038+0000 7fa9fe00d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:38.055 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:45:38.055 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:45:38.056 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:45:38.056 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:45:38.057 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:45:38.057 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:45:38.057 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:38.058 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:38.058 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:45:38.059 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:38.059 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:45:38.060 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:38.061 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:38.061 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:45:38.061 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:38.062 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:45:38.062 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:38.063 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:45:38.063 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:38.063 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:38.064 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:38.064 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:38.065 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:45:38.065 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:38.065 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:38.066 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:45:38.066 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:38.066 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:38.067 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:38.067 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:45:38.067 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:38.067 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:38.068 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:38.068 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:38.069 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:45:38.069 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:38.069 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:38.069 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.990623_217.1", 2022-09-13T07:45:38.070 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:38.070 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:38.070 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:38.071 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T07:45:38.071 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:38.071 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:38.071 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:38.071 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:38.072 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:45:38.072 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:38.072 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:38.073 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:45:38.073 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:38.073 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:38.073 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:38.074 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:45:38.074 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:38.074 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:38.074 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:38.075 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:45:38.075 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:38.076 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:45:38.076 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:45:38.076 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:45:38.076 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:38.077 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:45:38Z" 2022-09-13T07:45:38.077 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:45:38.079 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.990623_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.990623Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:45:38Z" } 2022-09-13T07:45:38.079 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:45:38.080 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:38.080 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:38.081 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.990623_217.1 2022-09-13T07:45:43.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:43.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:43.085 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:45:43.138 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:45:43.139 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:45:43.213 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:43.209+0000 7efc2c2f9600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:43.215 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:43.211+0000 7efc2c2f9600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:43.242 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:45:43.242 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:45:43.243 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:45:43.243 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:45:43.244 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:45:43.244 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:45:43.245 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:43.245 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:43.246 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:45:43.246 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:43.246 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:45:43.247 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:43.247 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:43.247 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:45:43.247 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:43.248 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:45:43.248 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:43.248 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:45:43.249 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:43.249 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:43.249 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:43.250 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:43.250 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:45:43.250 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:43.250 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:43.251 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:45:43.251 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:43.252 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:43.252 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:43.253 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:45:43.253 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:43.253 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:43.253 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:43.254 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:43.254 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:45:43.254 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:43.255 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:43.255 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.900410_211.1", 2022-09-13T07:45:43.255 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:43.255 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:43.256 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:43.256 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T07:45:43.256 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:43.257 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:43.257 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:43.258 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:43.258 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:45:43.259 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:43.259 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:43.259 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:45:43.259 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:43.260 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:43.260 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:43.260 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:45:43.261 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:43.261 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:43.262 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:43.262 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:45:43.262 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:43.263 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:45:43.263 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:45:43.263 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:45:43.263 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:43.263 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:45:43Z" 2022-09-13T07:45:43.264 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:45:43.265 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.900410_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.900410Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:45:43Z" } 2022-09-13T07:45:43.266 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:45:43.266 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:43.267 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:43.268 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.900410_211.1 2022-09-13T07:45:48.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:48.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:48.271 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:45:48.322 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:45:48.322 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:45:48.375 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:48.371+0000 7f65f2588600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:48.376 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:48.372+0000 7f65f2588600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:48.396 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:45:48.397 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:45:48.397 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:45:48.397 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:45:48.398 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:45:48.398 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:45:48.398 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:48.398 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:48.399 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:45:48.399 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:48.399 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:45:48.400 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:48.400 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:48.400 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:45:48.400 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:48.401 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:45:48.401 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:48.401 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:45:48.402 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:48.402 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:48.402 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:48.402 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:48.403 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:45:48.403 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:48.403 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:48.403 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:45:48.404 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:48.404 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:48.405 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:48.405 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:45:48.405 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:48.405 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:48.406 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:48.406 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:48.406 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:45:48.407 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:48.407 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:48.407 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:45:48.407 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:48.408 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:48.408 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:48.408 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:45:48.408 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:48.409 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:48.409 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:48.409 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:48.410 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:45:48.410 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:48.410 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:48.410 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:45:48.411 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:48.411 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:48.411 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:48.412 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:45:48.412 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:48.412 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:48.412 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:48.413 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:45:48.413 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:48.413 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:45:48.414 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:45:48.414 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:45:48.414 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:48.415 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:45:48Z" 2022-09-13T07:45:48.415 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:45:48.417 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:45:48Z" } 2022-09-13T07:45:48.418 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:45:48.418 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:48.419 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:48.419 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.831134_204.1 2022-09-13T07:45:53.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:53.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:53.422 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:45:53.472 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:45:53.473 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:45:53.503 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:53.499+0000 7f2b106b1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:53.504 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:53.500+0000 7f2b106b1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:53.524 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:45:53.525 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:45:53.525 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:45:53.525 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:45:53.525 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:45:53.526 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:45:53.526 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:53.526 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:53.527 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:45:53.527 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:53.527 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:45:53.527 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:53.528 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:53.528 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:45:53.528 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:53.529 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:45:53.529 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:53.529 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:45:53.529 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:53.530 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:53.530 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:53.530 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:53.530 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:45:53.531 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:53.531 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:53.531 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:45:53.532 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:53.532 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:53.532 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:53.532 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:45:53.533 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:53.533 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:53.534 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:53.534 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:53.535 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:45:53.535 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:53.536 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:53.536 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.070128_219.1", 2022-09-13T07:45:53.537 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:53.537 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:53.538 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:53.538 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T07:45:53.539 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:53.539 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:53.540 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:53.540 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:53.541 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:45:53.541 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:53.542 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:53.542 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:45:53.543 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:53.543 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:53.543 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:53.544 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:45:53.544 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:53.544 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:53.544 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:53.545 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:45:53.545 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:53.545 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:45:53.545 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:45:53.546 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:45:53.546 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:53.546 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:45:53Z" 2022-09-13T07:45:53.546 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:45:53.548 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.070128_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.070128Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:45:53Z" } 2022-09-13T07:45:53.548 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:45:53.549 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:53.549 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:53.550 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055099.070128_219.1 2022-09-13T07:45:58.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:58.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:45:58.552 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:45:58.603 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:45:58.603 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:45:58.651 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:58.648+0000 7f8b5d5d4600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:58.652 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:45:58.649+0000 7f8b5d5d4600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:45:58.675 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:45:58.675 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:45:58.676 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:45:58.677 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:45:58.677 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:45:58.677 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:45:58.677 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:58.678 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:58.678 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:45:58.678 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:58.678 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:45:58.679 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:58.679 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:58.680 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:45:58.680 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:58.680 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:45:58.680 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:58.681 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:45:58.681 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:58.681 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:58.681 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:58.681 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:58.682 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:45:58.682 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:58.682 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:58.683 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:45:58.683 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:58.683 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:58.683 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:58.684 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:45:58.684 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:58.684 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:58.684 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:58.685 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:58.685 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:45:58.685 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:58.685 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:58.686 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.900410_211.1", 2022-09-13T07:45:58.686 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:58.686 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:58.687 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:58.687 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T07:45:58.688 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:58.688 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:58.688 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:58.688 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:45:58.689 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:45:58.689 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:45:58.689 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:45:58.690 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:45:58.690 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:45:58.690 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:45:58.690 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:45:58.691 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:45:58.691 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:45:58.691 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:58.691 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:45:58.692 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:45:58.692 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:58.692 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:45:58.692 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:45:58.692 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:45:58.693 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:45:58.693 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:45:58Z" 2022-09-13T07:45:58.693 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:45:58.694 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.900410_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.900410Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:45:58Z" } 2022-09-13T07:45:58.695 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:45:58.696 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:58.696 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:45:58.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.900410_211.1 2022-09-13T07:46:03.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:03.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:03.700 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:46:03.755 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:46:03.756 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:46:03.797 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:03.793+0000 7fa13cad2600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:03.799 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:03.794+0000 7fa13cad2600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:03.814 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:46:03.814 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:46:03.815 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:46:03.815 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:46:03.815 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:46:03.816 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:46:03.816 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:03.816 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:03.817 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:46:03.817 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:03.817 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:46:03.818 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:03.818 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:03.819 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:46:03.819 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:03.819 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:46:03.820 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:03.820 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:46:03.820 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:03.820 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:03.821 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:03.821 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:03.821 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:46:03.821 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:03.822 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:03.822 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:46:03.822 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:03.822 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:03.823 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:03.823 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:46:03.824 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:03.824 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:03.824 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:03.825 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:03.825 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:46:03.825 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:03.826 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:03.826 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.907374_213.1", 2022-09-13T07:46:03.826 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:03.827 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:03.827 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:03.827 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.907374Z", 2022-09-13T07:46:03.828 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:03.828 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:03.829 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:03.829 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:03.829 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:46:03.829 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:03.830 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:03.830 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:46:03.830 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:03.831 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:03.831 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:03.832 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:46:03.832 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:03.832 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:03.832 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:03.833 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:46:03.833 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:03.833 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:46:03.835 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:46:03.836 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:46:03.836 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:03.836 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:46:03Z" 2022-09-13T07:46:03.837 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:46:03.841 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.907374_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.907374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:46:03Z" } 2022-09-13T07:46:03.842 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.907374_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.907374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:46:03.844 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:03.845 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.907374_213.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:03.845 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.907374_213.1 2022-09-13T07:46:08.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:08.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:08.849 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:46:08.900 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:46:08.901 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:46:08.929 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:08.925+0000 7f1b0a447600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:08.930 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:08.926+0000 7f1b0a447600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:08.942 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:46:08.942 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:46:08.942 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:46:08.943 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:46:08.943 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:46:08.943 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:46:08.944 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:08.944 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:08.944 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:46:08.944 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:08.945 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:46:08.945 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:08.945 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:08.946 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:46:08.946 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:08.946 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:46:08.946 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:08.947 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:46:08.947 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:08.947 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:08.947 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:08.948 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:08.948 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:46:08.948 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:08.949 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:08.949 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:46:08.949 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:08.949 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:08.950 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:08.950 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:46:08.950 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:08.950 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:08.951 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:08.951 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:08.951 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:46:08.951 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:08.952 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:08.952 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T07:46:08.952 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:08.952 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:08.953 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:08.953 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T07:46:08.953 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:08.953 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:08.954 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:08.954 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:08.954 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:46:08.954 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:08.955 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:08.955 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:46:08.955 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:08.955 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:08.956 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:08.956 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:46:08.956 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:08.956 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:08.957 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:08.957 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:46:08.957 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:08.957 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:46:08.958 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:46:08.958 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:46:08.958 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:08.958 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:46:08Z" 2022-09-13T07:46:08.959 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:46:08.960 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:46:08Z" } 2022-09-13T07:46:08.961 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:46:08.962 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:08.962 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:08.963 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.949151_215.1 2022-09-13T07:46:13.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:13.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:13.965 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:46:14.018 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:46:14.018 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:46:14.076 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:14.072+0000 7fb3ab8e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:14.090 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:14.075+0000 7fb3ab8e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:14.157 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:46:14.159 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:46:14.159 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:46:14.159 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:46:14.160 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:46:14.160 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:46:14.160 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:14.160 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:14.161 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:46:14.161 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:14.161 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:46:14.162 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:14.162 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:14.162 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:46:14.162 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:14.163 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:46:14.163 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:14.163 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:46:14.163 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:14.164 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:14.164 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:14.165 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:14.165 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:46:14.166 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:14.166 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:14.167 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:46:14.167 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:14.169 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:14.170 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:14.170 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:46:14.171 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:14.171 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:14.171 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:14.172 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:14.172 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:46:14.173 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:14.173 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:14.173 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T07:46:14.174 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:14.174 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:14.175 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:14.175 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T07:46:14.176 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:14.176 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:14.176 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:14.177 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:14.177 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:46:14.178 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:14.178 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:14.179 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:46:14.179 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:14.180 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:14.180 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:14.180 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:46:14.180 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:14.181 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:14.181 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:14.181 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:46:14.181 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:14.182 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:46:14.182 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:46:14.182 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:46:14.182 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:14.183 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:46:14Z" 2022-09-13T07:46:14.183 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:46:14.183 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:46:14Z" } 2022-09-13T07:46:14.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:46:14.184 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:14.185 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:14.186 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.832045_205.1 2022-09-13T07:46:19.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:19.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:19.190 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:46:19.240 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:46:19.241 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:46:19.266 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:19.262+0000 7f354aa51600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:19.267 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:19.263+0000 7f354aa51600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:19.279 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:46:19.279 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:46:19.280 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:46:19.280 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:46:19.280 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:46:19.281 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:46:19.281 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:19.281 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:19.282 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:46:19.283 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:19.283 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:46:19.284 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:19.284 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:19.285 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:46:19.285 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:19.285 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:46:19.286 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:19.287 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:46:19.287 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:19.287 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:19.288 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:19.288 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:19.289 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:46:19.289 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:19.290 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:19.290 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:46:19.291 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:19.291 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:19.292 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:19.292 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:46:19.293 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:19.293 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:19.294 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:19.294 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:19.295 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:46:19.295 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:19.296 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:19.296 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:46:19.297 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:19.297 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:19.298 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:19.298 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:46:19.299 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:19.299 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:19.300 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:19.300 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:19.301 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:46:19.301 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:19.302 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:19.302 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:46:19.303 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:19.303 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:19.304 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:19.304 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:46:19.305 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:19.305 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:19.306 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:19.306 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:46:19.307 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:19.307 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:46:19.308 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:46:19.308 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:46:19.309 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:19.309 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:46:19Z" 2022-09-13T07:46:19.310 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:46:19.312 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:46:19Z" } 2022-09-13T07:46:19.313 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:46:19.314 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:19.315 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:19.316 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.831134_204.1 2022-09-13T07:46:24.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:24.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:24.319 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:46:24.370 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:46:24.371 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:46:24.407 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:24.403+0000 7f1666650600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:24.408 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:24.404+0000 7f1666650600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:24.425 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:46:24.426 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:46:24.426 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:46:24.426 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:46:24.427 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:46:24.427 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:46:24.427 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:24.427 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:24.428 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:46:24.428 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:24.428 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:46:24.428 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:24.429 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:24.429 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:46:24.429 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:24.430 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:46:24.430 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:24.431 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:46:24.431 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:24.431 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:24.431 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:24.432 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:24.432 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:46:24.432 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:24.433 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:24.433 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:46:24.434 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:24.434 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:24.434 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:24.434 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:46:24.435 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:24.435 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:24.435 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:24.435 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:24.436 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:46:24.436 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:24.436 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:24.436 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.900410_211.1", 2022-09-13T07:46:24.437 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:24.437 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:24.437 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:24.437 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T07:46:24.438 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:24.438 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:24.438 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:24.438 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:24.439 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:46:24.439 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:24.439 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:24.440 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:46:24.440 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:24.440 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:24.440 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:24.441 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:46:24.441 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:24.441 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:24.442 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:24.442 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:46:24.443 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:24.443 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:46:24.443 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:46:24.443 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:46:24.444 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:24.444 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:46:24Z" 2022-09-13T07:46:24.445 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:46:24.446 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.900410_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.900410Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:46:24Z" } 2022-09-13T07:46:24.447 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:46:24.447 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:24.448 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:24.448 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.900410_211.1 2022-09-13T07:46:29.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:29.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:29.451 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:46:29.505 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:46:29.506 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:46:29.546 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:29.541+0000 7f7212b3d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:29.546 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:29.542+0000 7f7212b3d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:29.569 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:46:29.569 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:46:29.570 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:46:29.570 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:46:29.570 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:46:29.570 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:46:29.571 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:29.571 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:29.571 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:46:29.572 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:29.572 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:46:29.572 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:29.572 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:29.573 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:46:29.573 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:29.573 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:46:29.573 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:29.574 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:46:29.574 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:29.574 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:29.575 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:29.575 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:29.575 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:46:29.576 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:29.576 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:29.576 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:46:29.576 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:29.577 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:29.577 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:29.577 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:46:29.578 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:29.578 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:29.578 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:29.579 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:29.579 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:46:29.579 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:29.580 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:29.580 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:46:29.580 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:29.580 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:29.581 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:29.581 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:46:29.581 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:29.581 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:29.582 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:29.582 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:29.582 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:46:29.582 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:29.583 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:29.583 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:46:29.583 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:29.584 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:29.584 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:29.584 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:46:29.584 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:29.585 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:29.585 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:29.585 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:46:29.585 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:29.586 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:46:29.586 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:46:29.586 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:46:29.587 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:29.587 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:46:29Z" 2022-09-13T07:46:29.587 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:46:29.589 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:46:29Z" } 2022-09-13T07:46:29.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:46:29.590 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:29.590 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:29.591 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.831134_204.1 2022-09-13T07:46:34.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:34.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:34.593 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:46:34.667 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:46:34.668 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:46:34.760 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:34.756+0000 7f87fbf5c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:34.760 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:34.757+0000 7f87fbf5c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:34.780 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:46:34.781 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:46:34.781 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:46:34.781 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:46:34.781 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:46:34.782 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:46:34.782 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:34.782 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:34.782 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:46:34.783 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:34.783 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:46:34.783 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:34.783 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:34.784 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:46:34.784 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:34.784 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:46:34.785 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:34.785 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:46:34.785 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:34.786 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:34.786 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:34.787 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:34.787 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:46:34.787 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:34.787 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:34.788 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:46:34.788 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:34.788 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:34.789 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:34.789 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:46:34.789 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:34.789 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:34.790 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:34.790 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:34.790 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:46:34.790 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:34.791 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:34.791 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.858000_209.1", 2022-09-13T07:46:34.792 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:34.793 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:34.793 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:34.794 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T07:46:34.795 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:34.795 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:34.795 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:34.795 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:34.796 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:46:34.796 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:34.797 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:34.797 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:46:34.797 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:34.798 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:34.798 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:34.798 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:46:34.798 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:34.799 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:34.799 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:34.799 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:46:34.799 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:34.800 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:46:34.800 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:46:34.800 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:46:34.801 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:34.801 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:46:34Z" 2022-09-13T07:46:34.801 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:46:34.803 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.858000_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.858000Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:46:34Z" } 2022-09-13T07:46:34.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:46:34.804 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:34.805 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:34.805 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.858000_209.1 2022-09-13T07:46:39.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:39.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:39.808 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:46:39.859 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:46:39.860 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:46:39.895 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:39.892+0000 7fa58782b600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:39.896 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:39.893+0000 7fa58782b600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:39.908 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:46:39.909 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:46:39.909 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:46:39.909 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:46:39.909 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:46:39.910 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:46:39.910 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:39.911 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:39.911 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:46:39.911 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:39.911 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:46:39.912 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:39.912 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:39.912 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:46:39.912 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:39.913 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:46:39.913 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:39.913 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:46:39.913 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:39.914 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:39.914 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:39.914 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:39.914 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:46:39.915 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:39.915 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:39.915 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:46:39.915 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:39.916 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:39.916 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:39.916 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:46:39.916 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:39.917 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:39.917 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:39.917 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:39.917 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:46:39.918 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:39.918 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:39.918 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T07:46:39.918 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:39.919 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:39.919 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:39.920 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T07:46:39.920 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:39.920 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:39.920 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:39.921 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:39.921 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:46:39.921 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:39.921 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:39.922 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:46:39.922 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:39.922 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:39.922 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:39.923 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:46:39.923 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:39.923 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:39.923 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:39.924 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:46:39.924 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:39.924 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:46:39.925 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:46:39.925 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:46:39.925 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:39.925 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:46:39Z" 2022-09-13T07:46:39.926 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:46:39.927 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:46:39Z" } 2022-09-13T07:46:39.928 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:46:39.928 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:39.929 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:39.929 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.949151_215.1 2022-09-13T07:46:44.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:44.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:44.932 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:46:44.983 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:46:44.983 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:46:45.011 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:45.008+0000 7fdaf9d4e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:45.012 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:45.009+0000 7fdaf9d4e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:45.024 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:46:45.025 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:46:45.025 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:46:45.025 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:46:45.026 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:46:45.026 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:46:45.026 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:45.026 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:45.027 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:46:45.027 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:45.027 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:46:45.027 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:45.028 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:45.028 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:46:45.028 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:45.029 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:46:45.030 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:45.030 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:46:45.030 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:45.031 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:45.031 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:45.031 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:45.032 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:46:45.032 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:45.032 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:45.032 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:46:45.033 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:45.033 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:45.033 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:45.033 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:46:45.034 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:45.034 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:45.034 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:45.034 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:45.035 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:46:45.035 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:45.035 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:45.035 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.858000_209.1", 2022-09-13T07:46:45.036 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:45.036 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:45.036 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:45.036 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T07:46:45.037 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:45.037 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:45.037 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:45.037 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:45.038 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:46:45.038 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:45.038 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:45.038 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:46:45.039 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:45.039 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:45.039 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:45.039 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:46:45.040 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:45.040 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:45.040 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:45.040 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:46:45.040 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:45.041 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:46:45.041 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:46:45.041 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:46:45.041 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:45.042 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:46:45Z" 2022-09-13T07:46:45.042 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:46:45.043 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.858000_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.858000Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:46:45Z" } 2022-09-13T07:46:45.044 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:46:45.044 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:45.045 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:45.045 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.858000_209.1 2022-09-13T07:46:50.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:50.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:50.048 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:46:50.099 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:46:50.100 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:46:50.151 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:50.147+0000 7f7ebfe4a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:50.152 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:50.148+0000 7f7ebfe4a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:50.169 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:46:50.170 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:46:50.170 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:46:50.171 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:46:50.171 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:46:50.172 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:46:50.172 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:50.173 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:50.173 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:46:50.173 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:50.174 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:46:50.174 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:50.175 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:50.175 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:46:50.176 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:50.176 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:46:50.177 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:50.177 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:46:50.178 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:50.178 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:50.179 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:50.179 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:50.180 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:46:50.180 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:50.181 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:50.181 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:46:50.182 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:50.183 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:50.183 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:50.184 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:46:50.184 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:50.184 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:50.185 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:50.185 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:50.186 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:46:50.186 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:50.187 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:50.187 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.911619_214.1", 2022-09-13T07:46:50.188 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:50.188 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:50.189 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:50.189 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T07:46:50.190 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:50.190 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:50.191 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:50.191 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:50.191 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:46:50.192 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:50.192 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:50.193 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:46:50.193 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:50.194 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:50.194 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:50.195 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:46:50.195 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:50.196 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:50.196 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:50.197 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:46:50.197 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:50.198 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:46:50.198 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:46:50.198 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:46:50.199 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:50.199 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:46:50Z" 2022-09-13T07:46:50.200 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:46:50.201 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.911619_214.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.911619Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:46:50Z" } 2022-09-13T07:46:50.202 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:46:50.203 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:50.204 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:50.205 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.911619_214.1 2022-09-13T07:46:55.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:55.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:46:55.208 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:46:55.265 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:46:55.266 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:46:55.297 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:55.292+0000 7f8d752dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:55.298 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:46:55.294+0000 7f8d752dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:46:55.322 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:46:55.322 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:46:55.323 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:46:55.323 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:46:55.324 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:46:55.325 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:46:55.325 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:55.326 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:55.326 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:46:55.326 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:55.327 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:46:55.327 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:55.327 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:55.328 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:46:55.328 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:55.328 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:46:55.328 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:55.329 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:46:55.329 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:55.329 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:55.329 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:55.330 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:55.330 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:46:55.330 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:55.330 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:55.331 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:46:55.331 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:55.331 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:55.332 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:55.332 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:46:55.332 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:55.332 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:55.333 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:55.333 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:55.333 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:46:55.333 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:55.334 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:55.334 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.142561_227.1", 2022-09-13T07:46:55.334 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:55.334 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:55.335 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:55.335 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T07:46:55.335 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:55.336 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:55.336 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:55.336 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:46:55.336 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:46:55.337 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:46:55.337 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:46:55.337 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:46:55.337 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:46:55.338 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:46:55.338 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:46:55.338 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:46:55.339 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:46:55.339 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:55.339 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:46:55.339 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:46:55.340 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:55.340 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:46:55.340 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:46:55.340 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:46:55.341 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:46:55.341 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:46:55Z" 2022-09-13T07:46:55.341 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:46:55.343 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.142561_227.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.142561Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:46:55Z" } 2022-09-13T07:46:55.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:46:55.344 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:55.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:46:55.345 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055099.142561_227.1 2022-09-13T07:47:00.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:00.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:00.348 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:47:00.422 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:47:00.422 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:47:00.516 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:00.512+0000 7f7882f8c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:00.518 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:00.513+0000 7f7882f8c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:00.547 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:47:00.547 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:47:00.548 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:47:00.548 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:47:00.549 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:47:00.549 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:47:00.549 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:00.550 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:00.550 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:47:00.550 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:00.551 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:47:00.551 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:00.552 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:00.552 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:47:00.552 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:00.552 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:47:00.553 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:00.553 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:47:00.553 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:00.554 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:00.554 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:00.554 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:00.555 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:47:00.555 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:00.555 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:00.556 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:47:00.556 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:00.556 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:00.557 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:00.557 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:47:00.558 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:00.558 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:00.558 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:00.559 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:00.559 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:47:00.559 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:00.560 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:00.560 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:47:00.560 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:00.561 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:00.561 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:00.561 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:47:00.562 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:00.562 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:00.562 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:00.563 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:00.563 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:47:00.563 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:00.563 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:00.564 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:47:00.564 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:00.564 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:00.564 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:00.565 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:47:00.565 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:00.565 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:00.566 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:00.566 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:47:00.566 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:00.567 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:47:00.568 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:47:00.568 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:47:00.568 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:00.569 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:47:00Z" 2022-09-13T07:47:00.569 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:47:00.570 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:47:00Z" } 2022-09-13T07:47:00.571 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:47:00.572 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:00.572 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:00.573 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.834293_206.1 2022-09-13T07:47:05.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:05.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:05.577 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:47:05.628 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:47:05.628 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:47:05.667 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:05.660+0000 7f8a185a7600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:05.668 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:05.661+0000 7f8a185a7600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:05.684 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:47:05.684 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:47:05.685 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:47:05.686 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:47:05.686 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:47:05.687 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:47:05.687 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:05.688 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:05.689 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:47:05.689 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:05.690 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:47:05.690 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:05.691 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:05.691 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:47:05.692 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:05.693 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:47:05.693 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:05.694 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:47:05.695 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:05.695 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:05.696 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:05.696 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:05.697 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:47:05.697 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:05.698 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:05.698 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:47:05.699 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:05.700 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:05.700 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:05.701 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:47:05.701 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:05.702 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:05.702 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:05.703 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:05.703 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:47:05.704 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:05.704 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:05.705 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:47:05.705 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:05.706 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:05.707 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:05.707 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:47:05.708 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:05.708 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:05.709 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:05.709 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:05.710 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:47:05.710 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:05.711 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:05.711 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:47:05.712 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:05.712 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:05.713 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:05.713 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:47:05.714 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:05.714 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:05.715 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:05.715 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:47:05.716 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:05.716 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:47:05.717 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:47:05.717 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:47:05.718 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:05.718 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:47:05Z" 2022-09-13T07:47:05.719 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:47:05.720 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:47:05Z" } 2022-09-13T07:47:05.721 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:47:05.722 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:05.723 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:05.724 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.834293_206.1 2022-09-13T07:47:10.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:10.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:10.728 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:47:10.787 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:47:10.788 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:47:10.826 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:10.822+0000 7f383f8e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:10.828 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:10.823+0000 7f383f8e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:10.850 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:47:10.850 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:47:10.851 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:47:10.851 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:47:10.851 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:47:10.851 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:47:10.852 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:10.852 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:10.852 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:47:10.852 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:10.853 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:47:10.853 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:10.853 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:10.853 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:47:10.854 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:10.854 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:47:10.855 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:10.855 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:47:10.855 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:10.856 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:10.856 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:10.856 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:10.856 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:47:10.857 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:10.857 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:10.857 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:47:10.857 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:10.858 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:10.858 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:10.858 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:47:10.858 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:10.859 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:10.859 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:10.859 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:10.859 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:47:10.860 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:10.860 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:10.860 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.858000_209.1", 2022-09-13T07:47:10.860 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:10.861 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:10.861 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:10.862 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T07:47:10.862 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:10.862 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:10.863 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:10.863 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:10.863 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:47:10.863 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:10.864 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:10.864 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:47:10.864 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:10.865 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:10.866 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:10.866 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:47:10.867 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:10.867 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:10.868 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:10.868 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:47:10.869 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:10.869 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:47:10.870 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:47:10.871 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:47:10.871 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:10.872 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:47:10Z" 2022-09-13T07:47:10.872 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:47:10.874 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.858000_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.858000Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:47:10Z" } 2022-09-13T07:47:10.876 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:47:10.877 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:10.877 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:10.878 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.858000_209.1 2022-09-13T07:47:15.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:15.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:15.881 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:47:15.938 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:47:15.939 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:47:15.969 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:15.966+0000 7ff695ffb600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:15.970 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:15.967+0000 7ff695ffb600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:15.984 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:47:15.985 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:47:15.985 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:47:15.986 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:47:15.986 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:47:15.987 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:47:15.987 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:15.988 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:15.988 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:47:15.989 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:15.989 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:47:15.990 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:15.990 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:15.991 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:47:15.991 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:15.992 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:47:15.993 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:15.993 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:47:15.994 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:15.994 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:15.995 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:15.995 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:15.996 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:47:15.996 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:15.997 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:15.997 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:47:15.997 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:15.998 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:15.998 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:15.999 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:47:15.999 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:16.000 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:16.000 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:16.001 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:16.001 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:47:16.002 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:16.002 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:16.003 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.900410_211.1", 2022-09-13T07:47:16.003 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:16.004 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:16.004 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:16.005 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T07:47:16.005 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:16.006 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:16.006 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:16.007 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:16.007 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:47:16.008 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:16.008 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:16.009 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:47:16.009 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:16.010 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:16.011 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:16.011 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:47:16.012 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:16.012 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:16.013 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:16.013 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:47:16.014 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:16.014 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:47:16.015 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:47:16.015 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:47:16.016 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:16.016 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:47:15Z" 2022-09-13T07:47:16.017 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:47:16.019 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.900410_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.900410Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:47:15Z" } 2022-09-13T07:47:16.020 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:47:16.020 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:16.021 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:16.022 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.900410_211.1 2022-09-13T07:47:21.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:21.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:21.024 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:47:21.095 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:47:21.096 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:47:21.125 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:21.122+0000 7f23251aa600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:21.126 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:21.123+0000 7f23251aa600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:21.145 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:47:21.146 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:47:21.146 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:47:21.146 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:47:21.147 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:47:21.147 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:47:21.147 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:21.148 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:21.148 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:47:21.148 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:21.149 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:47:21.149 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:21.149 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:21.150 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:47:21.150 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:21.150 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:47:21.151 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:21.151 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:47:21.151 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:21.151 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:21.152 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:21.152 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:21.152 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:47:21.153 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:21.153 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:21.153 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:47:21.153 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:21.154 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:21.154 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:21.154 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:47:21.154 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:21.155 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:21.155 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:21.155 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:21.155 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:47:21.156 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:21.156 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:21.156 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:47:21.157 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:21.157 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:21.157 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:21.158 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:47:21.158 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:21.158 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:21.158 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:21.159 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:21.159 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:47:21.159 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:21.159 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:21.160 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:47:21.160 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:21.160 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:21.160 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:47:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:21.161 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:21.162 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:47:21.162 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:21.162 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:47:21.162 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:47:21.163 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:47:21.163 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:21.163 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:47:21Z" 2022-09-13T07:47:21.163 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:47:21.165 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:47:21Z" } 2022-09-13T07:47:21.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:47:21.166 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:21.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:21.168 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.831134_204.1 2022-09-13T07:47:26.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:26.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:26.170 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:47:26.219 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:47:26.219 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:47:26.249 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:26.245+0000 7fa3b4a6b600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:26.250 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:26.247+0000 7fa3b4a6b600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:26.262 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:47:26.262 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:47:26.263 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:47:26.263 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:47:26.263 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:47:26.264 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:47:26.264 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:26.264 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:26.264 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:47:26.265 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:26.265 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:47:26.265 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:26.265 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:26.266 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:47:26.266 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:26.266 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:47:26.267 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:26.267 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:47:26.267 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:26.267 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:26.268 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:26.268 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:26.268 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:47:26.268 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:26.269 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:26.269 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:47:26.269 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:26.269 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:26.270 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:26.270 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:47:26.270 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:26.271 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:26.271 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:26.271 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:26.271 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:47:26.272 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:26.272 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:26.272 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.861945_210.1", 2022-09-13T07:47:26.272 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:26.272 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:26.273 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:26.273 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T07:47:26.273 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:26.273 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:26.274 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:26.274 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:26.274 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:47:26.275 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:26.275 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:26.275 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:47:26.276 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:26.276 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:26.276 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:26.277 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:47:26.277 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:26.278 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:26.278 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:26.279 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:47:26.279 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:26.279 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:47:26.279 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:47:26.280 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:47:26.280 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:26.280 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:47:26Z" 2022-09-13T07:47:26.280 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:47:26.282 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.861945_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.861945Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:47:26Z" } 2022-09-13T07:47:26.282 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:47:26.283 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:26.283 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:26.284 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.861945_210.1 2022-09-13T07:47:31.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:31.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:31.288 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:47:31.339 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:47:31.339 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:47:31.392 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:31.387+0000 7f1ff068d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:31.393 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:31.388+0000 7f1ff068d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:31.406 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:47:31.407 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:47:31.408 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:47:31.409 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:47:31.410 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:47:31.411 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:47:31.411 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:31.412 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:31.412 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:47:31.413 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:31.414 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:47:31.414 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:31.415 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:31.415 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:47:31.416 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:31.416 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:47:31.416 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:31.417 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:47:31.417 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:31.418 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:31.418 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:31.418 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:31.419 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:47:31.419 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:31.420 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:31.420 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:47:31.421 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:31.421 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:31.422 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:31.422 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:47:31.423 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:31.423 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:31.424 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:31.424 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:31.425 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:47:31.425 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:31.426 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:31.426 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.900410_211.1", 2022-09-13T07:47:31.427 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:31.427 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:31.427 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:31.428 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T07:47:31.428 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:31.428 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:31.429 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:31.429 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:31.429 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:47:31.430 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:31.430 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:31.430 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:47:31.430 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:31.431 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:31.431 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:31.431 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:47:31.432 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:31.432 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:31.432 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:31.432 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:47:31.433 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:31.433 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:47:31.433 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:47:31.434 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:47:31.434 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:31.434 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:47:31Z" 2022-09-13T07:47:31.434 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:47:31.436 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.900410_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.900410Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:47:31Z" } 2022-09-13T07:47:31.437 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:47:31.437 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:31.438 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:31.439 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.900410_211.1 2022-09-13T07:47:36.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:36.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:36.441 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:47:36.497 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:47:36.498 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:47:36.527 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:36.522+0000 7f03318e3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:36.529 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:36.523+0000 7f03318e3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:36.546 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:47:36.547 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:47:36.547 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:47:36.548 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:47:36.548 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:47:36.549 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:47:36.549 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:36.549 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:36.549 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:47:36.550 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:36.550 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:47:36.550 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:36.551 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:36.551 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:47:36.551 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:36.551 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:47:36.552 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:36.552 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:47:36.552 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:36.552 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:36.553 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:36.553 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:36.553 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:47:36.553 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:36.554 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:36.554 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:47:36.554 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:36.556 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:36.557 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:36.557 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:47:36.557 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:36.557 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:36.558 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:36.558 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:36.558 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:47:36.559 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:36.559 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:36.559 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.186446_230.1", 2022-09-13T07:47:36.560 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:36.560 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:36.560 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:36.560 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.186446Z", 2022-09-13T07:47:36.561 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:36.561 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:36.561 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:36.561 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:36.562 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:47:36.562 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:36.562 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:36.562 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:47:36.563 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:36.563 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:36.563 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:36.564 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:47:36.564 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:36.564 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:36.564 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:36.565 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:47:36.565 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:36.565 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:47:36.565 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:47:36.566 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:47:36.566 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:36.567 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:47:36Z" 2022-09-13T07:47:36.567 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:47:36.569 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.186446_230.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.186446Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:47:36Z" } 2022-09-13T07:47:36.569 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.186446_230.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.186446Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:47:36.570 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:36.571 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.186446_230.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:36.571 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055099.186446_230.1 2022-09-13T07:47:41.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:41.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:41.574 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:47:41.631 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:47:41.632 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:47:41.664 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:41.660+0000 7f0cc516c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:41.665 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:41.661+0000 7f0cc516c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:41.680 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:47:41.681 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:47:41.681 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:47:41.681 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:47:41.681 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:47:41.682 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:47:41.682 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:41.682 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:41.683 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:47:41.683 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:41.683 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:47:41.683 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:41.684 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:41.684 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:47:41.684 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:41.684 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:47:41.685 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:41.685 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:47:41.685 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:41.685 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:41.686 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:41.686 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:41.686 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:47:41.686 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:41.687 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:41.687 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:47:41.687 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:41.688 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:41.688 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:41.688 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:47:41.689 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:41.689 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:41.689 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:41.689 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:41.689 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:47:41.690 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:41.690 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:41.690 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.220845_233.1", 2022-09-13T07:47:41.691 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:41.691 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:41.691 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:41.691 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T07:47:41.692 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:41.692 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:41.692 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:41.692 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:41.693 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:47:41.693 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:41.693 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:41.694 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:47:41.694 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:41.694 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:41.694 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:41.695 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:47:41.695 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:41.695 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:41.695 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:41.696 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:47:41.696 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:41.696 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:47:41.696 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:47:41.697 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:47:41.697 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:41.697 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:47:41Z" 2022-09-13T07:47:41.697 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:47:41.698 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.220845_233.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.220845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:47:41Z" } 2022-09-13T07:47:41.699 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:47:41.700 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:41.700 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:41.701 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055099.220845_233.1 2022-09-13T07:47:46.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:46.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:46.703 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:47:46.794 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:47:46.795 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:47:46.844 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:46.840+0000 7fe618480600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:46.845 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:46.841+0000 7fe618480600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:46.862 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:47:46.863 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:47:46.863 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:47:46.864 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:47:46.864 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:47:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:47:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:47:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:47:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:47:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:47:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:47:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:46.870 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:46.870 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:47:46.870 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:47:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:47:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:46.874 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:47:46.874 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:46.874 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.907374_213.1", 2022-09-13T07:47:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.907374Z", 2022-09-13T07:47:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:47:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:46.878 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:46.878 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:47:46.878 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:47:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:47:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:47:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:47:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:47:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:47:46Z" 2022-09-13T07:47:46.882 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:47:46.884 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.907374_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.907374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:47:46Z" } 2022-09-13T07:47:46.884 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.907374_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.907374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:47:46.885 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:46.886 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.907374_213.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:46.887 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.907374_213.1 2022-09-13T07:47:51.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:51.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:51.892 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:47:51.943 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:47:51.943 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:47:51.991 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:51.986+0000 7fea2d1a4600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:51.991 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:51.987+0000 7fea2d1a4600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:52.014 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:47:52.015 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:47:52.015 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:47:52.016 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:47:52.016 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:47:52.016 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:47:52.017 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:52.017 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:52.018 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:47:52.018 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:52.019 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:47:52.019 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:52.020 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:52.020 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:47:52.020 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:52.021 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:47:52.021 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:52.022 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:47:52.022 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:52.023 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:52.023 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:52.024 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:52.024 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:47:52.024 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:52.025 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:52.025 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:47:52.026 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:52.026 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:52.027 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:52.027 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:47:52.028 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:52.028 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:52.029 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:52.029 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:52.030 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:47:52.031 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:52.031 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:52.032 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.990623_217.1", 2022-09-13T07:47:52.032 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:52.033 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:52.033 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:52.034 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T07:47:52.034 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:52.035 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:52.035 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:52.036 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:52.036 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:47:52.037 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:52.037 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:52.038 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:47:52.038 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:52.039 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:52.040 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:52.040 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:47:52.040 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:52.042 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:52.042 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:52.043 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:47:52.043 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:52.043 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:47:52.043 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:47:52.044 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:47:52.044 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:52.044 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:47:51Z" 2022-09-13T07:47:52.045 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:47:52.046 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.990623_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.990623Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:47:51Z" } 2022-09-13T07:47:52.047 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:47:52.048 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:52.048 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:52.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.990623_217.1 2022-09-13T07:47:57.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:57.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:47:57.052 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:47:57.102 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:47:57.103 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:47:57.130 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:57.127+0000 7f0a5a096600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:57.131 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:47:57.128+0000 7f0a5a096600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:47:57.143 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:47:57.144 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:47:57.144 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:47:57.144 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:47:57.145 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:47:57.145 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:47:57.145 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:57.145 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:57.146 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:47:57.146 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:57.146 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:47:57.146 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:57.147 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:57.147 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:47:57.147 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:57.147 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:47:57.148 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:57.148 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:47:57.148 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:57.148 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:57.149 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:57.149 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:57.149 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:47:57.149 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:57.149 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:57.150 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:47:57.150 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:57.151 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:57.151 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:57.151 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:47:57.151 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:57.152 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:57.152 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:57.153 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:57.153 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:47:57.154 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:57.154 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:57.154 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:47:57.155 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:57.155 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:57.155 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:57.156 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:47:57.157 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:57.157 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:57.158 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:57.158 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:47:57.159 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:47:57.159 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:47:57.159 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:47:57.160 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:47:57.160 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:47:57.161 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:47:57.161 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:47:57.161 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:47:57.162 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:47:57.162 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:57.162 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:47:57.162 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:47:57.163 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:57.163 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:47:57.163 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:47:57.164 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:47:57.164 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:47:57.164 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:47:57Z" 2022-09-13T07:47:57.164 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:47:57.166 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:47:57Z" } 2022-09-13T07:47:57.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:47:57.167 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:57.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:47:57.168 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.831134_204.1 2022-09-13T07:48:02.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:02.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:02.172 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:48:02.230 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:48:02.231 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:48:02.265 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:02.261+0000 7f421a9ef600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:02.265 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:02.262+0000 7f421a9ef600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:02.279 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:48:02.280 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:48:02.280 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:48:02.281 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:48:02.281 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:48:02.282 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:48:02.282 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:02.283 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:02.283 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:48:02.284 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:02.285 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:48:02.285 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:02.286 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:02.287 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:48:02.287 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:02.288 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:48:02.288 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:02.290 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:48:02.290 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:02.290 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:02.291 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:02.291 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:02.292 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:48:02.292 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:02.293 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:02.293 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:48:02.294 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:02.294 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:02.294 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:02.295 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:48:02.295 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:02.295 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:02.296 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:02.296 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:02.296 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:48:02.297 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:02.297 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:02.297 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.854520_207.1", 2022-09-13T07:48:02.298 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:02.298 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:02.298 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:02.299 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T07:48:02.299 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:02.300 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:02.301 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:02.301 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:02.301 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:48:02.302 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:02.302 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:02.302 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:48:02.302 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:02.303 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:02.303 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:02.303 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:48:02.303 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:02.304 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:02.304 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:02.304 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:48:02.304 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:02.305 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:48:02.305 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:48:02.305 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:48:02.305 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:02.306 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:48:02Z" 2022-09-13T07:48:02.306 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:48:02.307 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.854520_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.854520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:48:02Z" } 2022-09-13T07:48:02.308 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:48:02.308 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:02.309 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:02.310 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.854520_207.1 2022-09-13T07:48:07.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:07.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:07.314 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:48:07.371 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:48:07.372 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:48:07.395 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:07.390+0000 7f2eb7923600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:07.396 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:07.391+0000 7f2eb7923600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:07.411 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:48:07.412 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:48:07.412 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:48:07.412 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:48:07.412 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:48:07.413 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:48:07.413 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:07.413 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:07.413 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:48:07.414 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:07.414 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:48:07.414 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:07.414 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:07.415 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:48:07.415 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:07.415 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:48:07.415 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:07.416 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:48:07.416 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:07.416 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:07.416 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:07.417 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:07.417 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:48:07.417 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:07.417 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:07.418 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:48:07.418 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:07.418 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:07.418 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:07.419 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:48:07.419 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:07.419 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:07.419 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:07.420 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:07.420 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:48:07.420 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:07.421 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:07.421 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.858000_209.1", 2022-09-13T07:48:07.421 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:07.421 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:07.422 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:07.422 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T07:48:07.422 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:07.422 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:07.423 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:07.423 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:07.423 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:48:07.423 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:07.424 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:07.424 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:48:07.424 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:07.424 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:07.425 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:07.425 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:48:07.425 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:07.425 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:07.426 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:07.426 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:48:07.426 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:07.426 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:48:07.427 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:48:07.427 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:48:07.427 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:07.427 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:48:07Z" 2022-09-13T07:48:07.428 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:48:07.429 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.858000_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.858000Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:48:07Z" } 2022-09-13T07:48:07.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:48:07.430 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:07.431 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:07.431 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.858000_209.1 2022-09-13T07:48:12.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:12.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:12.435 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:48:12.505 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:48:12.505 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:48:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:12.525+0000 7f7449f5e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:12.526+0000 7f7449f5e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:12.544 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:48:12.544 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:48:12.545 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:48:12.545 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:48:12.550 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:48:12.550 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:48:12.551 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:12.551 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:12.552 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:48:12.552 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:12.552 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:48:12.553 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:12.553 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:12.554 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:48:12.554 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:12.554 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:48:12.555 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:12.555 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:48:12.556 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:12.556 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:12.556 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:12.557 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:12.557 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:48:12.558 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:12.558 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:12.558 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:48:12.559 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:12.559 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:12.560 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:12.560 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:48:12.560 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:12.561 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:12.561 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:12.562 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:12.562 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:48:12.563 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:12.563 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:12.563 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.854520_207.1", 2022-09-13T07:48:12.564 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:12.564 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:12.564 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:12.565 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T07:48:12.565 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:12.566 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:12.566 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:12.566 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:12.567 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:48:12.567 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:12.568 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:12.568 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:48:12.568 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:12.569 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:12.569 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:12.570 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:48:12.570 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:12.570 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:12.571 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:12.571 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:48:12.572 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:12.572 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:48:12.572 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:48:12.573 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:48:12.573 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:12.574 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:48:12Z" 2022-09-13T07:48:12.574 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:48:12.576 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.854520_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.854520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:48:12Z" } 2022-09-13T07:48:12.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:48:12.578 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:12.579 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:12.579 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.854520_207.1 2022-09-13T07:48:17.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:17.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:17.584 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:48:17.636 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:48:17.637 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:48:17.679 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:17.675+0000 7f56b402a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:17.680 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:17.676+0000 7f56b402a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:17.698 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:48:17.699 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:48:17.699 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:48:17.700 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:48:17.700 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:48:17.701 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:48:17.701 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:17.702 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:17.702 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:48:17.703 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:17.703 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:48:17.704 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:17.704 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:17.705 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:48:17.706 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:17.706 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:48:17.707 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:17.707 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:48:17.707 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:17.708 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:17.708 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:17.709 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:17.709 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:48:17.710 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:17.710 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:17.710 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:48:17.711 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:17.711 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:17.712 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:17.712 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:48:17.713 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:17.713 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:17.714 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:17.714 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:17.714 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:48:17.715 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:17.715 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:17.715 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.854520_207.1", 2022-09-13T07:48:17.715 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:17.716 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:17.716 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:17.716 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T07:48:17.717 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:17.718 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:17.719 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:17.719 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:17.720 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:48:17.720 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:17.720 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:17.721 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:48:17.721 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:17.721 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:17.721 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:17.722 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:48:17.722 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:17.722 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:17.723 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:17.724 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:48:17.724 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:17.725 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:48:17.725 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:48:17.725 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:48:17.726 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:17.727 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:48:17Z" 2022-09-13T07:48:17.727 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:48:17.728 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.854520_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.854520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:48:17Z" } 2022-09-13T07:48:17.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:48:17.730 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:17.731 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:17.732 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.854520_207.1 2022-09-13T07:48:22.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:22.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:22.736 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:48:22.786 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:48:22.787 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:48:22.825 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:22.821+0000 7f1e58aa0600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:22.826 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:22.822+0000 7f1e58aa0600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:22.851 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:48:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:48:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:48:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:48:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:48:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:48:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:22.853 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:22.853 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:48:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:48:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:22.855 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:22.855 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:48:22.856 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:22.856 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:48:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:48:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:22.858 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:22.858 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:22.859 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:22.859 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:48:22.859 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:48:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:48:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:48:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T07:48:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T07:48:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:48:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:48:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:48:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:48:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:48:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:48:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:48:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:22.870 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:48:22Z" 2022-09-13T07:48:22.870 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:48:22.871 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:48:22Z" } 2022-09-13T07:48:22.871 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:48:22.872 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:22.872 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:22.873 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.832045_205.1 2022-09-13T07:48:27.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:27.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:27.876 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:48:27.929 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:48:27.930 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:48:27.981 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:27.975+0000 7f3055548600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:27.982 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:27.976+0000 7f3055548600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:28.000 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:48:28.001 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:48:28.002 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:48:28.002 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:48:28.003 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:48:28.003 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:48:28.004 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:28.004 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:28.005 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:48:28.005 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:28.006 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:48:28.007 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:28.007 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:28.008 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:48:28.008 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:28.009 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:48:28.009 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:28.010 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:48:28.010 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:28.011 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:28.011 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:28.012 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:28.014 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:48:28.014 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:28.015 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:28.015 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:48:28.015 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:28.015 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:28.016 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:28.016 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:48:28.016 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:28.016 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:28.017 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:28.017 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:28.017 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:48:28.017 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:28.018 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:28.018 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.953175_216.1", 2022-09-13T07:48:28.018 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:28.019 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:28.019 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:28.019 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T07:48:28.019 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:28.020 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:28.020 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:28.020 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:28.020 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:48:28.021 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:28.021 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:28.021 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:48:28.021 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:28.022 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:28.022 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:28.022 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:48:28.022 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:28.023 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:28.023 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:28.023 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:48:28.023 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:28.024 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:48:28.024 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:48:28.024 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:48:28.024 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:28.025 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:48:27Z" 2022-09-13T07:48:28.025 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:48:28.026 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.953175_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.953175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:48:27Z" } 2022-09-13T07:48:28.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:48:28.027 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:28.028 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:28.028 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.953175_216.1 2022-09-13T07:48:33.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:33.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:33.031 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:48:33.088 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:48:33.089 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:48:33.122 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:33.117+0000 7f4880825600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:33.123 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:33.119+0000 7f4880825600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:33.137 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:48:33.137 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:48:33.138 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:48:33.138 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:48:33.138 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:48:33.139 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:48:33.139 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:33.139 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:33.139 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:48:33.140 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:33.140 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:48:33.140 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:33.140 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:33.141 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:48:33.141 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:33.141 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:48:33.142 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:33.142 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:48:33.142 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:33.142 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:33.143 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:33.143 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:33.143 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:48:33.143 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:33.144 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:33.144 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:48:33.144 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:33.144 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:33.145 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:33.145 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:48:33.145 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:33.145 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:33.145 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:33.146 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:33.146 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:48:33.146 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:33.147 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:33.147 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T07:48:33.147 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:33.147 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:33.147 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:33.148 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T07:48:33.149 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:33.149 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:33.150 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:33.150 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:33.150 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:48:33.150 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:33.151 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:33.151 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:48:33.151 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:33.152 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:33.152 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:33.152 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:48:33.152 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:33.153 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:33.153 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:33.153 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:48:33.154 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:33.154 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:48:33.154 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:48:33.154 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:48:33.155 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:33.155 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:48:33Z" 2022-09-13T07:48:33.155 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:48:33.157 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:48:33Z" } 2022-09-13T07:48:33.157 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:48:33.158 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:33.159 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:33.159 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.949151_215.1 2022-09-13T07:48:38.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:38.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:38.163 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:48:38.220 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:48:38.220 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:48:38.287 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:38.283+0000 7ffbca1b6600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:38.297 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:38.293+0000 7ffbca1b6600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:38.373 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:48:38.374 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:48:38.374 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:48:38.375 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:48:38.375 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:48:38.375 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:48:38.376 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:38.376 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:38.377 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:48:38.377 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:38.377 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:48:38.378 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:38.378 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:38.379 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:48:38.379 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:38.380 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:48:38.381 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:38.381 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:48:38.382 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:38.382 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:38.382 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:38.382 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:38.383 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:48:38.383 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:38.383 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:38.383 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:48:38.384 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:38.384 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:38.384 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:38.385 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:48:38.385 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:38.385 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:38.386 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:38.386 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:38.387 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:48:38.387 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:38.387 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:38.388 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.220845_233.1", 2022-09-13T07:48:38.388 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:38.388 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:38.389 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:38.389 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T07:48:38.389 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:38.390 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:38.390 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:38.390 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:38.390 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:48:38.391 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:38.391 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:38.391 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:48:38.391 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:38.392 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:38.392 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:38.392 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:48:38.392 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:38.393 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:38.393 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:38.393 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:48:38.394 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:38.394 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:48:38.394 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:48:38.394 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:48:38.395 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:38.395 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:48:38Z" 2022-09-13T07:48:38.395 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:48:38.397 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.220845_233.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.220845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:48:38Z" } 2022-09-13T07:48:38.397 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:48:38.398 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:38.399 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:38.399 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055099.220845_233.1 2022-09-13T07:48:43.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:43.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:43.402 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:48:43.455 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:48:43.456 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:48:43.483 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:43.480+0000 7f5f745ed600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:43.484 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:43.481+0000 7f5f745ed600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:43.497 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:48:43.497 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:48:43.498 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:48:43.498 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:48:43.499 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:48:43.499 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:48:43.500 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:43.500 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:43.501 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:48:43.501 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:43.502 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:48:43.502 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:43.503 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:43.503 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:48:43.504 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:43.504 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:48:43.505 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:43.505 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:48:43.505 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:43.506 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:43.506 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:43.507 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:43.507 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:48:43.508 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:43.508 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:43.509 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:48:43.509 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:43.510 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:43.510 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:43.511 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:48:43.511 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:43.512 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:43.512 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:43.513 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:43.514 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:48:43.514 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:43.515 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:43.515 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.861945_210.1", 2022-09-13T07:48:43.516 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:43.516 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:43.517 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:43.517 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T07:48:43.518 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:43.518 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:43.519 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:43.519 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:43.519 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:48:43.520 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:43.520 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:43.521 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:48:43.521 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:43.521 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:43.522 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:43.522 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:48:43.522 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:43.522 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:43.523 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:43.523 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:48:43.523 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:43.523 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:48:43.524 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:48:43.524 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:48:43.524 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:43.524 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:48:43Z" 2022-09-13T07:48:43.525 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:48:43.526 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.861945_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.861945Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:48:43Z" } 2022-09-13T07:48:43.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:48:43.528 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:43.530 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:43.530 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.861945_210.1 2022-09-13T07:48:48.532 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:48.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:48.534 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:48:48.588 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:48:48.589 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:48:48.629 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:48.624+0000 7f690b7d0600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:48.630 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:48.625+0000 7f690b7d0600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:48.643 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:48:48.644 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:48:48.644 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:48:48.645 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:48:48.645 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:48:48.646 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:48:48.646 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:48.647 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:48.647 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:48:48.647 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:48.648 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:48:48.648 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:48.648 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:48.649 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:48:48.649 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:48.649 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:48:48.650 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:48.650 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:48:48.651 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:48.651 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:48.651 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:48.651 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:48.652 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:48:48.652 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:48.652 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:48.652 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:48:48.652 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:48.653 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:48.653 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:48.653 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:48:48.654 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:48.654 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:48.654 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:48.654 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:48.655 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:48:48.655 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:48.655 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:48.656 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:48:48.656 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:48.656 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:48.656 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:48.657 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:48:48.657 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:48.657 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:48.658 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:48.658 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:48.659 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:48:48.659 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:48.660 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:48.660 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:48:48.660 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:48.661 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:48.661 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:48.661 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:48:48.661 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:48.662 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:48.662 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:48.662 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:48:48.662 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:48.663 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:48:48.663 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:48:48.663 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:48:48.663 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:48.664 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:48:48Z" 2022-09-13T07:48:48.664 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:48:48.666 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:48:48Z" } 2022-09-13T07:48:48.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:48:48.667 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:48.668 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:48.668 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.834293_206.1 2022-09-13T07:48:53.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:53.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:53.671 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:48:53.725 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:48:53.726 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:48:53.763 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:53.758+0000 7f308a3ce600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:53.764 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:53.760+0000 7f308a3ce600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:53.783 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:48:53.784 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:48:53.784 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:48:53.784 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:48:53.785 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:48:53.785 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:48:53.785 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:53.786 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:53.786 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:48:53.787 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:53.787 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:48:53.787 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:53.788 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:53.788 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:48:53.788 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:53.789 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:48:53.789 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:53.789 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:48:53.790 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:53.790 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:53.791 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:53.791 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:53.791 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:48:53.792 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:53.792 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:53.792 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:48:53.793 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:53.793 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:53.793 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:53.794 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:48:53.794 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:53.795 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:53.795 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:53.796 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:53.796 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:48:53.797 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:53.797 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:53.797 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.953175_216.1", 2022-09-13T07:48:53.798 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:53.798 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:53.799 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:53.799 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T07:48:53.800 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:53.800 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:53.800 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:53.801 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:53.801 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:48:53.801 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:53.801 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:53.802 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:48:53.802 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:53.802 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:53.803 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:53.803 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:48:53.803 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:53.804 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:53.804 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:53.805 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:48:53.805 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:53.805 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:48:53.806 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:48:53.806 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:48:53.807 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:53.807 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:48:53Z" 2022-09-13T07:48:53.808 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:48:53.810 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.953175_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.953175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:48:53Z" } 2022-09-13T07:48:53.811 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:48:53.812 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:53.813 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:53.813 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.953175_216.1 2022-09-13T07:48:58.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:58.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:48:58.817 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:48:58.872 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:48:58.873 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:48:58.896 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:58.892+0000 7f09f4516600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:58.897 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:48:58.893+0000 7f09f4516600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:48:58.912 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:48:58.913 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:48:58.913 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:48:58.914 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:48:58.914 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:48:58.915 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:48:58.915 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:58.916 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:58.916 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:48:58.917 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:58.917 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:48:58.918 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:58.918 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:58.919 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:48:58.919 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:58.919 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:48:58.920 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:58.920 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:48:58.921 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:58.921 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:58.923 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:58.923 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:58.924 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:48:58.924 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:58.924 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:58.925 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:48:58.925 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:58.926 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:58.926 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:58.927 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:48:58.927 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:58.928 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:58.928 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:58.928 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:58.929 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:48:58.929 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:58.930 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:58.930 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.858000_209.1", 2022-09-13T07:48:58.931 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:58.931 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:58.932 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:58.932 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T07:48:58.933 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:58.933 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:58.934 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:58.934 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:48:58.935 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:48:58.935 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:48:58.937 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:48:58.937 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:48:58.938 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:48:58.938 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:48:58.939 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:48:58.940 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:48:58.940 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:48:58.941 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:58.941 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:48:58.942 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:48:58.943 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:58.943 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:48:58.944 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:48:58.944 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:48:58.945 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:48:58.946 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:48:58Z" 2022-09-13T07:48:58.946 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:48:58.947 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.858000_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.858000Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:48:58Z" } 2022-09-13T07:48:58.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:48:58.949 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:58.950 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:48:58.951 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.858000_209.1 2022-09-13T07:49:03.952 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:03.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:03.955 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:49:04.009 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:49:04.010 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:49:04.041 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:04.035+0000 7ff0f65f3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:04.041 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:04.036+0000 7ff0f65f3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:04.053 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:49:04.054 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:49:04.054 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:49:04.054 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:49:04.055 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:49:04.055 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:49:04.055 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:04.056 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:04.056 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:49:04.057 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:04.057 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:49:04.058 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:04.058 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:04.059 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:49:04.059 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:04.060 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:49:04.060 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:04.061 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:49:04.061 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:04.062 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:04.062 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:04.062 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:04.063 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:49:04.063 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:04.064 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:04.064 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:49:04.065 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:04.065 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:04.066 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:04.067 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:49:04.067 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:04.068 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:04.068 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:04.068 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:04.068 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:49:04.069 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:04.069 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:04.069 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T07:49:04.070 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:04.070 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:04.071 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:04.071 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T07:49:04.072 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:04.072 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:04.073 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:04.073 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:04.074 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:49:04.074 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:04.075 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:04.075 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:49:04.076 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:04.076 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:04.077 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:04.077 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:49:04.078 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:04.078 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:04.079 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:04.079 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:49:04.080 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:04.080 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:49:04.080 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:49:04.080 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:49:04.081 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:04.081 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:49:04Z" 2022-09-13T07:49:04.081 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:49:04.082 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:49:04Z" } 2022-09-13T07:49:04.083 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:49:04.084 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:04.084 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:04.084 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.949151_215.1 2022-09-13T07:49:09.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:09.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:09.088 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:49:09.179 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:49:09.180 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:49:09.210 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:09.206+0000 7fc17fa67600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:09.211 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:09.207+0000 7fc17fa67600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:09.227 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:49:09.227 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:49:09.228 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:49:09.228 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:49:09.229 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:49:09.229 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:49:09.229 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:09.230 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:09.230 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:49:09.231 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:09.231 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:49:09.232 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:09.232 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:09.233 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:49:09.233 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:09.233 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:49:09.233 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:09.234 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:49:09.234 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:09.234 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:09.235 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:09.235 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:09.235 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:49:09.235 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:09.236 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:09.236 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:49:09.237 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:09.237 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:09.238 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:09.239 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:49:09.239 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:09.240 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:09.240 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:09.240 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:09.241 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:49:09.241 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:09.242 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:09.242 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.857076_208.1", 2022-09-13T07:49:09.242 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:09.243 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:09.243 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:09.244 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T07:49:09.244 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:09.245 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:09.245 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:09.245 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:09.246 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:49:09.246 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:09.247 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:09.247 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:49:09.248 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:09.248 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:09.249 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:09.249 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:49:09.250 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:09.250 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:09.250 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:09.251 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:49:09.251 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:09.252 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:49:09.252 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:49:09.252 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:49:09.253 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:09.253 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:49:09Z" 2022-09-13T07:49:09.254 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:49:09.255 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.857076_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.857076Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:49:09Z" } 2022-09-13T07:49:09.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:49:09.257 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:09.258 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:09.258 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.857076_208.1 2022-09-13T07:49:14.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:14.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:14.262 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:49:14.313 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:49:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:49:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:14.357+0000 7fc5d2a7a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:14.363 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:14.358+0000 7fc5d2a7a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:14.375 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:49:14.375 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:49:14.376 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:49:14.376 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:49:14.377 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:49:14.377 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:49:14.378 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:14.378 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:14.379 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:49:14.379 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:14.380 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:49:14.380 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:14.381 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:14.381 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:49:14.381 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:14.382 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:49:14.382 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:14.383 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:49:14.383 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:14.383 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:14.384 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:14.384 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:14.385 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:49:14.385 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:14.386 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:14.386 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:49:14.386 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:14.387 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:14.387 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:14.388 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:49:14.388 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:14.389 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:14.389 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:14.389 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:14.390 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:49:14.390 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:14.390 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:14.390 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:49:14.391 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:14.391 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:14.391 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:14.391 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:49:14.392 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:14.392 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:14.392 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:14.393 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:14.393 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:49:14.393 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:14.393 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:14.394 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:49:14.407 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:14.408 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:14.408 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:14.409 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:49:14.410 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:14.410 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:14.411 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:14.411 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:49:14.412 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:14.412 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:49:14.414 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:49:14.414 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:49:14.418 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:14.418 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:49:14Z" 2022-09-13T07:49:14.419 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:49:14.421 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:49:14Z" } 2022-09-13T07:49:14.422 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:49:14.423 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:14.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:14.425 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.834293_206.1 2022-09-13T07:49:19.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:19.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:19.429 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:49:19.481 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:49:19.481 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:49:19.522 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:19.516+0000 7ffa8a41c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:19.523 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:19.517+0000 7ffa8a41c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:19.538 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:49:19.539 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:49:19.539 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:49:19.539 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:49:19.540 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:49:19.540 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:49:19.540 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:19.541 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:19.541 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:49:19.541 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:19.542 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:49:19.543 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:19.543 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:19.543 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:49:19.544 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:19.544 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:49:19.544 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:19.545 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:49:19.545 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:19.545 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:19.546 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:19.546 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:19.546 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:49:19.546 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:19.547 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:19.547 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:49:19.547 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:19.548 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:19.548 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:19.548 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:49:19.549 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:19.549 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:19.549 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:19.549 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:19.550 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:49:19.550 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:19.550 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:19.551 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.857076_208.1", 2022-09-13T07:49:19.551 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:19.551 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:19.551 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:19.552 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T07:49:19.552 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:19.552 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:19.552 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:19.553 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:19.553 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:49:19.553 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:19.554 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:19.554 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:49:19.554 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:19.554 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:19.555 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:19.555 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:49:19.555 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:19.555 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:19.556 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:19.556 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:49:19.556 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:19.557 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:49:19.557 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:49:19.557 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:49:19.557 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:19.558 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:49:19Z" 2022-09-13T07:49:19.558 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:49:19.559 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.857076_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.857076Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:49:19Z" } 2022-09-13T07:49:19.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:49:19.560 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:19.561 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:19.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.857076_208.1 2022-09-13T07:49:24.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:24.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:24.565 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:49:24.622 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:49:24.622 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:49:24.653 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:24.649+0000 7fcbc9600600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:24.654 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:24.651+0000 7fcbc9600600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:24.669 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:49:24.670 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:49:24.671 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:49:24.671 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:49:24.672 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:49:24.672 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:49:24.673 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:24.673 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:24.674 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:49:24.674 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:49:24.675 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:24.676 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:24.676 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:49:24.677 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:24.677 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:49:24.678 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:24.678 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:49:24.678 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:24.679 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:49:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:24.680 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:49:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:24.681 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:49:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:24.682 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:49:24.683 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:24.684 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:24.684 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T07:49:24.684 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:24.685 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:24.685 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:24.686 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T07:49:24.686 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:24.687 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:24.687 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:24.688 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:24.688 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:49:24.688 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:24.689 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:24.689 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:49:24.690 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:24.690 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:24.691 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:24.691 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:49:24.692 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:24.692 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:24.693 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:24.693 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:49:24.694 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:24.694 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:49:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:49:24.695 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:49:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:24.696 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:49:24Z" 2022-09-13T07:49:24.697 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:49:24.699 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:49:24Z" } 2022-09-13T07:49:24.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:49:24.701 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:24.702 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:24.704 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.949151_215.1 2022-09-13T07:49:29.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:29.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:29.711 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:49:29.799 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:49:29.800 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:49:29.853 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:29.849+0000 7fbe18ea3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:29.854 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:29.850+0000 7fbe18ea3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:29.880 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:49:29.881 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:49:29.882 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:49:29.882 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:49:29.882 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:49:29.883 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:49:29.883 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:29.884 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:29.885 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:49:29.885 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:29.885 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:49:29.887 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:29.887 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:29.887 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:49:29.888 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:29.888 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:49:29.889 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:29.889 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:49:29.890 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:29.890 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:29.891 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:29.891 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:29.891 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:49:29.892 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:29.892 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:29.893 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:49:29.893 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:29.894 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:29.894 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:29.895 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:49:29.898 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:29.898 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:29.899 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:29.899 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:29.900 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:49:29.911 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:29.913 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:29.913 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.911619_214.1", 2022-09-13T07:49:29.914 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:29.915 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:29.915 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:29.916 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T07:49:29.916 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:29.921 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:29.921 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:29.922 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:29.922 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:49:29.922 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:29.923 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:29.924 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:49:29.924 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:29.924 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:29.925 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:29.925 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:49:29.926 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:29.926 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:29.927 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:29.927 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:49:29.927 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:29.928 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:49:29.928 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:49:29.928 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:49:29.929 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:29.939 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:49:29Z" 2022-09-13T07:49:29.939 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:49:29.940 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.911619_214.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.911619Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:49:29Z" } 2022-09-13T07:49:29.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:49:29.948 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:29.949 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:29.950 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.911619_214.1 2022-09-13T07:49:34.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:34.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:34.953 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:49:35.002 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:49:35.003 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:49:35.029 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:35.025+0000 7fef12810600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:35.030 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:35.026+0000 7fef12810600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:35.044 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:49:35.045 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:49:35.045 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:49:35.045 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:49:35.045 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:49:35.046 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:49:35.046 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:35.046 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:35.046 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:49:35.047 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:35.047 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:49:35.047 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:35.047 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:35.048 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:49:35.048 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:35.048 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:49:35.048 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:35.049 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:49:35.049 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:35.049 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:35.049 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:35.050 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:35.050 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:49:35.050 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:35.050 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:35.051 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:49:35.051 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:35.051 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:35.051 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:35.052 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:49:35.052 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:35.052 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:35.052 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:35.053 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:35.053 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:49:35.053 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:35.053 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:35.054 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.858000_209.1", 2022-09-13T07:49:35.054 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:35.054 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:35.054 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:35.055 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T07:49:35.055 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:35.055 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:35.055 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:35.056 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:35.056 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:49:35.056 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:35.056 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:35.057 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:49:35.057 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:35.057 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:35.057 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:35.057 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:49:35.058 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:35.058 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:35.058 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:35.058 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:49:35.059 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:35.059 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:49:35.059 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:49:35.059 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:49:35.060 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:35.060 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:49:35Z" 2022-09-13T07:49:35.060 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:49:35.061 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.858000_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.858000Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:49:35Z" } 2022-09-13T07:49:35.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:49:35.063 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:35.063 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:35.064 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.858000_209.1 2022-09-13T07:49:40.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:40.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:40.066 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:49:40.121 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:49:40.122 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:49:40.165 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:40.160+0000 7f15e9a4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:40.166 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:40.161+0000 7f15e9a4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:40.180 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:49:40.181 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:49:40.182 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:49:40.182 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:49:40.183 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:49:40.183 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:49:40.184 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:40.184 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:40.185 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:49:40.185 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:40.186 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:49:40.186 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:40.187 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:40.187 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:49:40.188 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:40.188 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:49:40.188 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:40.189 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:49:40.189 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:40.189 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:40.189 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:40.190 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:40.190 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:49:40.191 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:40.191 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:40.191 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:49:40.192 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:40.192 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:40.193 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:40.193 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:49:40.194 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:40.194 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:40.195 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:40.195 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:40.196 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:49:40.196 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:40.196 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:40.197 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:49:40.197 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:40.198 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:40.198 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:40.199 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:49:40.199 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:40.200 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:40.200 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:40.200 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:40.201 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:49:40.201 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:40.201 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:40.202 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:49:40.202 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:40.202 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:40.202 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:40.203 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:49:40.204 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:40.204 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:40.205 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:40.205 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:49:40.206 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:40.207 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:49:40.207 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:49:40.207 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:49:40.208 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:40.208 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:49:40Z" 2022-09-13T07:49:40.209 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:49:40.210 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:49:40Z" } 2022-09-13T07:49:40.211 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:49:40.211 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:40.212 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:40.212 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.834293_206.1 2022-09-13T07:49:45.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:45.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:45.216 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:49:45.268 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:49:45.269 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:49:45.302 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:45.297+0000 7f91145ac600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:45.303 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:45.298+0000 7f91145ac600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:45.320 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:49:45.321 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:49:45.321 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:49:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:49:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:49:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:49:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:45.332 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:45.332 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:49:45.333 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:45.333 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:49:45.334 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:45.334 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:45.335 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:49:45.336 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:45.336 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:49:45.337 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:45.337 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:49:45.338 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:45.338 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:45.339 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:45.339 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:45.340 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:49:45.341 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:45.341 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:45.342 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:49:45.343 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:45.344 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:45.346 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:45.346 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:49:45.347 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:45.347 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:45.348 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:45.348 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:45.349 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:49:45.349 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:45.350 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:45.354 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.900410_211.1", 2022-09-13T07:49:45.355 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:45.355 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:45.356 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:45.357 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T07:49:45.358 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:45.366 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:45.366 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:45.367 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:45.368 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:49:45.368 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:45.369 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:45.370 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:49:45.370 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:45.371 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:45.372 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:45.372 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:49:45.373 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:45.373 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:45.374 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:45.375 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:49:45.375 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:45.376 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:49:45.376 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:49:45.377 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:49:45.377 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:45.378 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:49:45Z" 2022-09-13T07:49:45.379 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:49:45.381 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.900410_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.900410Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:49:45Z" } 2022-09-13T07:49:45.383 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:49:45.384 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:45.384 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:45.385 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.900410_211.1 2022-09-13T07:49:50.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:50.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:50.392 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:49:50.445 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:49:50.445 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:49:50.487 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:50.481+0000 7f5257cfc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:50.487 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:50.482+0000 7f5257cfc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:50.500 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:49:50.501 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:49:50.502 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:49:50.502 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:49:50.503 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:49:50.503 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:49:50.504 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:50.504 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:50.505 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:49:50.506 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:50.506 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:49:50.507 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:50.507 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:50.508 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:49:50.508 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:50.509 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:49:50.509 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:50.510 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:49:50.511 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:50.512 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:50.513 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:50.514 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:50.515 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:49:50.516 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:50.516 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:50.517 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:49:50.517 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:50.518 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:50.519 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:50.519 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:49:50.520 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:50.520 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:50.520 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:50.520 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:50.521 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:49:50.521 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:50.521 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:50.521 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T07:49:50.522 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:50.522 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:50.522 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:50.522 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T07:49:50.523 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:50.523 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:50.523 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:50.523 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:50.524 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:49:50.524 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:50.524 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:50.525 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:49:50.525 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:50.525 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:50.525 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:50.525 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:49:50.526 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:50.526 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:50.526 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:50.527 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:49:50.527 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:50.528 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:49:50.528 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:49:50.529 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:49:50.531 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:50.531 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:49:50Z" 2022-09-13T07:49:50.532 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:49:50.534 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:49:50Z" } 2022-09-13T07:49:50.535 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:49:50.535 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:50.536 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:50.536 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.949151_215.1 2022-09-13T07:49:55.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:55.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:49:55.539 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:49:55.597 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:49:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:49:55.639 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:55.635+0000 7fbf00cfc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:49:55.636+0000 7fbf00cfc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:49:55.656 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:49:55.657 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:49:55.663 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:49:55.664 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:49:55.664 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:49:55.665 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:49:55.665 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:55.666 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:55.667 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:49:55.667 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:55.667 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:49:55.668 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:55.668 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:55.669 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:49:55.669 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:55.670 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:49:55.670 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:55.671 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:49:55.672 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:55.672 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:55.672 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:55.673 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:55.673 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:49:55.674 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:55.675 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:55.675 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:49:55.676 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:55.676 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:55.676 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:55.677 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:49:55.677 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:55.677 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:55.677 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:55.679 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:55.680 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:49:55.680 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:55.681 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:55.681 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.070128_219.1", 2022-09-13T07:49:55.682 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:55.682 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:55.683 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:55.683 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T07:49:55.684 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:55.684 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:55.684 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:55.685 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:49:55.685 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:49:55.685 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:49:55.686 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:49:55.686 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:49:55.686 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:49:55.686 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:49:55.687 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:49:55.687 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:49:55.687 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:49:55.687 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:55.688 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:49:55.688 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:49:55.688 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:55.689 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:49:55.689 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:49:55.689 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:49:55.689 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:49:55.690 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:49:55Z" 2022-09-13T07:49:55.690 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:49:55.691 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.070128_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.070128Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:49:55Z" } 2022-09-13T07:49:55.692 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:49:55.692 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:55.693 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:49:55.693 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055099.070128_219.1 2022-09-13T07:50:00.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:00.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:00.697 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:50:00.760 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:50:00.761 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:50:00.867 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:00.863+0000 7f70b782e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:00.869 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:00.866+0000 7f70b782e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:00.910 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:50:00.910 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:50:00.911 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:50:00.911 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:50:00.912 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:50:00.912 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:50:00.912 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:00.913 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:00.913 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:50:00.913 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:00.914 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:50:00.914 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:00.915 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:00.915 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:50:00.916 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:00.917 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:50:00.917 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:00.917 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:50:00.918 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:00.918 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:00.919 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:00.919 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:00.919 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:50:00.920 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:00.920 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:00.921 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:50:00.921 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:00.921 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:00.922 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:00.924 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:50:00.924 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:00.924 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:00.925 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:00.925 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:00.925 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:50:00.925 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:00.926 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:00.926 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.105395_221.1", 2022-09-13T07:50:00.926 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:00.927 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:00.927 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:00.927 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T07:50:00.927 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:00.928 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:00.928 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:00.928 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:00.928 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:50:00.929 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:00.929 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:00.929 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:50:00.929 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:00.930 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:00.930 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:00.930 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:50:00.930 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:00.931 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:00.931 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:00.931 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:50:00.931 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:00.932 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:50:00.932 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:50:00.932 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:50:00.932 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:00.933 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:50:00Z" 2022-09-13T07:50:00.933 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:50:00.934 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.105395_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.105395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:50:00Z" } 2022-09-13T07:50:00.935 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.105395_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.105395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:50:00.935 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:00.936 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.105395_221.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:00.936 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055099.105395_221.1 2022-09-13T07:50:05.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:05.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:05.939 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:50:05.990 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:50:05.991 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:50:06.039 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:06.035+0000 7f13d82cd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:06.040 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:06.036+0000 7f13d82cd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:06.069 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:50:06.069 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:50:06.070 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:50:06.070 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:50:06.071 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:50:06.071 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:50:06.071 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:06.072 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:06.072 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:50:06.073 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:06.073 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:50:06.073 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:06.074 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:06.074 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:50:06.074 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:06.075 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:50:06.075 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:06.076 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:50:06.076 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:06.076 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:06.077 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:06.077 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:06.077 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:50:06.078 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:06.078 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:06.079 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:50:06.079 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:06.079 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:06.080 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:06.080 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:50:06.081 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:06.081 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:06.081 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:06.082 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:06.083 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:50:06.091 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:06.091 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:06.092 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T07:50:06.092 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:06.093 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:06.093 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:06.094 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T07:50:06.094 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:06.095 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:06.095 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:06.096 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:06.096 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:50:06.097 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:06.097 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:06.097 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:50:06.098 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:06.098 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:06.099 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:06.099 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:50:06.100 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:06.100 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:06.101 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:06.101 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:50:06.101 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:06.101 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:50:06.102 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:50:06.102 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:50:06.102 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:06.103 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:50:06Z" 2022-09-13T07:50:06.103 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:50:06.103 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:50:06Z" } 2022-09-13T07:50:06.104 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:50:06.104 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:06.105 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:06.105 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.832045_205.1 2022-09-13T07:50:11.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2022-09-13T07:50:11.119 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-39 2022-09-13T07:50:11.120 DEBUG:boto:path=/yxvvuh-39/ 2022-09-13T07:50:11.121 DEBUG:boto:auth_path=/yxvvuh-39/ 2022-09-13T07:50:11.122 DEBUG:boto:Method: PUT 2022-09-13T07:50:11.123 DEBUG:boto:Path: /yxvvuh-39/ 2022-09-13T07:50:11.124 DEBUG:boto:Data: 2022-09-13T07:50:11.125 DEBUG:boto:Headers: {} 2022-09-13T07:50:11.126 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:50:11.127 DEBUG:boto:Port: 8000 2022-09-13T07:50:11.128 DEBUG:boto:Params: {} 2022-09-13T07:50:11.130 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T07:50:11.131 DEBUG:boto:Token: None 2022-09-13T07:50:11.133 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:50:11 GMT /yxvvuh-39/ 2022-09-13T07:50:11.134 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nGoKqYVaD9ea7mQTyzPxW6JB7OM= 2022-09-13T07:50:11.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:50:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nGoKqYVaD9ea7mQTyzPxW6JB7OM=', 'Content-Length': '0'} 2022-09-13T07:50:11.158 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eef94920a47e8c27-0063203633-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:50:11.159 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-40 2022-09-13T07:50:11.160 DEBUG:boto:path=/yxvvuh-40/ 2022-09-13T07:50:11.160 DEBUG:boto:auth_path=/yxvvuh-40/ 2022-09-13T07:50:11.161 DEBUG:boto:Method: PUT 2022-09-13T07:50:11.161 DEBUG:boto:Path: /yxvvuh-40/ 2022-09-13T07:50:11.162 DEBUG:boto:Data: 2022-09-13T07:50:11.162 DEBUG:boto:Headers: {} 2022-09-13T07:50:11.163 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:50:11.164 DEBUG:boto:Port: 8000 2022-09-13T07:50:11.164 DEBUG:boto:Params: {} 2022-09-13T07:50:11.165 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T07:50:11.165 DEBUG:boto:Token: None 2022-09-13T07:50:11.166 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:50:11 GMT /yxvvuh-40/ 2022-09-13T07:50:11.167 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rjiSdm8WIwkoMwA7QiChkxSKqJo= 2022-09-13T07:50:11.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:50:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rjiSdm8WIwkoMwA7QiChkxSKqJo=', 'Content-Length': '0'} 2022-09-13T07:50:11.202 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c84b10d0cf50ebb4-0063203633-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:50:11.202 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-41 2022-09-13T07:50:11.203 DEBUG:boto:path=/yxvvuh-41/ 2022-09-13T07:50:11.203 DEBUG:boto:auth_path=/yxvvuh-41/ 2022-09-13T07:50:11.204 DEBUG:boto:Method: PUT 2022-09-13T07:50:11.204 DEBUG:boto:Path: /yxvvuh-41/ 2022-09-13T07:50:11.205 DEBUG:boto:Data: 2022-09-13T07:50:11.205 DEBUG:boto:Headers: {} 2022-09-13T07:50:11.206 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:50:11.206 DEBUG:boto:Port: 8001 2022-09-13T07:50:11.207 DEBUG:boto:Params: {} 2022-09-13T07:50:11.208 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-13T07:50:11.208 DEBUG:boto:Token: None 2022-09-13T07:50:11.209 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:50:11 GMT /yxvvuh-41/ 2022-09-13T07:50:11.209 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tdckn/zSbHPEQDwMRZhxQo2O2RU= 2022-09-13T07:50:11.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:50:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tdckn/zSbHPEQDwMRZhxQo2O2RU=', 'Content-Length': '0'} 2022-09-13T07:50:11.258 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f910536f596dfe57-0063203633-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:50:11.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:11.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:11.273 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:50:11.326 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:50:11.327 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:50:11.370 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:50:11.384 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:50:11.385 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:50:11.385 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:50:11.386 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:50:11.388 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:50:11.388 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:50:11.389 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:50:11.389 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:50:11.390 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:50:11.390 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:50:11.391 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663055411.238167_296.1", 2022-09-13T07:50:11.392 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:50:11.238167Z" 2022-09-13T07:50:11.392 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:50:11.393 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:50:11.393 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:50:11.394 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T07:50:11.394 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:50:11.395 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:50:11.397 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:11.398 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:50:11.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:11.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:11.401 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:50:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:50:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:50:11.500 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:11.496+0000 7fe9b3e0c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:11.501 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:11.497+0000 7fe9b3e0c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:11.518 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:50:11.518 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:50:11.519 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:50:11.519 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:50:11.520 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:50:11.520 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:50:11.521 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:11.521 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:11.522 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:50:11.522 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:11.523 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:50:11.523 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:11.524 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:11.524 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:50:11.525 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:11.527 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:50:11.528 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:11.528 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:50:11.529 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:11.530 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:11.530 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:11.531 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:11.531 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:50:11.532 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:11.532 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:11.533 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:50:11.533 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:11.534 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:11.534 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:11.536 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:50:11.536 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:11.537 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:11.538 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:11.539 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:11.539 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:50:11.540 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:11.540 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:11.541 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T07:50:11.542 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:11.543 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:11.546 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:11.547 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T07:50:11.547 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:11.547 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:11.548 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:11.548 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:11.548 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:50:11.549 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:11.549 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:11.549 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:50:11.549 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:11.550 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:11.550 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:11.551 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:50:11.551 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:11.552 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:11.552 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:11.553 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:50:11.553 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:11.554 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:50:11.554 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:50:11.555 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:50:11.555 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:11.556 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:50:11Z" 2022-09-13T07:50:11.556 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:50:11.558 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:50:11Z" } 2022-09-13T07:50:11.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:50:11.561 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:11.561 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:11.562 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.832045_205.1 2022-09-13T07:50:16.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:16.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:16.566 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:50:16.625 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:50:16.625 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:50:16.655 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:16.650+0000 7f7798e24600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:16.656 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:16.651+0000 7f7798e24600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:16.670 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:50:16.671 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:50:16.672 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:50:16.672 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:50:16.672 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:50:16.672 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:50:16.673 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:16.673 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:16.673 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:50:16.673 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:16.674 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:50:16.674 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:16.675 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:16.675 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:50:16.675 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:16.676 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:50:16.676 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:16.677 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:50:16.677 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:16.677 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:16.678 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:16.678 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:16.678 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:50:16.678 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:16.679 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:16.679 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:50:16.679 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:16.680 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:16.680 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:16.680 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:50:16.680 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:16.681 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:16.681 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:16.681 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:16.681 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:50:16.682 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:16.682 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:16.683 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.854520_207.1", 2022-09-13T07:50:16.683 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:16.683 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:16.683 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:16.684 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T07:50:16.684 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:16.684 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:16.684 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:16.685 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:16.685 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:50:16.685 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:16.685 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:16.686 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:50:16.686 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:16.686 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:16.687 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:16.687 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:50:16.687 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:16.687 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:16.688 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:16.688 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:50:16.688 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:16.688 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:50:16.689 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:50:16.689 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:50:16.689 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:16.698 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:50:16Z" 2022-09-13T07:50:16.698 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:50:16.699 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.854520_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.854520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:50:16Z" } 2022-09-13T07:50:16.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:50:16.701 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:16.701 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:16.702 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.854520_207.1 2022-09-13T07:50:21.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:21.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:21.705 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:50:21.754 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:50:21.755 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:50:21.791 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:21.787+0000 7f1a4aaf3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:21.792 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:21.788+0000 7f1a4aaf3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:21.806 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:50:21.807 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:50:21.807 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:50:21.808 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:50:21.808 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:50:21.808 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:50:21.809 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:21.809 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:21.809 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:50:21.809 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:21.810 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:50:21.810 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:21.811 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:21.811 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:50:21.811 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:21.811 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:50:21.812 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:21.812 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:50:21.812 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:21.812 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:21.813 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:21.813 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:21.813 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:50:21.813 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:21.814 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:21.814 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:50:21.815 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:21.815 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:21.815 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:21.815 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:50:21.816 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:21.816 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:21.816 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:21.817 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:21.817 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:50:21.818 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:21.819 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:21.819 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.911619_214.1", 2022-09-13T07:50:21.820 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:21.820 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:21.820 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:21.821 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T07:50:21.821 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:21.822 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:21.822 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:50:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:21.823 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:50:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:21.824 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:21.825 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:50:21.825 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:21.825 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:21.825 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:21.903 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:50:21.904 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:21.904 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:50:21.904 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:50:21.904 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:50:21.905 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:21.905 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:50:21Z" 2022-09-13T07:50:21.905 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:50:21.909 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.911619_214.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.911619Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:50:21Z" } 2022-09-13T07:50:21.911 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:50:21.911 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:21.912 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:21.913 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.911619_214.1 2022-09-13T07:50:26.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:26.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:26.917 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:50:26.982 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:50:26.983 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:50:27.025 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:27.020+0000 7f53ea506600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:27.026 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:27.021+0000 7f53ea506600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:27.038 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:50:27.039 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:50:27.039 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:50:27.039 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:50:27.040 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:50:27.040 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:50:27.041 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:27.041 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:27.042 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:50:27.042 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:27.043 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:50:27.043 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:27.044 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:27.044 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:50:27.044 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:27.044 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:50:27.045 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:27.045 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:50:27.045 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:27.045 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:27.046 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:27.046 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:27.047 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:50:27.047 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:27.047 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:27.047 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:50:27.048 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:27.048 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:27.048 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:27.049 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:50:27.049 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:27.049 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:27.049 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:27.050 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:27.050 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:50:27.050 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:27.050 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:27.051 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T07:50:27.051 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:27.051 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:27.051 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:27.052 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T07:50:27.052 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:27.052 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:27.052 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:27.053 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:27.053 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:50:27.053 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:27.053 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:27.054 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:50:27.054 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:27.054 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:27.055 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:27.055 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:50:27.055 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:27.055 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:27.056 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:27.056 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:50:27.056 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:27.056 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:50:27.057 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:50:27.057 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:50:27.057 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:27.057 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:50:27Z" 2022-09-13T07:50:27.058 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:50:27.059 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:50:27Z" } 2022-09-13T07:50:27.060 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:50:27.060 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:27.061 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:27.061 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.832045_205.1 2022-09-13T07:50:32.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:32.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:32.065 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:50:32.116 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:50:32.116 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:50:32.157 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:32.152+0000 7f4f8ab70600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:32.158 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:32.153+0000 7f4f8ab70600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:32.175 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:50:32.176 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:50:32.176 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:50:32.176 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:50:32.177 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:50:32.177 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:50:32.177 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:32.177 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:32.178 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:50:32.178 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:32.178 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:50:32.179 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:32.179 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:32.179 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:50:32.179 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:32.180 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:50:32.180 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:32.180 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:50:32.181 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:32.181 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:32.181 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:32.182 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:32.182 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:50:32.182 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:32.183 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:32.184 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:50:32.184 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:32.185 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:32.185 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:32.187 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:50:32.187 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:32.188 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:32.188 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:32.189 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:32.189 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:50:32.190 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:32.190 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:32.191 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:50:32.191 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:32.192 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:32.192 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:32.192 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:50:32.192 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:32.193 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:32.193 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:32.193 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:32.194 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:50:32.194 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:32.195 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:32.195 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:50:32.195 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:32.195 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:32.196 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:32.196 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:50:32.196 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:32.196 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:32.197 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:32.197 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:50:32.197 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:32.198 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:50:32.198 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:50:32.198 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:50:32.198 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:32.199 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:50:32Z" 2022-09-13T07:50:32.199 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:50:32.201 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:50:32Z" } 2022-09-13T07:50:32.201 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:50:32.202 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:32.202 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:32.203 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T07:50:37.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:37.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:37.205 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:50:37.261 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:50:37.261 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:50:37.296 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:37.288+0000 7fd9175ad600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:37.297 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:37.289+0000 7fd9175ad600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:37.308 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:50:37.309 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:50:37.309 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:50:37.309 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:50:37.309 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:50:37.310 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:50:37.310 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:37.310 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:37.311 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:50:37.311 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:37.311 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:50:37.312 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:37.312 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:37.313 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:50:37.313 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:37.314 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:50:37.314 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:37.315 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:50:37.315 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:37.316 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:37.316 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:37.316 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:37.317 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:50:37.317 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:37.318 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:37.318 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:50:37.319 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:37.319 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:37.319 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:37.319 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:50:37.320 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:37.320 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:37.320 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:37.321 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:37.321 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:50:37.322 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:37.322 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:37.323 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.994620_218.1", 2022-09-13T07:50:37.323 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:37.324 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:37.324 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:37.325 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.994620Z", 2022-09-13T07:50:37.325 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:37.326 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:37.326 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:37.327 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:37.327 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:50:37.328 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:37.328 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:37.329 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:50:37.329 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:37.329 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:37.330 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:37.330 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:50:37.330 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:37.331 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:37.331 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:37.332 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:50:37.332 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:37.333 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:50:37.333 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:50:37.333 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:50:37.334 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:37.334 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:50:37Z" 2022-09-13T07:50:37.335 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:50:37.335 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.994620_218.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.994620Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:50:37Z" } 2022-09-13T07:50:37.336 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.994620_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.994620Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:50:37.338 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:37.339 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.994620_218.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:37.339 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.994620_218.1 2022-09-13T07:50:42.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:42.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:42.345 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:50:42.439 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:50:42.440 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:50:42.487 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:42.483+0000 7f5e3598e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:42.488 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:42.485+0000 7f5e3598e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:42.511 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:50:42.512 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:50:42.512 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:50:42.512 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:50:42.513 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:50:42.513 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:50:42.514 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:42.514 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:42.514 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:50:42.515 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:42.515 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:50:42.515 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:42.515 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:42.516 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:50:42.516 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:42.516 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:50:42.517 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:42.517 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:50:42.518 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:42.518 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:42.519 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:42.519 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:42.519 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:50:42.521 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:42.521 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:42.521 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:50:42.522 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:42.522 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:42.522 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:42.523 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:50:42.523 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:42.523 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:42.523 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:42.524 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:42.524 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:50:42.524 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:42.524 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:42.525 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T07:50:42.525 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:42.525 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:42.525 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:42.525 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T07:50:42.526 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:42.526 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:42.526 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:42.526 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:42.527 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:50:42.527 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:42.527 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:42.527 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:50:42.528 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:42.528 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:42.528 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:42.528 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:50:42.529 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:42.529 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:42.529 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:42.529 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:50:42.530 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:42.530 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:50:42.530 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:50:42.531 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:50:42.531 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:42.531 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:50:42Z" 2022-09-13T07:50:42.534 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:50:42.536 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:50:42Z" } 2022-09-13T07:50:42.537 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:50:42.537 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:42.539 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:42.540 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.949151_215.1 2022-09-13T07:50:47.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:47.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:47.544 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:50:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:50:47.599 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:50:47.637 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:47.633+0000 7fcd5684d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:47.634+0000 7fcd5684d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:47.653 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:50:47.654 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:50:47.655 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:50:47.656 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:50:47.656 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:50:47.657 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:50:47.657 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:47.658 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:47.658 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:50:47.658 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:47.659 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:50:47.659 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:47.660 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:47.660 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:50:47.661 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:47.661 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:50:47.661 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:47.670 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:50:47.671 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:47.671 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:47.672 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:47.674 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:47.674 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:50:47.674 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:47.674 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:47.675 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:50:47.675 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:47.675 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:47.675 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:47.676 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:50:47.676 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:47.677 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:47.677 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:47.677 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:47.678 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:50:47.678 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:47.678 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:47.679 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.070128_219.1", 2022-09-13T07:50:47.679 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:47.680 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:47.680 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:47.680 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T07:50:47.681 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:47.681 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:47.682 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:47.683 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:47.683 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:50:47.684 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:47.684 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:47.685 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:50:47.685 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:47.686 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:47.686 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:47.687 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:50:47.687 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:47.688 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:47.688 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:47.688 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:50:47.689 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:47.689 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:50:47.690 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:50:47.690 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:50:47.691 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:47.691 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:50:47Z" 2022-09-13T07:50:47.691 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:50:47.692 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.070128_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.070128Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:50:47Z" } 2022-09-13T07:50:47.693 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:50:47.694 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:47.694 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:47.695 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.070128_219.1 2022-09-13T07:50:52.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:52.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:52.699 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:50:52.751 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:50:52.752 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:50:52.801 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:52.793+0000 7f97f5467600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:52.806 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:52.794+0000 7f97f5467600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:52.811 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:50:52.812 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:50:52.812 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:50:52.813 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:50:52.814 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:50:52.814 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:50:52.816 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:52.817 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:52.818 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:50:52.818 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:52.819 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:50:52.819 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:52.820 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:52.820 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:50:52.821 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:52.821 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:50:52.822 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:52.823 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:50:52.823 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:52.824 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:52.824 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:52.825 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:52.825 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:50:52.826 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:52.827 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:52.827 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:50:52.828 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:52.828 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:52.829 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:52.830 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:50:52.830 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:52.831 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:52.832 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:52.832 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:52.834 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:50:52.835 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:52.836 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:52.836 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:50:52.836 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:52.837 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:52.837 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:52.837 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:50:52.837 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:52.838 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:52.838 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:52.839 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:52.839 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:50:52.840 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:52.840 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:52.840 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:50:52.841 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:52.841 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:52.841 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:52.842 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:50:52.842 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:52.842 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:52.843 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:52.843 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:50:52.843 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:52.843 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:50:52.844 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:50:52.844 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:50:52.845 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:52.845 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:50:52Z" 2022-09-13T07:50:52.845 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:50:52.847 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:50:52Z" } 2022-09-13T07:50:52.848 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:50:52.849 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:52.849 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:52.849 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T07:50:57.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:57.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:50:57.853 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:50:57.903 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:50:57.904 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:50:57.937 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:57.928+0000 7f1ac1573600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:57.938 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:50:57.929+0000 7f1ac1573600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:50:57.946 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:50:57.947 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:50:57.947 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:50:57.947 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:50:57.948 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:50:57.948 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:50:57.948 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:57.948 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:57.949 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:50:57.949 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:57.949 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:50:57.950 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:57.950 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:57.950 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:50:57.951 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:57.951 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:50:57.952 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:57.952 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:50:57.952 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:57.953 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:57.954 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:57.954 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:57.954 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:50:57.955 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:57.955 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:57.955 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:50:57.956 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:57.956 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:57.956 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:57.956 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:50:57.957 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:57.957 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:57.957 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:57.957 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:57.958 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:50:57.958 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:57.958 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:57.959 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.146590_228.1", 2022-09-13T07:50:57.959 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:57.959 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:57.959 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:57.960 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.146590Z", 2022-09-13T07:50:57.960 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:57.960 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:57.960 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:57.961 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:50:57.961 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:50:57.962 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:50:57.962 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:50:57.963 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:50:57.963 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:50:57.964 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:50:57.965 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:50:57.965 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:50:57.966 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:50:57.966 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:57.967 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:50:57.967 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:50:57.968 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:57.968 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:50:57.969 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:50:57.969 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:50:57.970 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:50:57.971 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:50:57Z" 2022-09-13T07:50:57.971 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:50:57.973 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.146590_228.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.146590Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:50:57Z" } 2022-09-13T07:50:57.973 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.146590_228.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.146590Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:50:57.974 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:57.975 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.146590_228.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:50:57.976 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.146590_228.1 2022-09-13T07:51:02.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:02.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:02.980 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:51:03.031 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:51:03.032 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:51:03.065 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:03.061+0000 7f55e7eab600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:03.066 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:03.062+0000 7f55e7eab600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:03.079 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:51:03.080 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:51:03.080 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:51:03.080 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:51:03.081 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:51:03.081 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:51:03.081 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:03.081 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:03.082 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:51:03.082 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:03.082 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:51:03.083 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:03.083 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:03.083 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:51:03.083 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:03.084 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:51:03.084 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:03.084 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:51:03.084 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:03.085 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:03.085 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:03.085 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:03.086 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:51:03.086 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:03.087 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:03.087 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:51:03.087 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:03.088 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:03.088 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:03.088 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:51:03.089 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:03.089 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:03.089 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:03.089 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:03.090 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:51:03.090 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:03.090 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:03.090 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.105395_221.1", 2022-09-13T07:51:03.091 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:03.091 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:03.091 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:03.091 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T07:51:03.092 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:03.092 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:03.092 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:03.092 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:03.093 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:51:03.093 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:03.094 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:03.094 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:51:03.094 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:03.095 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:03.095 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:03.095 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:51:03.096 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:03.096 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:03.096 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:03.097 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:51:03.097 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:03.097 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:51:03.098 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:51:03.098 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:51:03.098 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:03.099 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:51:03Z" 2022-09-13T07:51:03.099 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:51:03.100 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.105395_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.105395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:51:03Z" } 2022-09-13T07:51:03.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.105395_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.105395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:51:03.102 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:03.102 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.105395_221.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:03.103 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.105395_221.1 2022-09-13T07:51:08.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:08.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:08.107 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:51:08.158 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:51:08.159 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:51:08.187 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:08.182+0000 7fb131da7600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:08.188 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:08.183+0000 7fb131da7600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:08.200 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:51:08.201 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:51:08.202 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:51:08.202 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:51:08.203 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:51:08.203 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:51:08.204 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:08.204 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:08.205 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:51:08.205 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:08.206 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:51:08.206 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:08.207 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:08.207 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:51:08.208 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:08.208 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:51:08.209 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:08.209 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:51:08.210 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:08.210 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:08.211 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:08.211 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:08.212 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:51:08.212 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:08.213 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:08.213 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:51:08.213 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:08.214 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:08.214 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:08.215 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:51:08.215 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:08.216 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:08.216 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:08.217 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:08.217 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:51:08.218 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:08.218 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:08.219 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:51:08.219 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:08.220 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:08.220 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:08.221 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:51:08.221 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:08.222 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:08.223 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:08.223 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:08.224 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:51:08.224 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:51:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:08.226 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:08.226 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:08.227 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:51:08.227 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:08.228 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:08.228 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:08.229 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:51:08.229 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:08.230 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:51:08.230 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:51:08.231 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:51:08.232 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:08.232 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:51:08Z" 2022-09-13T07:51:08.233 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:51:08.235 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:51:08Z" } 2022-09-13T07:51:08.236 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:51:08.237 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:08.238 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:08.239 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T07:51:13.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:13.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:13.242 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:51:13.331 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:51:13.332 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:51:13.365 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:13.360+0000 7fc1852c7600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:13.366 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:13.362+0000 7fc1852c7600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:13.381 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:51:13.382 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:51:13.382 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:51:13.383 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:51:13.383 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:51:13.384 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:51:13.384 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:13.384 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:13.385 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:51:13.385 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:13.385 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:51:13.385 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:13.386 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:13.386 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:51:13.386 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:13.387 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:51:13.388 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:13.388 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:51:13.389 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:13.389 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:13.390 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:13.391 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:13.391 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:51:13.392 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:13.393 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:13.394 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:51:13.394 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:13.395 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:13.395 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:13.396 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:51:13.396 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:13.397 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:13.397 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:13.399 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:13.399 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:51:13.400 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:13.400 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:13.401 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T07:51:13.401 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:13.402 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:13.402 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:13.403 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T07:51:13.403 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:13.404 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:13.404 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:13.405 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:13.406 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:51:13.406 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:13.407 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:13.407 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:51:13.408 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:13.408 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:13.408 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:13.409 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:51:13.409 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:13.410 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:13.410 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:13.411 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:51:13.411 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:13.412 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:51:13.412 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:51:13.413 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:51:13.413 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:13.413 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:51:13Z" 2022-09-13T07:51:13.413 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:51:13.414 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:51:13Z" } 2022-09-13T07:51:13.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:51:13.415 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:13.416 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:13.416 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.832045_205.1 2022-09-13T07:51:18.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:18.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:18.419 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:51:18.469 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:51:18.470 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:51:18.500 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:18.495+0000 7fb0cc987600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:18.501 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:18.496+0000 7fb0cc987600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:18.513 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:51:18.514 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:51:18.515 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:51:18.515 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:51:18.516 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:51:18.516 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:51:18.516 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:18.516 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:18.517 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:51:18.518 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:18.518 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:51:18.518 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:18.519 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:18.519 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:51:18.519 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:18.519 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:51:18.520 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:18.520 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:51:18.520 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:18.520 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:18.521 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:18.521 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:18.521 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:51:18.521 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:18.522 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:18.522 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:51:18.522 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:18.523 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:18.523 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:18.523 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:51:18.523 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:18.524 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:18.524 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:18.524 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:18.524 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:51:18.525 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:18.525 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:18.525 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.861945_210.1", 2022-09-13T07:51:18.525 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:18.526 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:18.526 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:18.526 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T07:51:18.526 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:18.527 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:18.527 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:18.527 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:18.527 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:51:18.528 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:18.528 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:18.528 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:51:18.528 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:18.529 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:18.529 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:18.529 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:51:18.529 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:18.530 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:18.530 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:18.530 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:51:18.530 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:18.531 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:51:18.531 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:51:18.531 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:51:18.532 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:18.532 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:51:18Z" 2022-09-13T07:51:18.532 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:51:18.533 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.861945_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.861945Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:51:18Z" } 2022-09-13T07:51:18.534 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:51:18.535 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:18.535 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:18.536 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.861945_210.1 2022-09-13T07:51:23.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:23.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:23.539 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:51:23.591 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:51:23.591 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:51:23.635 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:23.631+0000 7fe03d378600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:23.636 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:23.632+0000 7fe03d378600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:23.657 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:51:23.659 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:51:23.660 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:51:23.660 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:51:23.661 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:51:23.662 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:51:23.664 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:23.665 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:23.667 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:51:23.668 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:23.670 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:51:23.671 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:23.672 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:23.672 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:51:23.673 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:23.673 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:51:23.674 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:23.674 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:51:23.675 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:23.675 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:23.676 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:23.676 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:23.677 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:51:23.677 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:23.678 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:23.679 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:51:23.679 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:23.680 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:23.681 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:23.681 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:51:23.682 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:23.682 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:23.683 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:23.683 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:23.684 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:51:23.684 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:23.686 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:23.686 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.953175_216.1", 2022-09-13T07:51:23.687 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:23.687 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:23.688 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:23.689 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T07:51:23.690 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:23.690 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:23.691 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:23.691 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:23.692 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:51:23.692 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:23.693 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:23.693 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:51:23.694 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:23.694 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:23.695 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:23.695 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:51:23.696 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:23.697 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:23.697 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:23.697 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:51:23.698 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:23.698 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:51:23.699 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:51:23.699 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:51:23.699 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:23.699 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:51:23Z" 2022-09-13T07:51:23.700 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:51:23.700 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.953175_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.953175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:51:23Z" } 2022-09-13T07:51:23.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:51:23.701 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:23.702 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:23.702 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.953175_216.1 2022-09-13T07:51:28.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:28.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:28.706 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:51:28.763 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:51:28.764 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:51:28.817 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:28.813+0000 7eff283fe600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:28.817 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:28.814+0000 7eff283fe600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:28.840 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:51:28.840 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:51:28.841 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:51:28.841 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:51:28.841 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:51:28.841 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:51:28.842 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:28.842 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:28.842 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:51:28.843 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:28.843 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:51:28.843 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:28.843 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:28.843 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:51:28.844 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:28.844 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:51:28.844 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:28.845 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:51:28.845 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:28.845 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:28.845 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:28.846 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:28.846 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:51:28.846 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:28.846 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:28.847 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:51:28.847 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:28.847 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:28.847 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:28.848 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:51:28.848 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:28.848 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:28.848 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:28.849 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:28.849 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:51:28.849 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:28.849 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:28.849 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:51:28.850 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:28.850 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:28.850 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:28.850 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:51:28.851 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:28.851 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:28.851 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:28.851 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:28.852 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:51:28.852 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:28.852 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:28.852 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:51:28.853 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:28.853 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:28.853 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:28.854 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:51:28.854 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:28.854 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:28.854 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:28.855 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:51:28.855 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:28.855 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:51:28.856 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:51:28.856 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:51:28.856 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:28.856 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:51:28Z" 2022-09-13T07:51:28.857 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:51:28.858 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:51:28Z" } 2022-09-13T07:51:28.858 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:51:28.859 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:28.859 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:28.860 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.831134_204.1 2022-09-13T07:51:33.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:33.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:33.863 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:51:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:51:33.914 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:51:33.965 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:33.961+0000 7f24b092d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:33.966 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:33.962+0000 7f24b092d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:33.989 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:51:33.989 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:51:33.990 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:51:33.990 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:51:33.991 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:51:33.991 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:51:33.991 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:33.992 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:33.992 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:51:33.993 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:33.993 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:51:33.993 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:33.994 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:33.994 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:51:33.997 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:33.998 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:51:33.999 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:34.000 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:51:34.000 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:34.000 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:34.001 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:34.001 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:34.001 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:51:34.001 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:34.002 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:34.002 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:51:34.002 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:34.002 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:34.003 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:34.003 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:51:34.003 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:34.003 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:34.004 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:34.004 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:34.004 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:51:34.004 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:34.004 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:34.005 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.858000_209.1", 2022-09-13T07:51:34.005 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:34.005 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:34.005 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:34.006 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T07:51:34.006 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:34.006 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:34.006 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:34.007 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:34.007 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:51:34.007 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:34.007 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:34.008 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:51:34.008 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:34.008 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:34.009 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:34.009 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:51:34.009 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:34.009 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:34.010 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:34.011 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:51:34.011 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:34.011 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:51:34.011 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:51:34.012 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:51:34.012 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:34.012 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:51:33Z" 2022-09-13T07:51:34.012 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:51:34.013 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.858000_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.858000Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:51:33Z" } 2022-09-13T07:51:34.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:51:34.015 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:34.015 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:34.016 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.858000_209.1 2022-09-13T07:51:39.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:39.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:39.021 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:51:39.075 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:51:39.076 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:51:39.104 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:39.100+0000 7fef0a407600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:39.105 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:39.101+0000 7fef0a407600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:39.119 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:51:39.119 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:51:39.120 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:51:39.120 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:51:39.120 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:51:39.121 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:51:39.121 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:39.121 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:39.122 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:51:39.122 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:39.122 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:51:39.123 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:39.123 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:39.124 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:51:39.124 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:39.125 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:51:39.125 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:39.127 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:51:39.127 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:39.128 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:39.128 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:39.129 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:39.129 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:51:39.130 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:39.130 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:39.131 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:51:39.131 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:39.132 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:39.132 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:39.133 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:51:39.133 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:39.134 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:39.134 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:39.135 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:39.135 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:51:39.136 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:39.136 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:39.137 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:51:39.137 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:39.138 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:39.138 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:39.139 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:51:39.139 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:39.140 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:39.140 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:39.141 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:39.141 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:51:39.142 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:39.142 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:39.143 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:51:39.143 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:39.144 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:39.144 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:39.145 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:51:39.145 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:39.146 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:39.146 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:39.147 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:51:39.147 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:39.148 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:51:39.148 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:51:39.149 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:51:39.149 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:39.150 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:51:39Z" 2022-09-13T07:51:39.150 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:51:39.152 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:51:39Z" } 2022-09-13T07:51:39.153 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:51:39.154 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:39.155 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:39.156 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T07:51:44.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:44.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:44.160 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:51:44.213 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:51:44.214 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:51:44.246 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:44.239+0000 7f678cf71600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:44.247 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:44.240+0000 7f678cf71600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:44.257 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:51:44.257 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:51:44.258 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:51:44.258 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:51:44.258 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:51:44.258 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:51:44.259 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:44.259 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:44.259 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:51:44.260 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:44.260 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:51:44.260 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:44.260 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:44.261 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:51:44.261 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:44.261 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:51:44.261 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:44.262 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:51:44.262 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:44.262 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:44.262 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:44.263 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:44.263 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:51:44.263 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:44.263 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:44.264 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:51:44.264 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:44.264 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:44.264 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:44.265 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:51:44.265 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:44.265 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:44.265 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:44.266 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:44.266 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:51:44.266 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:44.266 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:44.267 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.900410_211.1", 2022-09-13T07:51:44.267 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:44.267 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:44.268 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:44.268 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T07:51:44.268 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:44.268 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:44.269 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:44.269 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:44.269 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:51:44.269 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:44.269 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:44.270 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:51:44.270 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:44.270 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:44.271 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:44.271 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:51:44.271 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:44.271 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:44.272 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:44.272 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:51:44.272 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:44.272 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:51:44.273 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:51:44.273 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:51:44.273 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:44.273 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:51:44Z" 2022-09-13T07:51:44.274 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:51:44.275 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.900410_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.900410Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:51:44Z" } 2022-09-13T07:51:44.276 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:51:44.276 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:44.277 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:44.277 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.900410_211.1 2022-09-13T07:51:49.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:49.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:49.280 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:51:49.333 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:51:49.334 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:51:49.366 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:49.361+0000 7f68b1cd8600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:49.367 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:49.362+0000 7f68b1cd8600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:49.381 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:51:49.382 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:51:49.382 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:51:49.383 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:51:49.383 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:51:49.384 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:51:49.384 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:49.385 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:49.385 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:51:49.386 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:49.386 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:51:49.387 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:49.388 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:49.388 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:51:49.389 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:49.389 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:51:49.390 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:49.390 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:51:49.391 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:49.391 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:49.392 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:49.392 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:49.393 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:51:49.393 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:49.394 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:49.394 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:51:49.395 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:49.395 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:49.396 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:49.397 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:51:49.398 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:49.398 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:49.399 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:49.399 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:49.400 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:51:49.400 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:49.401 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:49.401 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.111513_224.1", 2022-09-13T07:51:49.402 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:49.402 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:49.403 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:49.403 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.111513Z", 2022-09-13T07:51:49.404 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:49.404 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:49.405 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:49.406 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:49.406 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:51:49.407 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:49.407 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:49.408 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:51:49.408 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:49.408 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:49.409 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:49.409 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:51:49.409 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:49.409 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:49.410 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:49.410 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:51:49.410 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:49.410 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:51:49.411 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:51:49.411 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:51:49.411 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:49.411 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:51:49Z" 2022-09-13T07:51:49.412 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:51:49.413 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.111513_224.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.111513Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:51:49Z" } 2022-09-13T07:51:49.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.111513_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.111513Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:51:49.415 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:49.415 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.111513_224.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:49.416 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.111513_224.1 2022-09-13T07:51:54.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:54.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:54.419 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:51:54.479 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:51:54.480 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:51:54.513 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:54.509+0000 7efcc83d6600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:54.515 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:54.510+0000 7efcc83d6600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:54.531 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:51:54.531 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:51:54.532 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:51:54.532 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:51:54.532 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:51:54.533 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:51:54.533 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:54.535 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:54.535 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:51:54.536 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:54.537 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:51:54.537 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:54.538 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:54.539 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:51:54.539 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:54.540 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:51:54.540 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:54.540 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:51:54.541 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:54.541 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:54.542 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:54.542 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:54.543 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:51:54.543 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:54.544 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:54.544 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:51:54.545 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:54.545 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:54.546 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:54.546 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:51:54.546 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:54.547 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:54.547 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:54.548 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:54.548 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:51:54.549 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:54.549 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:54.550 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.858000_209.1", 2022-09-13T07:51:54.550 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:54.550 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:54.551 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:54.552 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T07:51:54.553 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:54.553 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:54.553 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:54.554 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:54.554 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:51:54.555 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:54.555 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:54.556 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:51:54.556 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:54.557 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:54.557 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:54.558 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:51:54.558 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:54.559 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:54.559 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:54.559 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:51:54.559 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:54.560 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:51:54.560 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:51:54.560 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:51:54.561 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:54.561 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:51:54Z" 2022-09-13T07:51:54.562 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:51:54.564 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.858000_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.858000Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:51:54Z" } 2022-09-13T07:51:54.565 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:51:54.566 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:54.566 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:54.567 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.858000_209.1 2022-09-13T07:51:59.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:59.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:51:59.571 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:51:59.628 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:51:59.628 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:51:59.739 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:59.734+0000 7ff95ed6a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:59.740 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:51:59.736+0000 7ff95ed6a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:51:59.791 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:51:59.792 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:51:59.793 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:51:59.794 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:51:59.794 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:51:59.795 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:51:59.795 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:59.796 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:59.796 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:51:59.797 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:59.797 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:51:59.798 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:59.799 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:59.799 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:51:59.800 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:59.801 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:51:59.801 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:59.802 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:51:59.802 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:59.803 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:59.804 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:59.804 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:59.805 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:51:59.805 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:59.806 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:59.806 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:51:59.807 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:59.807 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:59.808 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:59.809 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:51:59.810 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:59.810 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:59.811 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:59.811 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:59.812 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:51:59.812 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:59.813 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:59.813 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.900410_211.1", 2022-09-13T07:51:59.814 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:59.814 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:59.815 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:59.815 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T07:51:59.815 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:59.816 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:59.816 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:59.817 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:51:59.817 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:51:59.818 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:51:59.818 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:51:59.819 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:51:59.820 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:51:59.820 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:51:59.821 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:51:59.821 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:51:59.822 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:51:59.822 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:59.823 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:51:59.823 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:51:59.824 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:59.824 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:51:59.825 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:51:59.826 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:51:59.826 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:51:59.827 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:51:59Z" 2022-09-13T07:51:59.827 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:51:59.829 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.900410_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.900410Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:51:59Z" } 2022-09-13T07:51:59.830 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:51:59.831 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:59.832 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:51:59.833 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.900410_211.1 2022-09-13T07:52:04.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:04.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:04.837 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:52:04.890 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:52:04.891 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:52:04.930 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:04.925+0000 7f07b6211600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:04.931 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:04.926+0000 7f07b6211600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:04.947 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:52:04.947 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:52:04.947 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:52:04.948 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:52:04.948 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:52:04.949 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:52:04.949 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:04.949 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:04.950 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:52:04.950 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:04.950 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:52:04.951 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:04.951 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:04.952 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:52:04.952 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:04.952 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:52:04.952 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:04.953 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:52:04.953 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:04.953 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:04.953 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:04.954 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:04.954 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:52:04.955 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:04.955 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:04.956 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:52:04.956 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:04.957 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:04.957 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:04.958 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:52:04.958 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:04.959 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:04.959 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:04.960 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:04.960 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:52:04.961 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:04.961 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:04.962 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:52:04.962 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:04.963 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:04.964 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:04.964 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:52:04.965 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:04.965 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:04.966 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:04.966 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:04.967 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:52:04.967 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:04.968 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:04.968 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:52:04.969 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:04.969 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:04.970 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:04.970 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:52:04.971 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:04.971 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:04.971 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:04.972 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:52:04.972 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:04.973 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:52:04.973 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:52:04.973 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:52:04.974 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:04.974 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:52:04Z" 2022-09-13T07:52:04.974 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:52:04.975 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:52:04Z" } 2022-09-13T07:52:04.976 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:52:04.977 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:04.977 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:04.978 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T07:52:09.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:09.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:09.983 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:52:10.035 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:52:10.036 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:52:10.064 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:10.061+0000 7f10a0c99600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:10.065 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:10.062+0000 7f10a0c99600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:10.085 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:52:10.085 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:52:10.086 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:52:10.086 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:52:10.086 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:52:10.087 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:52:10.087 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:10.087 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:10.087 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:52:10.088 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:10.088 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:52:10.089 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:10.089 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:10.089 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:52:10.090 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:10.090 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:52:10.091 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:10.091 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:52:10.091 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:10.092 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:10.092 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:10.092 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:10.093 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:52:10.093 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:10.093 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:10.093 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:52:10.094 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:10.094 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:10.095 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:10.095 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:52:10.095 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:10.095 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:10.096 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:10.096 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:10.096 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:52:10.096 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:10.097 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:10.097 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:52:10.097 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:10.097 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:10.098 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:10.098 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:52:10.098 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:10.098 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:10.099 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:10.099 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:10.100 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:52:10.100 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:10.100 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:10.100 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:52:10.101 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:10.101 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:10.101 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:10.102 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:52:10.102 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:10.103 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:10.103 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:10.103 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:52:10.104 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:10.104 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:52:10.104 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:52:10.104 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:52:10.105 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:10.105 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:52:10Z" 2022-09-13T07:52:10.106 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:52:10.106 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:52:10Z" } 2022-09-13T07:52:10.107 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:52:10.108 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:10.108 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:10.109 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T07:52:15.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:15.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:15.112 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:52:15.165 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:52:15.166 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:52:15.209 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:15.206+0000 7f699eca7600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:15.210 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:15.207+0000 7f699eca7600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:15.228 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:52:15.229 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:52:15.229 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:52:15.229 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:52:15.230 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:52:15.230 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:52:15.230 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:15.231 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:15.231 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:52:15.231 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:15.231 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:52:15.232 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:15.232 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:15.232 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:52:15.232 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:15.233 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:52:15.233 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:15.233 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:52:15.233 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:15.234 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:15.234 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:15.234 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:15.234 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:52:15.235 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:15.235 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:15.235 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:52:15.236 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:15.236 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:15.236 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:15.237 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:52:15.237 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:15.237 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:15.238 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:15.238 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:15.238 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:52:15.239 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:15.239 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:15.239 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.070128_219.1", 2022-09-13T07:52:15.239 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:15.240 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:15.240 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:15.240 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T07:52:15.240 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:15.241 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:15.241 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:15.241 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:15.241 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:52:15.242 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:15.242 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:15.242 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:52:15.242 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:15.243 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:15.243 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:15.243 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:52:15.243 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:15.244 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:15.244 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:15.244 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:52:15.244 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:15.245 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:52:15.245 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:52:15.245 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:52:15.246 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:15.246 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:52:15Z" 2022-09-13T07:52:15.246 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:52:15.248 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.070128_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.070128Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:52:15Z" } 2022-09-13T07:52:15.249 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:52:15.249 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:15.250 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:15.250 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.070128_219.1 2022-09-13T07:52:20.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:20.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:20.254 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:52:20.333 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:52:20.334 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:52:20.380 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:20.375+0000 7fcb0118f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:20.381 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:20.376+0000 7fcb0118f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:20.393 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:52:20.394 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:52:20.395 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:52:20.395 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:52:20.396 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:52:20.396 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:52:20.396 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:20.397 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:20.397 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:52:20.397 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:20.397 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:52:20.398 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:20.398 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:20.398 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:52:20.398 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:20.399 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:52:20.400 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:20.401 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:52:20.401 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:20.401 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:20.402 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:20.402 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:20.403 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:52:20.403 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:20.404 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:20.404 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:52:20.405 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:20.405 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:20.405 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:20.406 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:52:20.406 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:20.407 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:20.407 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:20.408 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:20.408 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:52:20.409 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:20.409 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:20.410 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.857076_208.1", 2022-09-13T07:52:20.410 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:20.411 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:20.411 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:20.412 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T07:52:20.412 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:20.413 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:20.413 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:20.413 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:20.414 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:52:20.414 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:20.415 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:20.415 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:52:20.416 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:20.416 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:20.417 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:20.417 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:52:20.418 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:20.418 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:20.419 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:20.419 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:52:20.420 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:20.420 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:52:20.421 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:52:20.421 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:52:20.422 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:20.422 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:52:20Z" 2022-09-13T07:52:20.423 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:52:20.425 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.857076_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.857076Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:52:20Z" } 2022-09-13T07:52:20.426 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:52:20.427 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:20.428 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:20.429 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.857076_208.1 2022-09-13T07:52:25.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:25.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:25.432 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:52:25.488 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:52:25.489 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:52:25.533 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:25.529+0000 7fcb442d1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:25.534 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:25.530+0000 7fcb442d1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:25.551 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:52:25.551 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:52:25.552 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:52:25.552 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:52:25.552 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:52:25.553 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:52:25.554 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:25.554 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:25.555 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:52:25.555 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:25.556 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:52:25.556 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:25.557 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:25.557 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:52:25.557 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:25.558 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:52:25.559 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:25.559 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:52:25.560 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:25.560 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:25.561 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:25.561 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:25.561 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:52:25.562 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:25.562 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:25.563 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:52:25.563 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:25.564 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:25.564 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:25.565 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:52:25.565 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:25.566 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:25.566 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:25.567 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:25.568 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:52:25.568 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:25.569 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:25.569 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T07:52:25.570 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:25.571 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:25.571 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:25.572 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T07:52:25.572 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:25.573 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:25.574 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:25.574 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:25.574 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:52:25.575 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:25.575 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:25.575 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:52:25.576 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:25.577 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:25.577 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:25.578 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:52:25.578 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:25.579 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:25.579 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:25.580 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:52:25.580 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:25.581 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:52:25.581 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:52:25.582 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:52:25.582 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:25.583 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:52:25Z" 2022-09-13T07:52:25.583 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:52:25.585 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:52:25Z" } 2022-09-13T07:52:25.585 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:52:25.586 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:25.587 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:25.587 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.949151_215.1 2022-09-13T07:52:30.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:30.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:30.590 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:52:30.641 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:52:30.642 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:52:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:30.665+0000 7f89b0222600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:30.671 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:30.667+0000 7f89b0222600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:30.687 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:52:30.687 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:52:30.688 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:52:30.688 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:52:30.690 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:52:30.691 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:52:30.691 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:30.692 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:30.692 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:52:30.693 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:30.694 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:52:30.694 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:30.695 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:30.695 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:52:30.696 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:30.697 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:52:30.698 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:30.698 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:52:30.699 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:30.700 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:30.700 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:30.701 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:30.701 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:52:30.701 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:30.701 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:30.702 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:52:30.702 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:30.702 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:30.703 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:30.703 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:52:30.703 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:30.703 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:30.703 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:30.704 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:30.704 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:52:30.704 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:30.704 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:30.705 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:52:30.705 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:30.705 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:30.705 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:30.706 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:52:30.706 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:30.706 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:30.706 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:30.707 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:30.707 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:52:30.707 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:30.707 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:30.708 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:52:30.708 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:30.708 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:30.708 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:30.709 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:52:30.709 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:30.709 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:30.710 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:30.710 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:52:30.710 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:30.711 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:52:30.711 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:52:30.711 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:52:30.712 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:30.712 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:52:30Z" 2022-09-13T07:52:30.712 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:52:30.713 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:52:30Z" } 2022-09-13T07:52:30.714 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:52:30.714 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:30.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:30.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.831134_204.1 2022-09-13T07:52:35.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:35.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:35.718 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:52:35.773 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:52:35.774 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:52:35.807 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:35.801+0000 7f92c737a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:35.808 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:35.802+0000 7f92c737a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:35.820 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:52:35.820 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:52:35.821 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:52:35.821 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:52:35.821 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:52:35.822 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:52:35.822 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:35.823 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:35.823 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:52:35.824 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:35.824 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:52:35.825 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:35.826 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:35.826 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:52:35.827 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:35.827 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:52:35.828 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:35.828 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:52:35.829 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:35.829 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:35.829 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:35.830 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:35.830 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:52:35.831 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:35.831 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:35.832 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:52:35.832 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:35.833 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:35.833 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:35.834 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:52:35.834 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:35.835 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:35.835 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:35.836 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:35.836 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:52:35.837 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:35.837 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:35.837 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:52:35.838 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:35.838 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:35.839 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:35.839 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:52:35.840 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:35.840 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:35.841 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:35.841 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:35.842 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:52:35.842 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:35.843 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:35.843 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:52:35.843 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:35.844 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:35.844 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:35.845 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:52:35.845 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:35.845 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:35.846 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:35.846 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:52:35.846 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:35.847 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:52:35.847 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:52:35.848 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:52:35.848 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:35.848 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:52:35Z" 2022-09-13T07:52:35.849 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:52:35.850 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:52:35Z" } 2022-09-13T07:52:35.851 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:52:35.851 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:35.852 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:35.852 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T07:52:40.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:40.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:40.855 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:52:40.948 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:52:40.948 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:52:41.041 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:41.036+0000 7fb79d045600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:41.037+0000 7fb79d045600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:41.067 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:52:41.067 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:52:41.067 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:52:41.068 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:52:41.068 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:52:41.068 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:52:41.068 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:41.069 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:41.069 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:52:41.069 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:41.069 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:52:41.070 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:41.070 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:41.070 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:52:41.071 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:41.071 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:52:41.071 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:41.072 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:52:41.072 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:41.072 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:41.073 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:41.073 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:41.073 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:52:41.074 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:41.074 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:41.074 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:52:41.075 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:41.075 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:41.075 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:41.075 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:52:41.076 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:41.076 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:41.076 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:41.076 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:41.077 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:52:41.077 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:41.077 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:41.077 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.907374_213.1", 2022-09-13T07:52:41.078 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:41.078 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:41.078 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:41.078 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.907374Z", 2022-09-13T07:52:41.079 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:41.079 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:41.079 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:41.079 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:41.080 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:52:41.080 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:41.080 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:41.081 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:52:41.081 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:41.082 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:41.082 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:41.082 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:52:41.082 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:41.083 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:41.083 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:41.083 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:52:41.084 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:41.084 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:52:41.084 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:52:41.084 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:52:41.085 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:41.085 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:52:41Z" 2022-09-13T07:52:41.085 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:52:41.086 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.907374_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.907374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:52:41Z" } 2022-09-13T07:52:41.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.907374_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.907374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:52:41.088 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:41.088 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.907374_213.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:41.089 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.907374_213.1 2022-09-13T07:52:46.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:46.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:46.093 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:52:46.143 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:52:46.143 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:52:46.172 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:46.167+0000 7fd51def5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:46.173 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:46.168+0000 7fd51def5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:46.185 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:52:46.186 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:52:46.186 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:52:46.186 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:52:46.187 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:52:46.187 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:52:46.187 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:46.188 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:46.188 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:52:46.188 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:46.188 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:52:46.188 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:46.189 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:46.189 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:52:46.189 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:46.189 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:52:46.190 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:46.190 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:52:46.190 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:46.191 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:46.191 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:46.191 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:46.191 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:52:46.192 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:46.192 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:46.192 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:52:46.192 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:46.193 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:46.193 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:46.193 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:52:46.193 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:46.193 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:46.194 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:46.194 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:46.195 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:52:46.195 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:46.195 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:46.195 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T07:52:46.196 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:46.196 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:46.196 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:46.197 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T07:52:46.197 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:46.198 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:46.198 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:46.198 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:46.199 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:52:46.199 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:46.200 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:46.200 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:52:46.201 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:46.201 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:46.202 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:46.202 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:52:46.203 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:46.203 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:46.203 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:46.204 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:52:46.204 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:46.204 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:52:46.205 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:52:46.205 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:52:46.206 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:46.206 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:52:46Z" 2022-09-13T07:52:46.206 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:52:46.208 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:52:46Z" } 2022-09-13T07:52:46.208 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:52:46.209 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:46.209 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:46.210 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.832045_205.1 2022-09-13T07:52:51.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:51.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:51.213 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:52:51.269 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:52:51.270 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:52:51.305 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:51.300+0000 7f07d162e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:51.306 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:51.301+0000 7f07d162e600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:51.321 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:52:51.321 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:52:51.322 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:52:51.322 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:52:51.322 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:52:51.323 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:52:51.323 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:51.323 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:51.323 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:52:51.324 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:51.324 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:52:51.324 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:51.325 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:51.325 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:52:51.325 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:51.325 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:52:51.326 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:51.326 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:52:51.326 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:51.327 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:51.327 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:51.327 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:51.327 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:52:51.328 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:51.328 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:51.329 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:52:51.329 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:51.329 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:51.330 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:51.330 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:52:51.330 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:51.331 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:51.331 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:51.331 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:51.331 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:52:51.332 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:51.332 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:51.332 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T07:52:51.333 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:51.333 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:51.333 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:51.333 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T07:52:51.334 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:51.334 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:51.335 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:51.335 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:51.335 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:52:51.335 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:51.336 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:51.336 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:52:51.336 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:51.336 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:51.337 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:51.337 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:52:51.337 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:51.337 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:51.338 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:51.338 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:52:51.338 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:51.338 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:52:51.339 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:52:51.339 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:52:51.340 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:51.340 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:52:51Z" 2022-09-13T07:52:51.341 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:52:51.341 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:52:51Z" } 2022-09-13T07:52:51.342 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:52:51.343 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:51.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:51.345 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.832045_205.1 2022-09-13T07:52:56.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:56.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:52:56.347 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:52:56.399 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:52:56.399 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:52:56.429 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:56.426+0000 7ff1df136600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:56.430 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:52:56.427+0000 7ff1df136600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:52:56.443 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:52:56.443 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:52:56.443 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:52:56.444 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:52:56.444 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:52:56.445 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:52:56.445 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:56.445 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:56.445 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:52:56.446 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:56.446 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:52:56.446 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:56.447 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:56.447 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:52:56.447 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:56.448 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:52:56.448 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:56.448 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:52:56.448 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:56.449 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:56.449 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:56.449 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:56.449 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:52:56.450 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:56.450 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:56.450 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:52:56.451 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:56.451 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:56.451 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:56.452 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:52:56.452 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:56.452 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:56.453 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:56.453 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:56.453 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:52:56.453 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:56.454 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:56.454 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.070128_219.1", 2022-09-13T07:52:56.454 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:56.455 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:56.455 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:56.455 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T07:52:56.455 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:56.456 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:56.456 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:56.456 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:52:56.457 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:52:56.457 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:52:56.457 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:52:56.457 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:52:56.458 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:52:56.458 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:52:56.458 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:52:56.458 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:52:56.459 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:52:56.459 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:56.460 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:52:56.460 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:52:56.460 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:56.461 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:52:56.461 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:52:56.461 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:52:56.462 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:52:56.462 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:52:56Z" 2022-09-13T07:52:56.462 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:52:56.464 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.070128_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.070128Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:52:56Z" } 2022-09-13T07:52:56.464 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:52:56.465 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:56.466 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:52:56.466 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.070128_219.1 2022-09-13T07:53:01.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:01.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:01.469 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:53:01.521 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:53:01.521 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:53:01.553 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:01.550+0000 7fb3c4093600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:01.554 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:01.551+0000 7fb3c4093600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:01.602 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:53:01.602 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:53:01.603 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:53:01.603 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:53:01.603 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:53:01.604 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:53:01.604 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:01.604 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:01.605 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:53:01.605 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:01.605 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:53:01.605 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:01.606 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:01.606 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:53:01.607 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:01.607 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:53:01.607 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:01.608 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:53:01.608 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:01.609 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:01.610 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:01.611 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:01.611 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:53:01.612 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:01.612 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:01.613 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:53:01.614 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:01.614 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:01.615 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:01.615 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:53:01.615 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:01.616 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:01.616 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:01.616 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:01.616 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:53:01.617 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:01.617 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:01.618 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.072712_220.1", 2022-09-13T07:53:01.618 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:01.618 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:01.618 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:01.619 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.072712Z", 2022-09-13T07:53:01.619 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:01.619 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:01.620 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:01.620 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:01.620 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:53:01.620 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:01.621 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:01.621 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:53:01.621 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:01.621 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:01.622 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:01.622 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:53:01.622 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:01.623 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:01.623 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:01.623 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:53:01.623 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:01.624 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:53:01.624 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:53:01.624 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:53:01.624 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:01.625 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:53:01Z" 2022-09-13T07:53:01.625 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:53:01.626 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.072712_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.072712Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:53:01Z" } 2022-09-13T07:53:01.627 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.072712_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.072712Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:53:01.628 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:01.628 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.072712_220.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:01.629 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.072712_220.1 2022-09-13T07:53:06.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:06.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:06.631 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:53:06.713 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:53:06.713 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:53:06.763 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:06.758+0000 7ff625af5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:06.764 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:06.759+0000 7ff625af5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:06.779 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:53:06.780 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:53:06.780 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:53:06.780 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:53:06.780 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:53:06.781 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:53:06.781 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:06.781 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:06.781 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:53:06.782 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:06.782 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:53:06.782 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:06.782 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:06.783 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:53:06.783 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:06.783 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:53:06.783 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:06.784 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:53:06.784 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:06.784 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:06.784 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:06.785 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:06.785 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:53:06.785 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:06.785 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:06.786 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:53:06.786 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:06.786 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:06.786 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:06.787 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:53:06.787 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:06.787 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:06.788 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:06.788 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:06.788 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:53:06.788 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:06.789 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:06.789 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.906479_212.1", 2022-09-13T07:53:06.789 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:06.789 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:06.790 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:06.790 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.906479Z", 2022-09-13T07:53:06.790 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:06.791 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:06.791 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:06.791 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:06.791 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:53:06.791 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:06.792 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:06.792 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:53:06.792 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:06.792 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:06.793 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:06.793 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:53:06.793 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:06.793 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:06.794 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:06.794 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:53:06.794 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:06.794 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:53:06.795 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:53:06.795 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:53:06.795 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:06.795 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:53:06Z" 2022-09-13T07:53:06.796 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:53:06.797 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.906479_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.906479Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:53:06Z" } 2022-09-13T07:53:06.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.906479_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.906479Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:53:06.798 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:06.799 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.906479_212.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:06.799 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.906479_212.1 2022-09-13T07:53:11.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:11.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:11.802 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:53:11.853 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:53:11.853 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:53:11.892 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:11.887+0000 7fc694a25600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:11.893 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:11.888+0000 7fc694a25600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:11.908 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:53:11.908 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:53:11.908 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:53:11.909 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:53:11.909 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:53:11.909 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:53:11.909 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:11.910 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:11.910 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:53:11.910 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:11.911 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:53:11.911 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:11.911 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:11.912 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:53:11.912 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:11.912 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:53:11.913 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:11.913 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:53:11.913 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:11.913 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:11.914 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:11.914 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:11.914 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:53:11.914 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:11.915 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:11.915 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:53:11.915 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:11.915 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:11.916 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:11.916 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:53:11.916 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:11.916 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:11.917 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:11.917 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:11.917 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:53:11.917 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:11.918 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:11.918 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:53:11.918 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:11.918 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:11.919 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:11.919 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:53:11.919 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:11.919 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:11.919 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:11.920 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:11.920 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:53:11.920 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:11.920 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:11.921 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:53:11.921 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:11.921 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:11.921 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:11.922 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:53:11.922 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:11.923 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:11.923 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:11.923 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:53:11.923 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:11.924 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:53:11.924 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:53:11.924 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:53:11.924 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:11.925 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:53:11Z" 2022-09-13T07:53:11.925 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:53:11.926 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:53:11Z" } 2022-09-13T07:53:11.927 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:53:11.927 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:11.928 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:11.928 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.831134_204.1 2022-09-13T07:53:16.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:16.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:16.932 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:53:16.981 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:53:16.981 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:53:17.009 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:17.004+0000 7fde16b67600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:17.010 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:17.006+0000 7fde16b67600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:17.027 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:53:17.028 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:53:17.028 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:53:17.028 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:53:17.029 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:53:17.029 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:53:17.029 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:17.029 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:17.030 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:53:17.030 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:17.030 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:53:17.030 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:17.031 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:17.031 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:53:17.031 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:17.031 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:53:17.032 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:17.032 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:53:17.032 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:17.033 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:17.033 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:17.033 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:17.034 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:53:17.034 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:17.034 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:17.034 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:53:17.035 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:17.035 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:17.035 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:17.035 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:53:17.036 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:17.036 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:17.036 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:17.036 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:17.037 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:53:17.037 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:17.037 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:17.037 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.857076_208.1", 2022-09-13T07:53:17.038 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:17.038 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:17.038 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:17.038 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T07:53:17.039 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:17.039 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:17.039 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:17.039 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:17.040 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:53:17.040 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:17.040 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:17.040 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:53:17.041 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:17.041 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:17.041 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:17.041 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:53:17.042 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:17.042 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:17.042 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:17.042 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:53:17.043 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:17.043 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:53:17.043 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:53:17.043 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:53:17.044 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:17.044 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:53:17Z" 2022-09-13T07:53:17.044 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:53:17.046 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.857076_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.857076Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:53:17Z" } 2022-09-13T07:53:17.047 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:53:17.048 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:17.048 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:17.048 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.857076_208.1 2022-09-13T07:53:22.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:22.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:22.052 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:53:22.102 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:53:22.103 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:53:22.129 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:22.125+0000 7f3d35434600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:22.131 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:22.126+0000 7f3d35434600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:22.143 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:53:22.144 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:53:22.145 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:53:22.145 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:53:22.147 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:53:22.147 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:53:22.148 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:22.148 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:22.149 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:53:22.150 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:22.150 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:53:22.151 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:22.151 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:22.152 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:53:22.152 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:22.153 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:53:22.153 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:22.154 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:53:22.154 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:22.155 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:22.155 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:22.156 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:22.156 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:53:22.157 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:22.158 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:22.158 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:53:22.158 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:22.159 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:22.160 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:22.160 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:53:22.161 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:22.161 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:22.162 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:22.162 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:22.163 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:53:22.163 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:22.164 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:22.164 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.857076_208.1", 2022-09-13T07:53:22.165 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:22.165 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:22.166 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:22.166 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T07:53:22.167 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:22.167 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:22.168 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:22.168 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:22.169 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:53:22.169 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:22.170 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:22.170 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:53:22.171 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:22.172 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:22.172 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:22.173 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:53:22.173 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:22.174 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:22.174 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:22.175 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:53:22.175 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:22.176 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:53:22.176 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:53:22.177 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:53:22.177 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:22.178 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:53:22Z" 2022-09-13T07:53:22.179 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:53:22.181 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.857076_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.857076Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:53:22Z" } 2022-09-13T07:53:22.182 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:53:22.183 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:22.184 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:22.185 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.857076_208.1 2022-09-13T07:53:27.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:27.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:27.188 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:53:27.239 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:53:27.240 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:53:27.276 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:27.272+0000 7f90694a3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:27.277 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:27.273+0000 7f90694a3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:27.293 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:53:27.294 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:53:27.294 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:53:27.295 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:53:27.296 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:53:27.296 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:53:27.297 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:27.297 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:27.298 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:53:27.298 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:27.299 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:53:27.299 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:27.300 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:27.300 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:53:27.300 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:27.301 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:53:27.301 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:27.301 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:53:27.301 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:27.302 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:27.302 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:27.303 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:27.303 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:53:27.304 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:27.304 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:27.304 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:53:27.305 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:27.305 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:27.306 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:27.306 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:53:27.307 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:27.307 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:27.309 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:27.309 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:27.310 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:53:27.310 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:27.310 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:27.311 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.953175_216.1", 2022-09-13T07:53:27.311 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:27.311 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:27.311 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:27.312 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T07:53:27.312 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:27.313 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:27.313 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:27.313 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:27.314 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:53:27.314 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:27.314 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:27.314 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:53:27.315 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:27.315 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:27.316 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:27.316 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:53:27.316 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:27.317 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:27.317 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:27.317 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:53:27.317 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:27.318 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:53:27.318 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:53:27.318 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:53:27.319 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:27.319 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:53:27Z" 2022-09-13T07:53:27.319 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:53:27.321 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.953175_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.953175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:53:27Z" } 2022-09-13T07:53:27.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:53:27.322 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:27.322 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:27.323 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.953175_216.1 2022-09-13T07:53:32.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:32.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:32.325 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:53:32.381 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:53:32.382 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:53:32.418 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:32.414+0000 7f60832e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:32.418 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:32.415+0000 7f60832e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:32.433 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:53:32.433 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:53:32.434 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:53:32.434 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:53:32.434 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:53:32.434 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:53:32.435 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:32.435 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:32.435 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:53:32.435 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:32.436 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:53:32.436 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:32.436 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:32.436 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:53:32.437 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:32.437 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:53:32.437 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:32.437 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:53:32.438 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:32.438 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:32.438 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:32.438 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:53:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:53:32.440 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:32.440 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:32.440 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:32.440 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:53:32.441 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:32.441 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:32.441 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:32.441 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:32.442 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:53:32.442 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:32.442 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:32.442 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.854520_207.1", 2022-09-13T07:53:32.443 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:32.443 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:32.443 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:32.443 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T07:53:32.444 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:32.444 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:32.444 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:32.444 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:32.445 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:53:32.445 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:32.445 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:32.445 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:53:32.446 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:32.446 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:32.446 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:32.446 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:53:32.447 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:32.447 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:32.447 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:32.447 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:53:32.448 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:32.448 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:53:32.448 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:53:32.448 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:53:32.449 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:32.449 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:53:32Z" 2022-09-13T07:53:32.449 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:53:32.450 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.854520_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.854520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:53:32Z" } 2022-09-13T07:53:32.451 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:53:32.452 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:32.452 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:32.452 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.854520_207.1 2022-09-13T07:53:37.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:37.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:37.457 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:53:37.527 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:53:37.528 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:53:37.652 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:37.647+0000 7f6ae54dd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:37.649+0000 7f6ae54dd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:37.675 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:53:37.675 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:53:37.676 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:53:37.676 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:53:37.676 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:53:37.677 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:53:37.677 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:37.677 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:37.677 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:53:37.678 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:37.678 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:53:37.679 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:37.679 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:37.680 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:53:37.680 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:37.681 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:53:37.681 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:37.683 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:53:37.683 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:37.684 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:37.684 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:37.685 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:37.685 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:53:37.686 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:37.686 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:37.686 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:53:37.687 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:37.687 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:37.689 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:37.690 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:53:37.691 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:37.692 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:37.692 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:37.693 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:37.693 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:53:37.693 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:37.694 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:37.704 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T07:53:37.704 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:37.704 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:37.705 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:37.705 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T07:53:37.705 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:37.706 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:37.706 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:37.707 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:37.707 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:53:37.707 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:37.708 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:37.708 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:53:37.708 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:37.708 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:37.709 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:37.709 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:53:37.709 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:37.709 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:37.710 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:37.710 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:53:37.710 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:37.710 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:53:37.711 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:53:37.711 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:53:37.711 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:37.712 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:53:37Z" 2022-09-13T07:53:37.712 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:53:37.715 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:53:37Z" } 2022-09-13T07:53:37.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:53:37.717 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:37.718 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:37.718 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.832045_205.1 2022-09-13T07:53:42.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:42.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:42.722 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:53:42.772 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:53:42.773 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:53:42.811 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:42.808+0000 7fc163459600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:42.812 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:42.809+0000 7fc163459600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:42.829 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:53:42.830 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:53:42.830 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:53:42.830 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:53:42.830 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:53:42.831 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:53:42.831 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:42.831 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:42.831 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:53:42.832 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:42.832 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:53:42.832 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:42.833 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:42.833 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:53:42.833 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:42.833 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:53:42.834 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:42.834 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:53:42.834 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:42.835 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:42.835 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:42.835 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:42.835 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:53:42.836 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:42.836 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:42.836 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:53:42.836 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:42.837 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:42.837 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:42.837 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:53:42.837 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:42.838 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:42.838 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:42.838 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:42.839 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:53:42.839 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:42.839 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:42.840 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:53:42.840 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:42.840 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:42.840 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:42.841 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:53:42.841 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:42.841 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:42.841 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:42.842 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:42.842 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:53:42.842 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:42.842 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:42.843 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:53:42.843 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:42.843 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:42.843 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:42.844 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:53:42.844 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:42.844 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:42.844 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:42.845 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:53:42.845 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:42.845 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:53:42.845 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:53:42.846 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:53:42.846 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:42.846 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:53:42Z" 2022-09-13T07:53:42.847 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:53:42.848 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:53:42Z" } 2022-09-13T07:53:42.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:53:42.850 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:42.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:42.851 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.831134_204.1 2022-09-13T07:53:47.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:47.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:47.853 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:53:47.906 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:53:47.906 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:53:47.936 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:47.931+0000 7f098e11a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:47.937 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:47.932+0000 7f098e11a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:47.950 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:53:47.950 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:53:47.951 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:53:47.951 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:53:47.951 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:53:47.951 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:53:47.952 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:47.952 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:47.952 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:53:47.953 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:47.953 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:53:47.953 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:47.953 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:47.953 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:53:47.954 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:47.954 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:53:47.954 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:47.954 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:53:47.955 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:47.955 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:47.955 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:47.956 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:47.956 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:53:47.956 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:47.956 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:47.957 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:53:47.957 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:47.957 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:47.957 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:47.958 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:53:47.958 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:47.958 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:47.958 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:47.959 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:47.959 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:53:47.959 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:47.959 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:47.960 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T07:53:47.960 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:47.960 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:47.960 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:47.961 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T07:53:47.961 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:47.961 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:47.961 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:47.962 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:47.962 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:53:47.962 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:47.963 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:47.963 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:53:47.963 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:47.964 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:47.964 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:47.964 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:53:47.965 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:47.965 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:47.965 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:47.966 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:53:47.966 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:47.967 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:53:47.967 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:53:47.968 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:53:47.968 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:47.969 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:53:47Z" 2022-09-13T07:53:47.969 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:53:47.971 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:53:47Z" } 2022-09-13T07:53:47.972 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:53:47.973 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:47.973 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:47.974 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.141531_226.1 2022-09-13T07:53:52.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:52.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:52.978 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:53:53.035 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:53:53.035 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:53:53.078 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:53.073+0000 7ff1589e8600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:53.079 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:53.075+0000 7ff1589e8600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:53.095 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:53:53.095 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:53:53.095 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:53:53.096 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:53:53.096 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:53:53.096 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:53:53.096 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:53.097 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:53.097 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:53:53.097 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:53.097 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:53:53.098 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:53.098 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:53.098 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:53:53.098 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:53.099 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:53:53.099 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:53.099 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:53:53.100 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:53.100 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:53.100 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:53.101 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:53.102 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:53:53.102 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:53.102 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:53.103 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:53:53.103 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:53.104 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:53.105 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:53.105 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:53:53.105 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:53.106 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:53.106 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:53.107 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:53.107 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:53:53.108 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:53.108 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:53.108 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.858000_209.1", 2022-09-13T07:53:53.109 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:53.109 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:53.109 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:53.109 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T07:53:53.110 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:53.110 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:53.111 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:53.111 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:53.112 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:53:53.112 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:53.113 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:53.113 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:53:53.114 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:53.114 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:53.115 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:53.115 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:53:53.116 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:53.116 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:53.117 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:53.118 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:53:53.118 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:53.119 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:53:53.119 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:53:53.120 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:53:53.121 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:53.121 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:53:53Z" 2022-09-13T07:53:53.122 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:53:53.124 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.858000_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.858000Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:53:53Z" } 2022-09-13T07:53:53.125 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:53:53.126 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:53.127 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:53.128 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.858000_209.1 2022-09-13T07:53:58.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:58.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:53:58.132 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:53:58.236 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:53:58.237 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:53:58.318 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:58.313+0000 7efdda06d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:58.319 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:53:58.314+0000 7efdda06d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:53:58.340 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:53:58.341 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:53:58.341 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:53:58.342 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:53:58.342 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:53:58.342 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:53:58.343 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:58.343 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:58.343 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:53:58.343 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:58.344 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:53:58.344 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:58.344 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:58.344 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:53:58.345 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:58.345 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:53:58.346 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:58.346 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:53:58.346 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:58.346 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:58.347 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:58.347 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:58.347 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:53:58.348 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:58.348 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:58.348 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:53:58.349 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:58.349 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:58.349 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:58.350 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:53:58.350 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:58.351 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:58.351 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:58.351 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:58.352 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:53:58.352 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:58.353 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:58.353 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:53:58.354 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:58.354 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:58.355 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:58.355 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:53:58.356 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:58.356 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:58.357 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:58.357 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:53:58.358 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:53:58.358 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:53:58.359 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:53:58.359 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:53:58.360 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:53:58.360 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:53:58.361 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:53:58.361 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:53:58.361 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:53:58.361 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:58.362 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:53:58.362 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:53:58.362 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:58.363 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:53:58.363 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:53:58.363 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:53:58.363 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:53:58.364 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:53:58Z" 2022-09-13T07:53:58.364 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:53:58.365 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:53:58Z" } 2022-09-13T07:53:58.366 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:53:58.367 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:58.367 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:53:58.368 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.831134_204.1 2022-09-13T07:54:03.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:03.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:03.371 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:54:03.421 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:54:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:54:03.488 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:03.484+0000 7f2407785600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:03.489 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:03.485+0000 7f2407785600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:03.514 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:54:03.514 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:54:03.515 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:54:03.515 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:54:03.515 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:54:03.516 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:54:03.516 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:03.516 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:03.517 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:54:03.517 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:03.517 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:54:03.517 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:03.518 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:03.518 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:54:03.518 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:03.530 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:54:03.530 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:03.531 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:54:03.531 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:03.531 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:03.532 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:03.532 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:03.532 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:54:03.532 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:03.533 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:03.533 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:54:03.533 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:03.533 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:03.534 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:03.534 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:54:03.534 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:03.535 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:03.535 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:03.535 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:03.535 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:54:03.536 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:03.536 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:03.536 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.900410_211.1", 2022-09-13T07:54:03.536 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:03.537 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:03.537 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:03.537 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T07:54:03.537 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:03.538 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:03.538 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:03.538 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:03.539 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:54:03.539 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:03.539 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:03.539 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:54:03.540 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:03.540 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:03.540 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:03.540 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:54:03.541 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:03.541 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:03.541 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:03.542 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:54:03.542 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:03.542 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:54:03.542 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:54:03.543 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:54:03.543 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:03.543 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:54:03Z" 2022-09-13T07:54:03.543 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:54:03.545 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.900410_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.900410Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:54:03Z" } 2022-09-13T07:54:03.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:54:03.546 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:03.546 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:03.547 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.900410_211.1 2022-09-13T07:54:08.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:08.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:08.551 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:54:08.601 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:54:08.601 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:54:08.636 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:08.632+0000 7fe553e20600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:08.637 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:08.633+0000 7fe553e20600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:08.660 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:54:08.660 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:54:08.660 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:54:08.661 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:54:08.661 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:54:08.661 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:54:08.662 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:08.662 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:08.662 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:54:08.663 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:08.663 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:54:08.663 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:08.663 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:08.664 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:54:08.664 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:08.664 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:54:08.665 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:08.665 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:54:08.666 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:08.666 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:08.666 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:08.666 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:08.667 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:54:08.667 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:08.667 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:08.667 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:54:08.668 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:08.668 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:08.668 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:08.668 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:54:08.669 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:08.669 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:08.669 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:08.669 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:08.670 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:54:08.670 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:08.671 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:08.671 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.854520_207.1", 2022-09-13T07:54:08.671 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:08.671 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:08.672 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:08.672 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T07:54:08.672 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:08.673 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:08.673 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:08.673 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:08.673 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:54:08.674 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:08.674 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:08.674 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:54:08.674 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:08.675 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:08.675 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:08.675 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:54:08.676 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:08.676 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:08.676 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:08.676 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:54:08.677 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:08.677 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:54:08.677 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:54:08.677 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:54:08.678 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:08.678 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:54:08Z" 2022-09-13T07:54:08.678 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:54:08.680 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.854520_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.854520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:54:08Z" } 2022-09-13T07:54:08.680 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:54:08.681 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:08.681 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:08.682 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.854520_207.1 2022-09-13T07:54:13.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:13.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:13.686 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:54:13.736 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:54:13.737 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:54:13.782 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:13.778+0000 7f1643e64600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:13.783 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:13.779+0000 7f1643e64600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:13.802 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:54:13.803 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:54:13.803 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:54:13.803 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:54:13.804 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:54:13.804 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:54:13.804 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:13.805 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:13.805 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:54:13.805 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:13.805 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:54:13.806 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:13.806 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:13.806 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:54:13.807 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:13.807 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:54:13.807 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:13.807 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:54:13.808 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:13.808 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:13.808 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:13.808 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:13.809 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:54:13.809 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:13.809 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:13.809 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:54:13.810 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:13.810 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:13.810 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:13.810 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:54:13.811 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:13.811 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:13.811 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:13.811 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:13.812 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:54:13.812 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:13.812 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:13.812 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T07:54:13.813 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:13.813 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:13.813 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:13.813 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T07:54:13.814 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:13.814 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:13.814 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:13.814 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:13.815 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:54:13.815 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:13.815 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:13.815 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:54:13.816 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:13.816 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:13.816 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:13.817 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:54:13.817 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:13.817 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:13.818 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:13.818 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:54:13.818 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:13.818 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:54:13.819 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:54:13.819 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:54:13.819 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:13.819 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:54:13Z" 2022-09-13T07:54:13.820 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:54:13.821 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:54:13Z" } 2022-09-13T07:54:13.822 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:54:13.823 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:13.823 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:13.824 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.832045_205.1 2022-09-13T07:54:18.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:18.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:18.826 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:54:18.880 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:54:18.881 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:54:18.925 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:18.922+0000 7f30f33be600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:18.926 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:18.923+0000 7f30f33be600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:18.940 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:54:18.941 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:54:18.941 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:54:18.942 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:54:18.943 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:54:18.943 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:54:18.944 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:18.944 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:18.945 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:54:18.945 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:18.946 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:54:18.947 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:18.947 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:18.948 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:54:18.948 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:18.949 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:54:18.949 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:18.949 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:54:18.950 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:18.950 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:18.950 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:18.951 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:18.951 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:54:18.951 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:18.951 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:18.952 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:54:18.952 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:18.952 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:18.952 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:18.952 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:54:18.953 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:18.953 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:18.953 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:18.954 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:18.954 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:54:18.954 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:18.954 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:18.955 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:54:18.955 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:18.955 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:18.955 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:18.956 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:54:18.956 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:18.956 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:18.956 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:18.957 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:18.957 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:54:18.957 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:18.957 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:18.958 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:54:18.958 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:18.958 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:18.958 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:18.959 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:54:18.959 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:18.959 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:18.959 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:18.960 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:54:18.960 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:18.960 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:54:18.960 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:54:18.961 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:54:18.961 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:18.961 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:54:18Z" 2022-09-13T07:54:18.961 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:54:18.963 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:54:18Z" } 2022-09-13T07:54:18.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:54:18.964 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:18.964 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:18.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.831134_204.1 2022-09-13T07:54:23.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:23.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:23.968 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:54:24.056 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:54:24.057 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:54:24.119 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:24.116+0000 7f07898e8600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:24.120 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:24.117+0000 7f07898e8600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:24.133 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:54:24.134 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:54:24.134 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:54:24.134 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:54:24.135 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:54:24.135 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:54:24.135 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:24.136 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:24.136 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:54:24.136 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:24.137 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:54:24.137 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:24.137 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:24.137 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:54:24.138 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:24.138 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:54:24.138 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:24.138 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:54:24.139 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:24.139 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:24.139 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:24.139 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:24.140 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:54:24.140 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:24.140 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:24.140 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:54:24.141 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:24.141 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:24.141 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:24.142 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:54:24.142 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:24.142 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:24.142 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:24.143 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:24.143 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:54:24.143 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:24.144 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:24.144 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.857076_208.1", 2022-09-13T07:54:24.144 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:24.144 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:24.145 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:24.145 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T07:54:24.145 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:24.145 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:24.146 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:24.146 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:24.146 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:54:24.147 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:24.147 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:24.147 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:54:24.147 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:24.148 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:24.148 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:24.148 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:54:24.149 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:24.149 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:24.149 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:24.149 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:54:24.150 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:24.150 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:54:24.150 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:54:24.151 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:54:24.151 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:24.151 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:54:24Z" 2022-09-13T07:54:24.151 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:54:24.153 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.857076_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.857076Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:54:24Z" } 2022-09-13T07:54:24.153 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:54:24.154 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:24.154 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:24.155 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.857076_208.1 2022-09-13T07:54:29.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:29.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:29.159 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:54:29.210 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:54:29.210 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:54:29.238 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:29.234+0000 7f4292868600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:29.239 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:29.235+0000 7f4292868600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:29.253 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:54:29.254 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:54:29.254 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:54:29.254 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:54:29.254 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:54:29.255 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:54:29.255 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:29.256 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:29.256 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:54:29.257 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:29.257 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:54:29.257 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:29.257 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:29.258 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:54:29.258 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:29.258 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:54:29.258 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:29.259 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:54:29.259 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:29.259 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:29.259 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:29.260 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:29.260 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:54:29.260 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:29.260 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:29.261 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:54:29.261 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:29.261 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:29.261 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:29.262 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:54:29.262 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:29.262 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:29.263 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:29.263 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:29.263 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:54:29.263 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:29.264 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:29.264 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:54:29.264 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:29.264 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:29.265 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:29.265 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:54:29.265 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:29.265 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:29.266 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:29.266 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:29.266 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:54:29.266 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:29.267 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:29.267 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:54:29.267 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:29.267 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:29.268 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:29.268 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:54:29.268 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:29.268 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:29.269 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:29.269 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:54:29.270 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:29.270 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:54:29.270 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:54:29.270 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:54:29.271 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:29.271 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:54:29Z" 2022-09-13T07:54:29.271 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:54:29.272 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:54:29Z" } 2022-09-13T07:54:29.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:54:29.273 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:29.274 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:29.275 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T07:54:34.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:34.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:34.280 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:54:34.333 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:54:34.334 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:54:34.366 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:34.361+0000 7f415d91f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:34.366 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:34.362+0000 7f415d91f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:34.384 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:54:34.385 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:54:34.385 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:54:34.385 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:54:34.386 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:54:34.386 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:54:34.386 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:34.387 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:34.387 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:54:34.387 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:34.388 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:54:34.388 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:34.388 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:34.389 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:54:34.390 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:34.390 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:54:34.391 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:34.391 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:54:34.392 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:34.392 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:34.393 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:34.393 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:34.394 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:54:34.394 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:34.395 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:34.395 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:54:34.395 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:34.396 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:34.396 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:34.397 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:54:34.397 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:34.398 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:34.398 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:34.399 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:34.399 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:54:34.400 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:34.400 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:34.401 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.854520_207.1", 2022-09-13T07:54:34.401 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:34.402 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:34.402 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:34.403 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T07:54:34.403 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:34.403 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:34.404 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:34.404 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:34.405 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:54:34.405 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:34.406 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:34.406 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:54:34.406 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:34.406 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:34.407 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:34.407 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:54:34.407 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:34.407 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:34.408 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:34.408 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:54:34.408 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:34.408 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:54:34.409 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:54:34.409 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:54:34.409 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:34.410 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:54:34Z" 2022-09-13T07:54:34.410 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:54:34.412 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.854520_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.854520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:54:34Z" } 2022-09-13T07:54:34.413 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:54:34.414 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:34.414 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:34.415 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.854520_207.1 2022-09-13T07:54:39.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:39.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:39.417 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:54:39.474 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:54:39.475 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:54:39.514 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:39.509+0000 7f4fa5eca600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:39.515 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:39.511+0000 7f4fa5eca600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:39.529 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:54:39.530 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:54:39.530 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:54:39.531 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:54:39.531 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:54:39.532 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:54:39.532 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:39.533 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:39.533 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:54:39.534 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:39.534 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:54:39.535 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:39.535 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:39.536 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:54:39.537 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:39.537 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:54:39.538 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:39.538 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:54:39.539 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:39.539 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:39.539 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:39.539 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:39.540 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:54:39.540 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:39.540 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:39.541 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:54:39.541 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:39.541 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:39.542 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:39.542 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:54:39.542 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:39.543 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:39.543 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:39.543 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:39.543 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:54:39.544 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:39.544 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:39.544 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:54:39.545 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:39.545 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:39.546 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:39.546 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:54:39.546 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:39.547 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:39.547 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:39.547 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:39.548 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:54:39.548 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:39.548 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:39.549 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:54:39.549 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:39.549 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:39.549 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:39.550 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:54:39.550 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:39.550 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:39.550 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:39.551 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:54:39.551 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:39.552 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:54:39.552 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:54:39.553 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:54:39.553 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:39.554 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:54:39Z" 2022-09-13T07:54:39.554 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:54:39.556 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:54:39Z" } 2022-09-13T07:54:39.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:54:39.557 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:39.558 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:39.558 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T07:54:44.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:44.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:44.561 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:54:44.613 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:54:44.614 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:54:44.683 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:44.678+0000 7f87f004f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:44.692 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:44.687+0000 7f87f004f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:44.765 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:54:44.765 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:54:44.765 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:54:44.766 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:54:44.766 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:54:44.766 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:54:44.766 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:44.767 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:44.767 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:54:44.767 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:44.768 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:54:44.768 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:44.768 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:44.769 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:54:44.769 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:44.769 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:54:44.769 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:44.770 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:54:44.770 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:44.770 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:44.770 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:44.771 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:44.771 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:54:44.771 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:44.771 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:44.772 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:54:44.772 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:44.772 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:44.772 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:44.773 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:54:44.773 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:44.773 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:44.774 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:44.774 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:44.774 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:54:44.774 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:44.775 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:44.775 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.861945_210.1", 2022-09-13T07:54:44.775 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:44.775 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:44.776 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:44.776 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T07:54:44.776 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:44.776 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:44.777 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:44.777 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:44.777 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:54:44.777 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:44.778 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:44.778 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:54:44.778 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:44.779 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:44.779 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:44.779 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:54:44.779 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:44.780 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:44.780 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:44.780 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:54:44.780 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:44.781 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:54:44.781 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:54:44.781 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:54:44.782 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:44.782 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:54:44Z" 2022-09-13T07:54:44.782 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:54:44.783 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.861945_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.861945Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:54:44Z" } 2022-09-13T07:54:44.783 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:54:44.784 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:44.784 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:44.785 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.861945_210.1 2022-09-13T07:54:49.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:49.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:49.790 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:54:49.840 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:54:49.840 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:54:49.874 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:49.869+0000 7f0553f87600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:49.875 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:49.870+0000 7f0553f87600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:49.890 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:54:49.890 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:54:49.891 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:54:49.891 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:54:49.891 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:54:49.891 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:54:49.892 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:49.892 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:49.892 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:54:49.892 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:49.893 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:54:49.893 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:49.894 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:49.894 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:54:49.894 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:49.895 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:54:49.895 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:49.896 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:54:49.896 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:49.897 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:49.897 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:49.897 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:49.898 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:54:49.898 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:49.899 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:49.900 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:54:49.900 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:49.901 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:49.901 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:49.902 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:54:49.902 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:49.903 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:49.903 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:49.904 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:49.904 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:54:49.905 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:49.905 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:49.906 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.906479_212.1", 2022-09-13T07:54:49.906 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:49.906 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:49.907 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:49.907 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.906479Z", 2022-09-13T07:54:49.908 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:49.908 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:49.909 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:49.909 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:49.909 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:54:49.909 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:49.910 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:49.911 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:54:49.911 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:49.911 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:49.911 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:49.912 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:54:49.912 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:49.912 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:49.912 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:49.913 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:54:49.913 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:49.913 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:54:49.914 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:54:49.914 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:54:49.914 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:49.914 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:54:49Z" 2022-09-13T07:54:49.915 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:54:49.916 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.906479_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.906479Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:54:49Z" } 2022-09-13T07:54:49.917 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.906479_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.906479Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:54:49.917 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:49.918 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.906479_212.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:49.918 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.906479_212.1 2022-09-13T07:54:54.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:54.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:54:54.921 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:54:54.971 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:54:54.972 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:54:55.009 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:55.004+0000 7fc2288af600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:55.010 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:54:55.005+0000 7fc2288af600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:54:55.024 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:54:55.025 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:54:55.025 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:54:55.025 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:54:55.025 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:54:55.026 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:54:55.028 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:55.028 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:55.028 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:54:55.029 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:55.029 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:54:55.029 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:55.029 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:55.030 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:54:55.030 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:55.030 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:54:55.031 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:55.031 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:54:55.031 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:55.032 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:55.032 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:55.032 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:55.032 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:54:55.033 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:55.033 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:55.034 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:54:55.035 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:55.035 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:55.036 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:55.037 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:54:55.038 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:55.038 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:55.039 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:55.039 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:55.040 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:54:55.041 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:55.041 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:55.042 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.953175_216.1", 2022-09-13T07:54:55.043 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:55.043 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:55.044 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:55.044 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T07:54:55.045 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:55.045 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:55.046 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:55.046 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:54:55.047 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:54:55.048 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:54:55.049 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:54:55.049 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:54:55.049 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:54:55.050 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:54:55.050 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:54:55.051 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:54:55.051 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:54:55.052 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:55.052 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:54:55.053 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:54:55.053 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:55.054 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:54:55.055 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:54:55.055 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:54:55.056 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:54:55.056 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:54:55Z" 2022-09-13T07:54:55.057 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:54:55.059 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.953175_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.953175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:54:55Z" } 2022-09-13T07:54:55.061 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:54:55.062 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:55.063 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:54:55.064 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.953175_216.1 2022-09-13T07:55:00.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:00.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:00.067 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:55:00.119 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:55:00.119 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:55:00.151 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:00.148+0000 7fdba8fc5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:00.152 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:00.149+0000 7fdba8fc5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:00.165 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:55:00.166 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:55:00.167 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:55:00.167 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:55:00.168 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:55:00.168 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:55:00.169 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:00.169 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:00.170 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:55:00.171 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:00.171 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:55:00.171 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:00.172 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:00.172 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:55:00.172 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:00.173 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:55:00.173 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:00.173 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:55:00.174 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:00.174 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:00.174 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:00.174 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:00.175 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:55:00.175 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:00.175 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:00.176 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:55:00.176 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:00.176 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:00.176 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:00.177 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:55:00.177 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:00.177 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:00.179 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:00.179 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:00.180 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:55:00.180 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:00.181 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:00.181 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.990623_217.1", 2022-09-13T07:55:00.182 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:00.182 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:00.183 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:00.184 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T07:55:00.184 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:00.185 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:00.185 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:00.185 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:00.190 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:55:00.190 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:00.191 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:00.191 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:55:00.191 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:00.191 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:00.192 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:00.192 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:55:00.192 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:00.192 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:00.193 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:00.193 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:55:00.193 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:00.194 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:55:00.194 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:55:00.194 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:55:00.195 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:00.195 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:55:00Z" 2022-09-13T07:55:00.195 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:55:00.196 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.990623_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.990623Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:55:00Z" } 2022-09-13T07:55:00.197 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:55:00.198 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:00.198 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:00.199 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.990623_217.1 2022-09-13T07:55:05.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:05.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:05.203 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:55:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:55:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:55:05.293 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:05.290+0000 7f20c6ca0600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:05.294 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:05.291+0000 7f20c6ca0600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:05.309 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:55:05.309 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:55:05.310 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:55:05.310 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:55:05.311 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:55:05.311 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:55:05.312 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:05.312 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:05.313 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:55:05.313 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:05.314 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:55:05.314 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:05.314 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:05.315 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:55:05.315 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:05.316 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:55:05.316 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:05.316 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:55:05.317 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:05.317 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:05.318 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:05.318 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:05.318 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:55:05.319 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:05.319 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:05.320 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:55:05.320 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:05.321 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:05.321 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:05.322 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:55:05.322 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:05.322 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:05.323 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:05.323 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:05.324 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:55:05.324 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:05.324 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:05.325 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.861945_210.1", 2022-09-13T07:55:05.325 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:05.326 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:05.326 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:05.326 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T07:55:05.327 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:05.327 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:05.328 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:05.328 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:05.329 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:55:05.329 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:05.329 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:05.329 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:55:05.330 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:05.330 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:05.330 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:05.330 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:55:05.331 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:05.331 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:05.331 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:05.332 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:55:05.332 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:05.333 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:55:05.333 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:55:05.334 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:55:05.334 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:05.334 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:55:05Z" 2022-09-13T07:55:05.335 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:55:05.336 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.861945_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.861945Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:55:05Z" } 2022-09-13T07:55:05.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:55:05.338 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:05.338 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:05.339 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.861945_210.1 2022-09-13T07:55:10.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:10.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:10.341 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:55:10.434 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:55:10.434 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:55:10.479 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:10.474+0000 7f6c8e182600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:10.480 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:10.476+0000 7f6c8e182600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:10.501 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:55:10.502 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:55:10.502 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:55:10.503 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:55:10.503 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:55:10.503 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:55:10.503 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:10.504 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:10.504 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:55:10.504 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:10.505 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:55:10.505 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:10.505 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:10.505 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:55:10.506 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:10.506 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:55:10.506 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:10.507 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:55:10.507 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:10.507 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:10.507 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:10.508 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:10.508 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:55:10.508 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:10.509 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:10.509 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:55:10.509 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:10.509 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:10.510 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:10.510 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:55:10.510 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:10.511 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:10.511 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:10.511 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:10.511 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:55:10.512 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:10.512 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:10.512 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.953175_216.1", 2022-09-13T07:55:10.513 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:10.513 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:10.513 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:10.514 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T07:55:10.514 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:10.514 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:10.514 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:10.515 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:10.515 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:55:10.515 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:10.516 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:10.516 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:55:10.516 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:10.516 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:10.517 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:10.517 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:55:10.517 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:10.518 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:10.518 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:10.518 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:55:10.518 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:10.519 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:55:10.519 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:55:10.519 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:55:10.519 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:10.520 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:55:10Z" 2022-09-13T07:55:10.520 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:55:10.521 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.953175_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.953175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:55:10Z" } 2022-09-13T07:55:10.522 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:55:10.523 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:10.523 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:10.524 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.953175_216.1 2022-09-13T07:55:15.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:15.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:15.527 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:55:15.576 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:55:15.577 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:55:15.622 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:15.617+0000 7fa46b071600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:15.622 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:15.618+0000 7fa46b071600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:15.637 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:55:15.637 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:55:15.638 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:55:15.638 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:55:15.638 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:55:15.638 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:55:15.639 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:15.639 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:15.639 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:55:15.639 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:15.640 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:55:15.640 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:15.640 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:15.640 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:55:15.641 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:15.641 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:55:15.641 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:15.641 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:55:15.642 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:15.642 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:15.643 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:15.643 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:15.643 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:55:15.644 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:15.644 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:15.644 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:55:15.645 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:15.645 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:15.645 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:15.646 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:55:15.646 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:15.647 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:15.647 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:15.647 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:15.648 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:55:15.648 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:15.649 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:15.649 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.994620_218.1", 2022-09-13T07:55:15.649 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:15.650 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:15.650 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:15.650 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.994620Z", 2022-09-13T07:55:15.651 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:15.651 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:15.652 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:15.652 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:15.652 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:55:15.653 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:15.653 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:15.653 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:55:15.654 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:15.654 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:15.655 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:15.655 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:55:15.655 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:15.656 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:15.656 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:15.656 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:55:15.657 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:15.657 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:55:15.658 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:55:15.658 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:55:15.659 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:15.659 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:55:15Z" 2022-09-13T07:55:15.659 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:55:15.661 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.994620_218.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.994620Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:55:15Z" } 2022-09-13T07:55:15.662 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.994620_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.994620Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:55:15.663 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:15.664 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.994620_218.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:15.664 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.994620_218.1 2022-09-13T07:55:20.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... FAIL 2022-09-13T07:55:20.691 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-42 2022-09-13T07:55:20.692 DEBUG:boto:path=/yxvvuh-42/ 2022-09-13T07:55:20.692 DEBUG:boto:auth_path=/yxvvuh-42/ 2022-09-13T07:55:20.693 DEBUG:boto:Method: PUT 2022-09-13T07:55:20.694 DEBUG:boto:Path: /yxvvuh-42/ 2022-09-13T07:55:20.694 DEBUG:boto:Data: 2022-09-13T07:55:20.695 DEBUG:boto:Headers: {} 2022-09-13T07:55:20.695 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:55:20.696 DEBUG:boto:Port: 8000 2022-09-13T07:55:20.696 DEBUG:boto:Params: {} 2022-09-13T07:55:20.697 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T07:55:20.698 DEBUG:boto:Token: None 2022-09-13T07:55:20.699 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:55:20 GMT /yxvvuh-42/ 2022-09-13T07:55:20.700 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lqxlXU5oIW3YMtlnnT1V5sULMUM= 2022-09-13T07:55:20.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lqxlXU5oIW3YMtlnnT1V5sULMUM=', 'Content-Length': '0'} 2022-09-13T07:55:20.721 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008edf101371b6fe3a-0063203768-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:55:20.722 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-43 2022-09-13T07:55:20.723 DEBUG:boto:path=/yxvvuh-43/ 2022-09-13T07:55:20.724 DEBUG:boto:auth_path=/yxvvuh-43/ 2022-09-13T07:55:20.725 DEBUG:boto:Method: PUT 2022-09-13T07:55:20.726 DEBUG:boto:Path: /yxvvuh-43/ 2022-09-13T07:55:20.727 DEBUG:boto:Data: 2022-09-13T07:55:20.728 DEBUG:boto:Headers: {} 2022-09-13T07:55:20.728 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:55:20.729 DEBUG:boto:Port: 8000 2022-09-13T07:55:20.730 DEBUG:boto:Params: {} 2022-09-13T07:55:20.732 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T07:55:20.733 DEBUG:boto:Token: None 2022-09-13T07:55:20.734 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:55:20 GMT /yxvvuh-43/ 2022-09-13T07:55:20.735 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:s7aclqHmfZ5cHyWE3dTtBRwnUIg= 2022-09-13T07:55:20.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:s7aclqHmfZ5cHyWE3dTtBRwnUIg=', 'Content-Length': '0'} 2022-09-13T07:55:20.773 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000076f9250361f73fca-0063203768-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:55:20.774 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-44 2022-09-13T07:55:20.775 DEBUG:boto:path=/yxvvuh-44/ 2022-09-13T07:55:20.776 DEBUG:boto:auth_path=/yxvvuh-44/ 2022-09-13T07:55:20.776 DEBUG:boto:Method: PUT 2022-09-13T07:55:20.777 DEBUG:boto:Path: /yxvvuh-44/ 2022-09-13T07:55:20.778 DEBUG:boto:Data: 2022-09-13T07:55:20.779 DEBUG:boto:Headers: {} 2022-09-13T07:55:20.780 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:55:20.781 DEBUG:boto:Port: 8001 2022-09-13T07:55:20.782 DEBUG:boto:Params: {} 2022-09-13T07:55:20.783 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-13T07:55:20.784 DEBUG:boto:Token: None 2022-09-13T07:55:20.785 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 07:55:20 GMT /yxvvuh-44/ 2022-09-13T07:55:20.786 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Pi2VJBjexyAetMoqh/6mk/6wgXM= 2022-09-13T07:55:20.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pi2VJBjexyAetMoqh/6mk/6wgXM=', 'Content-Length': '0'} 2022-09-13T07:55:20.820 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d0c0f03f92b95778-0063203768-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:55:20.821 DEBUG:boto:path=/yxvvuh-42/ 2022-09-13T07:55:20.822 DEBUG:boto:auth_path=/yxvvuh-42/ 2022-09-13T07:55:20.822 DEBUG:boto:Method: HEAD 2022-09-13T07:55:20.823 DEBUG:boto:Path: /yxvvuh-42/ 2022-09-13T07:55:20.823 DEBUG:boto:Data: 2022-09-13T07:55:20.824 DEBUG:boto:Headers: {} 2022-09-13T07:55:20.824 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:55:20.825 DEBUG:boto:Port: 8000 2022-09-13T07:55:20.825 DEBUG:boto:Params: {} 2022-09-13T07:55:20.826 DEBUG:boto:Token: None 2022-09-13T07:55:20.827 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:55:20 GMT /yxvvuh-42/ 2022-09-13T07:55:20.827 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:CpfA5K/OCkrRH5QXw3NUlvpZKO4= 2022-09-13T07:55:20.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CpfA5K/OCkrRH5QXw3NUlvpZKO4=', 'Content-Length': '0'} 2022-09-13T07:55:20.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000effb8abb96b7d4fe-0063203768-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:55:20.833 DEBUG:boto:path=/yxvvuh-42/key 2022-09-13T07:55:20.833 DEBUG:boto:auth_path=/yxvvuh-42/key 2022-09-13T07:55:20.834 DEBUG:boto:Method: PUT 2022-09-13T07:55:20.834 DEBUG:boto:Path: /yxvvuh-42/key 2022-09-13T07:55:20.835 DEBUG:boto:Data: 2022-09-13T07:55:20.835 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-09-13T07:55:20.836 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T07:55:20.836 DEBUG:boto:Port: 8000 2022-09-13T07:55:20.837 DEBUG:boto:Params: {} 2022-09-13T07:55:20.837 DEBUG:boto:Token: None 2022-09-13T07:55:20.838 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Tue, 13 Sep 2022 07:55:20 GMT /yxvvuh-42/key 2022-09-13T07:55:20.838 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:FQ4oWajkhlINkDdn7fZltEQD35M= 2022-09-13T07:55:20.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FQ4oWajkhlINkDdn7fZltEQD35M='} 2022-09-13T07:55:20.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca8f3101537b0d8c-0063203768-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:55:20.847 DEBUG:boto:path=/yxvvuh-43/ 2022-09-13T07:55:20.848 DEBUG:boto:auth_path=/yxvvuh-43/ 2022-09-13T07:55:20.848 DEBUG:boto:Method: HEAD 2022-09-13T07:55:20.849 DEBUG:boto:Path: /yxvvuh-43/ 2022-09-13T07:55:20.849 DEBUG:boto:Data: 2022-09-13T07:55:20.850 DEBUG:boto:Headers: {} 2022-09-13T07:55:20.850 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:55:20.851 DEBUG:boto:Port: 8000 2022-09-13T07:55:20.851 DEBUG:boto:Params: {} 2022-09-13T07:55:20.852 DEBUG:boto:Token: None 2022-09-13T07:55:20.852 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:55:20 GMT /yxvvuh-43/ 2022-09-13T07:55:20.853 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:lCIw+DQBs08CZAg9rRu2OdkrqK4= 2022-09-13T07:55:20.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lCIw+DQBs08CZAg9rRu2OdkrqK4=', 'Content-Length': '0'} 2022-09-13T07:55:20.857 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f4f45dcfdaa9a3c-0063203768-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:55:20.857 DEBUG:boto:path=/yxvvuh-43/key 2022-09-13T07:55:20.858 DEBUG:boto:auth_path=/yxvvuh-43/key 2022-09-13T07:55:20.858 DEBUG:boto:Method: PUT 2022-09-13T07:55:20.859 DEBUG:boto:Path: /yxvvuh-43/key 2022-09-13T07:55:20.859 DEBUG:boto:Data: 2022-09-13T07:55:20.860 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-09-13T07:55:20.860 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T07:55:20.861 DEBUG:boto:Port: 8000 2022-09-13T07:55:20.861 DEBUG:boto:Params: {} 2022-09-13T07:55:20.862 DEBUG:boto:Token: None 2022-09-13T07:55:20.863 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Tue, 13 Sep 2022 07:55:20 GMT /yxvvuh-43/key 2022-09-13T07:55:20.863 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:K/qNJQIRJ0eIl1+1mk8EcT4AiEg= 2022-09-13T07:55:20.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:K/qNJQIRJ0eIl1+1mk8EcT4AiEg='} 2022-09-13T07:55:20.876 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a98329e91e8276e-0063203768-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:55:20.877 DEBUG:boto:path=/yxvvuh-44/ 2022-09-13T07:55:20.878 DEBUG:boto:auth_path=/yxvvuh-44/ 2022-09-13T07:55:20.879 DEBUG:boto:Method: HEAD 2022-09-13T07:55:20.879 DEBUG:boto:Path: /yxvvuh-44/ 2022-09-13T07:55:20.879 DEBUG:boto:Data: 2022-09-13T07:55:20.880 DEBUG:boto:Headers: {} 2022-09-13T07:55:20.880 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:55:20.881 DEBUG:boto:Port: 8001 2022-09-13T07:55:20.882 DEBUG:boto:Params: {} 2022-09-13T07:55:20.882 DEBUG:boto:Token: None 2022-09-13T07:55:20.883 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 07:55:20 GMT /yxvvuh-44/ 2022-09-13T07:55:20.884 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:JGvc3iKZ7YsLL0yItktM90Guz7k= 2022-09-13T07:55:20.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JGvc3iKZ7YsLL0yItktM90Guz7k=', 'Content-Length': '0'} 2022-09-13T07:55:20.888 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000644c6fe21b5aac73-0063203768-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:55:20.889 DEBUG:boto:path=/yxvvuh-44/key 2022-09-13T07:55:20.889 DEBUG:boto:auth_path=/yxvvuh-44/key 2022-09-13T07:55:20.890 DEBUG:boto:Method: PUT 2022-09-13T07:55:20.890 DEBUG:boto:Path: /yxvvuh-44/key 2022-09-13T07:55:20.891 DEBUG:boto:Data: 2022-09-13T07:55:20.891 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-09-13T07:55:20.892 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T07:55:20.892 DEBUG:boto:Port: 8001 2022-09-13T07:55:20.893 DEBUG:boto:Params: {} 2022-09-13T07:55:20.893 DEBUG:boto:Token: None 2022-09-13T07:55:20.894 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Tue, 13 Sep 2022 07:55:20 GMT /yxvvuh-44/key 2022-09-13T07:55:20.895 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YdVWUqTMqRV5JCaPP/hfq28sqTs= 2022-09-13T07:55:20.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YdVWUqTMqRV5JCaPP/hfq28sqTs='} 2022-09-13T07:55:20.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e9b49658f732d00-0063203768-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T07:55:20.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:20.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:20.906 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:55:20.959 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:55:20.960 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:55:20.988 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T07:55:21.001 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T07:55:21.001 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:55:21.002 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:55:21.002 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T07:55:21.002 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:55:21.003 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:55:21.003 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T07:55:21.003 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T07:55:21.004 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:55:21.004 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:55:21.004 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663055720.804779_308.1", 2022-09-13T07:55:21.004 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:55:20.804779Z" 2022-09-13T07:55:21.005 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T07:55:21.005 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T07:55:21.005 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:55:21.005 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T07:55:21.006 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T07:55:21.006 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T07:55:21.008 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:21.008 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T07:55:21.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:21.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:21.011 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:55:21.062 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:55:21.063 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:55:21.096 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:21.092+0000 7f7ff4dcd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:21.097 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:21.093+0000 7f7ff4dcd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:21.118 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:55:21.118 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:55:21.118 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:55:21.119 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:55:21.119 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:55:21.119 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:55:21.119 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:21.120 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:21.120 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:55:21.120 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:21.120 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:55:21.121 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:21.121 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:21.121 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:55:21.121 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:21.122 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:55:21.122 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:21.122 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:55:21.122 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:21.123 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:21.123 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:21.123 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:21.123 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:55:21.124 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:21.124 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:21.124 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:55:21.124 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:21.125 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:21.125 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:21.125 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:55:21.125 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:21.126 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:21.126 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:21.126 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:21.126 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:55:21.127 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:21.127 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:21.127 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.854520_207.1", 2022-09-13T07:55:21.127 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:21.128 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:21.128 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:21.128 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T07:55:21.128 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:21.129 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:21.129 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:21.129 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:21.129 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:55:21.130 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:21.130 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:21.130 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:55:21.130 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:21.131 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:21.131 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:21.131 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:55:21.131 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:21.132 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:21.132 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:21.132 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:55:21.132 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:21.133 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:55:21.133 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:55:21.133 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:55:21.133 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:21.133 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:55:21Z" 2022-09-13T07:55:21.134 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:55:21.135 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.854520_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.854520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:55:21Z" } 2022-09-13T07:55:21.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:55:21.136 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:21.136 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:21.137 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.854520_207.1 2022-09-13T07:55:26.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:26.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:26.140 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:55:26.192 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:55:26.193 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:55:26.223 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:26.218+0000 7fc28550d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:26.224 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:26.219+0000 7fc28550d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:26.239 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:55:26.240 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:55:26.240 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:55:26.240 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:55:26.241 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:55:26.241 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:55:26.241 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:26.241 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:26.242 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:55:26.242 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:26.242 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:55:26.242 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:26.243 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:26.243 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:55:26.243 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:26.243 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:55:26.244 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:26.244 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:55:26.244 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:26.245 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:26.245 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:26.247 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:26.247 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:55:26.247 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:26.247 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:26.248 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:55:26.248 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:26.248 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:26.249 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:26.249 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:55:26.249 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:26.249 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:26.250 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:26.250 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:26.250 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:55:26.250 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:26.251 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:26.251 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.854520_207.1", 2022-09-13T07:55:26.251 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:26.251 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:26.252 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:26.252 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T07:55:26.252 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:26.253 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:26.253 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:26.253 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:26.254 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:55:26.254 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:26.254 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:26.255 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:55:26.255 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:26.256 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:26.256 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:26.256 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:55:26.257 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:26.257 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:26.257 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:26.258 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:55:26.258 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:26.258 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:55:26.259 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:55:26.259 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:55:26.259 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:26.259 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:55:26Z" 2022-09-13T07:55:26.260 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:55:26.261 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.854520_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.854520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:55:26Z" } 2022-09-13T07:55:26.261 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:55:26.262 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:26.263 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:26.263 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.854520_207.1 2022-09-13T07:55:31.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:31.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:31.266 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:55:31.318 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:55:31.318 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:55:31.360 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:31.355+0000 7f880f69d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:31.361 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:31.357+0000 7f880f69d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:31.448 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:55:31.448 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:55:31.448 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:55:31.449 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:55:31.449 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:55:31.449 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:55:31.450 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:31.450 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:31.450 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:55:31.451 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:31.451 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:55:31.451 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:31.451 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:31.452 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:55:31.452 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:31.452 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:55:31.452 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:31.453 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:55:31.453 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:31.453 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:31.454 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:31.454 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:31.455 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:55:31.455 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:31.455 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:31.456 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:55:31.456 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:31.456 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:31.457 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:31.457 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:55:31.458 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:31.458 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:31.458 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:31.458 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:31.459 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:55:31.459 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:31.459 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:31.459 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:55:31.460 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:31.460 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:31.460 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:31.461 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:55:31.461 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:31.461 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:31.462 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:31.462 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:31.462 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:55:31.463 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:31.463 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:31.463 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:55:31.464 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:31.464 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:31.465 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:31.465 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:55:31.465 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:31.465 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:31.466 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:31.466 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:55:31.466 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:31.466 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:55:31.467 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:55:31.467 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:55:31.467 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:31.467 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:55:31Z" 2022-09-13T07:55:31.468 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:55:31.469 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:55:31Z" } 2022-09-13T07:55:31.470 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:55:31.470 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:31.471 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:31.471 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.831134_204.1 2022-09-13T07:55:36.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:36.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:36.474 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:55:36.557 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:55:36.557 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:55:36.596 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:36.592+0000 7fba184b9600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:36.597 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:36.593+0000 7fba184b9600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:36.613 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:55:36.613 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:55:36.614 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:55:36.615 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:55:36.615 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:55:36.616 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:55:36.616 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:36.617 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:36.617 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:55:36.618 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:36.619 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:55:36.619 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:36.620 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:36.620 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:55:36.621 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:36.622 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:55:36.622 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:36.623 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:55:36.623 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:36.624 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:36.624 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:36.625 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:36.625 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:55:36.626 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:36.626 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:36.626 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:55:36.626 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:36.627 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:36.627 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:36.627 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:55:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:55:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.857076_208.1", 2022-09-13T07:55:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T07:55:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:55:36.633 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:36.633 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:36.633 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:55:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:55:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:55:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:55:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:55:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:55:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:55:36Z" 2022-09-13T07:55:36.638 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:55:36.639 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.857076_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.857076Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:55:36Z" } 2022-09-13T07:55:36.640 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:55:36.641 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:36.642 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:36.643 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.857076_208.1 2022-09-13T07:55:41.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:41.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:41.647 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:55:41.699 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:55:41.700 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:55:41.732 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:41.728+0000 7fbc7190b600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:41.733 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:41.729+0000 7fbc7190b600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:41.749 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:55:41.750 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:55:41.751 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:55:41.751 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:55:41.752 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:55:41.752 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:55:41.753 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:41.753 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:41.754 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:55:41.755 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:41.755 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:55:41.756 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:41.756 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:41.756 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:55:41.757 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:41.757 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:55:41.758 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:41.758 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:55:41.759 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:41.759 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:41.759 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:41.760 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:41.760 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:55:41.761 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:41.761 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:41.763 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:55:41.764 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:41.764 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:41.765 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:41.766 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:55:41.766 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:41.767 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:41.767 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:41.767 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:41.768 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:55:41.768 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:41.769 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:41.769 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.858000_209.1", 2022-09-13T07:55:41.770 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:41.770 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:41.770 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:41.771 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T07:55:41.771 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:41.772 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:41.772 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:41.773 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:41.773 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:55:41.774 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:41.775 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:41.775 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:55:41.776 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:41.776 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:41.777 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:41.777 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:55:41.778 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:41.779 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:41.779 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:41.779 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:55:41.780 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:41.780 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:55:41.781 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:55:41.781 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:55:41.782 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:41.782 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:55:41Z" 2022-09-13T07:55:41.782 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:55:41.789 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.858000_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.858000Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:55:41Z" } 2022-09-13T07:55:41.791 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:55:41.791 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:41.792 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:41.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.858000_209.1 2022-09-13T07:55:46.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:46.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:46.808 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:55:46.859 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:55:46.859 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:55:46.911 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:46.908+0000 7f5a66b04600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:46.912 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:46.909+0000 7f5a66b04600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:46.928 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:55:46.929 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:55:46.929 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:55:46.930 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:55:46.930 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:55:46.931 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:55:46.931 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:46.931 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:46.932 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:55:46.932 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:46.932 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:55:46.932 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:46.933 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:46.933 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:55:46.933 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:46.933 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:55:46.934 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:46.934 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:55:46.934 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:46.935 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:46.935 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:46.935 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:46.935 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:55:46.936 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:46.936 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:46.936 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:55:46.936 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:46.937 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:46.937 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:46.937 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:55:46.938 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:46.938 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:46.938 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:46.938 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:46.939 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:55:46.939 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:46.939 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:46.939 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.994620_218.1", 2022-09-13T07:55:46.940 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:46.940 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:46.940 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:46.940 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.994620Z", 2022-09-13T07:55:46.941 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:46.953 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:46.954 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:46.955 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:46.955 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:55:46.955 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:46.956 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:46.956 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:55:46.957 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:46.957 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:46.958 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:46.959 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:55:46.959 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:46.959 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:46.960 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:46.960 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:55:46.961 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:46.961 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:55:46.965 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:55:46.965 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:55:46.966 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:46.966 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:55:46Z" 2022-09-13T07:55:46.966 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:55:46.969 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.994620_218.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.994620Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:55:46Z" } 2022-09-13T07:55:46.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.994620_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.994620Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:55:46.970 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:46.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.994620_218.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:46.971 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.994620_218.1 2022-09-13T07:55:51.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:51.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:51.975 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:55:52.026 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:55:52.026 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:55:52.070 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:52.062+0000 7fbd5d94b600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:52.071 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:52.063+0000 7fbd5d94b600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:52.080 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:55:52.080 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:55:52.080 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:55:52.080 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:55:52.081 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:55:52.081 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:55:52.081 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:52.082 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:52.082 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:55:52.082 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:52.082 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:55:52.083 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:52.083 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:52.083 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:55:52.083 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:52.084 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:55:52.084 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:52.084 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:55:52.084 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:52.085 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:52.085 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:52.085 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:52.085 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:55:52.086 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:52.086 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:52.086 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:55:52.086 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:52.087 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:52.087 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:52.087 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:55:52.087 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:52.088 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:52.088 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:52.088 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:52.089 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:55:52.089 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:52.089 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:52.089 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.861945_210.1", 2022-09-13T07:55:52.090 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:52.090 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:52.090 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:52.090 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T07:55:52.091 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:52.091 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:52.091 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:52.092 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:52.092 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:55:52.092 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:52.092 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:52.093 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:55:52.093 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:52.093 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:52.093 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:52.093 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:55:52.094 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:52.095 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:52.096 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:52.096 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:55:52.096 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:52.097 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:55:52.097 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:55:52.097 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:55:52.098 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:52.098 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:55:52Z" 2022-09-13T07:55:52.099 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:55:52.100 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.861945_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.861945Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:55:52Z" } 2022-09-13T07:55:52.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:55:52.102 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:52.102 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:52.103 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.861945_210.1 2022-09-13T07:55:57.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:57.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:55:57.106 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:55:57.159 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:55:57.160 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:55:57.203 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:57.199+0000 7f3843b0a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:57.205 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:55:57.200+0000 7f3843b0a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:55:57.223 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:55:57.224 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:55:57.224 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:55:57.224 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:55:57.225 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:55:57.225 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:55:57.225 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:57.225 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:57.226 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:55:57.226 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:57.226 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:55:57.227 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:57.227 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:57.227 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:55:57.227 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:57.228 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:55:57.228 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:57.228 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:55:57.228 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:57.229 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:57.229 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:57.229 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:57.229 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:55:57.229 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:57.230 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:57.230 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:55:57.230 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:57.231 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:57.231 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:57.231 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:55:57.231 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:57.232 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:57.232 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:57.233 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:57.233 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:55:57.233 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:57.234 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:57.234 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T07:55:57.235 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:57.235 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:57.235 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:57.236 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T07:55:57.236 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:57.236 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:57.237 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:57.237 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:55:57.238 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:55:57.238 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:55:57.238 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:55:57.239 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:55:57.239 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:55:57.239 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:55:57.240 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:55:57.240 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:55:57.240 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:55:57.240 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:57.241 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:55:57.241 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:55:57.241 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:57.242 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:55:57.242 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:55:57.243 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:55:57.243 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:55:57.244 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:55:57Z" 2022-09-13T07:55:57.244 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:55:57.245 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:55:57Z" } 2022-09-13T07:55:57.246 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:55:57.247 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:57.247 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:55:57.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.832045_205.1 2022-09-13T07:56:02.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:02.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:02.250 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:56:02.324 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:56:02.325 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:56:02.445 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:02.440+0000 7fa335e6f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:02.453 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:02.449+0000 7fa335e6f600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:02.484 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:56:02.485 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:56:02.485 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:56:02.487 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:56:02.487 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:56:02.488 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:56:02.488 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:02.489 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:02.489 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:56:02.490 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:02.490 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:56:02.491 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:02.491 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:02.492 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:56:02.492 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:02.493 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:56:02.493 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:02.494 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:56:02.494 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:02.494 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:02.495 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:02.495 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:02.496 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:56:02.496 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:02.497 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:02.497 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:56:02.497 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:02.498 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:02.498 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:02.498 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:56:02.499 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:02.499 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:02.499 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:02.499 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:02.500 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:56:02.500 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:02.500 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:02.500 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T07:56:02.501 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:02.501 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:02.501 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:02.501 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T07:56:02.502 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:02.502 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:02.502 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:02.503 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:02.503 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:56:02.504 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:02.504 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:02.504 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:56:02.505 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:02.505 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:02.505 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:02.505 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:56:02.506 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:02.506 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:02.506 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:02.507 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:56:02.507 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:02.507 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:56:02.507 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:56:02.508 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:56:02.508 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:02.508 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:56:02Z" 2022-09-13T07:56:02.508 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:56:02.510 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:56:02Z" } 2022-09-13T07:56:02.511 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:56:02.512 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:02.513 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:02.513 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.949151_215.1 2022-09-13T07:56:07.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:07.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:07.517 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:56:07.567 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:56:07.567 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:56:07.609 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:07.605+0000 7f6483a87600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:07.610 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:07.606+0000 7f6483a87600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:07.626 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:56:07.626 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:56:07.627 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:56:07.627 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:56:07.627 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:56:07.627 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:56:07.628 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:07.628 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:07.628 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:56:07.628 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:07.629 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:56:07.629 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:07.629 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:07.629 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:56:07.630 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:07.630 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:56:07.631 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:07.631 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:56:07.631 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:07.632 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:07.633 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:07.633 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:07.633 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:56:07.633 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:07.634 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:07.634 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:56:07.634 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:07.635 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:07.635 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:07.635 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:56:07.635 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:07.636 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:07.636 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:07.636 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:07.637 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:56:07.637 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:07.638 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:07.638 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T07:56:07.638 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:07.639 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:07.639 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:07.639 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T07:56:07.639 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:07.640 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:07.640 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:07.640 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:07.641 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:56:07.641 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:07.642 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:07.642 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:56:07.644 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:07.644 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:07.645 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:07.645 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:56:07.646 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:07.646 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:07.647 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:07.647 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:56:07.648 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:07.648 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:56:07.649 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:56:07.649 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:56:07.649 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:07.650 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:56:07Z" 2022-09-13T07:56:07.650 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:56:07.653 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:56:07Z" } 2022-09-13T07:56:07.655 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:56:07.655 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:07.656 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:07.656 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.949151_215.1 2022-09-13T07:56:12.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:12.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:12.659 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:56:12.715 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:56:12.716 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:56:12.765 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:12.760+0000 7fede5cb8600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:12.766 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:12.761+0000 7fede5cb8600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:12.779 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:56:12.780 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:56:12.780 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:56:12.781 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:56:12.781 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:56:12.782 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:56:12.782 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:12.782 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:12.783 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:56:12.783 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:12.784 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:56:12.784 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:12.785 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:12.785 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:56:12.786 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:12.786 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:56:12.787 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:12.787 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:56:12.788 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:12.788 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:12.789 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:12.789 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:12.789 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:56:12.790 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:12.790 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:12.791 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:56:12.791 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:12.791 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:12.792 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:12.792 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:56:12.793 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:12.793 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:12.794 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:12.794 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:12.795 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:56:12.795 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:12.795 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:12.796 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.854520_207.1", 2022-09-13T07:56:12.796 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:12.796 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:12.796 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:12.797 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T07:56:12.797 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:12.797 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:12.797 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:12.798 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:12.798 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:56:12.799 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:12.799 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:12.799 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:56:12.799 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:12.800 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:12.800 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:12.800 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:56:12.801 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:12.801 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:12.801 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:12.801 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:56:12.802 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:12.802 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:56:12.802 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:56:12.803 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:56:12.803 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:12.803 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:56:12Z" 2022-09-13T07:56:12.804 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:56:12.806 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.854520_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.854520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:56:12Z" } 2022-09-13T07:56:12.807 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:56:12.808 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:12.809 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:12.810 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.854520_207.1 2022-09-13T07:56:17.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:17.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:17.814 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:56:17.876 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:56:17.877 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:56:17.933 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:17.928+0000 7f124924d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:17.934 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:17.929+0000 7f124924d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:17.957 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:56:17.958 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:56:17.958 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:56:17.959 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:56:17.959 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:56:17.960 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:56:17.960 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:17.960 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:17.961 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:56:17.961 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:17.962 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:56:17.962 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:17.962 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:17.963 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:56:17.963 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:17.963 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:56:17.963 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:17.964 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:56:17.964 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:17.965 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:17.965 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:17.966 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:17.966 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:56:17.966 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:17.967 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:17.967 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:56:17.967 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:17.967 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:17.968 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:17.968 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:56:17.968 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:17.968 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:17.969 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:17.969 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:17.969 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:56:17.969 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:17.970 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:17.970 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.990623_217.1", 2022-09-13T07:56:17.970 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:17.970 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:17.971 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:17.971 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T07:56:17.971 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:17.972 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:17.972 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:17.972 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:17.973 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:56:17.973 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:17.974 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:17.974 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:56:17.975 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:17.975 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:17.976 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:17.976 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:56:17.977 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:17.977 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:17.978 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:17.978 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:56:17.979 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:17.979 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:56:17.979 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:56:17.980 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:56:17.980 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:17.981 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:56:17Z" 2022-09-13T07:56:17.981 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:56:17.983 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.990623_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.990623Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:56:17Z" } 2022-09-13T07:56:17.984 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:56:17.985 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:17.985 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:17.986 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.990623_217.1 2022-09-13T07:56:22.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:22.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:22.989 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:56:23.044 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:56:23.045 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:56:23.093 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:23.090+0000 7f915b73d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:23.094 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:23.091+0000 7f915b73d600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:23.107 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:56:23.108 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:56:23.109 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:56:23.109 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:56:23.109 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:56:23.110 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:56:23.110 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:23.111 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:23.111 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:56:23.111 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:23.112 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:56:23.112 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:23.113 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:23.113 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:56:23.113 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:23.114 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:56:23.114 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:23.115 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:56:23.115 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:23.115 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:23.116 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:23.116 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:23.116 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:56:23.116 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:23.117 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:23.117 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:56:23.117 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:23.118 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:23.118 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:23.118 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:56:23.118 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:23.119 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:23.119 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:23.120 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:23.120 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:56:23.121 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:23.121 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:23.122 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.900410_211.1", 2022-09-13T07:56:23.122 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:23.123 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:23.123 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:23.124 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T07:56:23.124 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:23.125 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:23.125 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:23.125 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:23.126 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:56:23.126 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:23.127 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:23.127 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:56:23.128 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:23.128 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:23.129 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:23.129 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:56:23.130 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:23.130 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:23.131 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:23.131 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:56:23.132 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:23.132 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:56:23.133 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:56:23.133 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:56:23.133 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:23.134 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:56:23Z" 2022-09-13T07:56:23.135 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:56:23.136 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.900410_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.900410Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:56:23Z" } 2022-09-13T07:56:23.137 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:56:23.138 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:23.139 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:23.140 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.900410_211.1 2022-09-13T07:56:28.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:28.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:28.144 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:56:28.195 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:56:28.196 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:56:28.234 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:28.231+0000 7fa7e515a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:28.236 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:28.232+0000 7fa7e515a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:28.253 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:56:28.255 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:56:28.256 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:56:28.257 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:56:28.258 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:56:28.258 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:56:28.259 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:28.259 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:28.259 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:56:28.260 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:28.260 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:56:28.260 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:28.261 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:28.261 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:56:28.262 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:28.262 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:56:28.262 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:28.263 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:56:28.263 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:28.263 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:28.264 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:28.264 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:28.265 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:56:28.265 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:28.265 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:28.266 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:56:28.266 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:28.267 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:28.267 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:28.267 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:56:28.268 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:28.268 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:28.268 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:28.269 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:28.269 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:56:28.269 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:28.270 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:28.270 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.953175_216.1", 2022-09-13T07:56:28.271 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:28.271 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:28.272 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:28.274 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T07:56:28.274 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:28.275 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:28.275 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:28.275 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:28.276 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:56:28.276 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:28.277 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:28.277 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:56:28.278 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:28.278 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:28.278 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:28.279 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:56:28.279 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:28.280 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:28.280 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:28.281 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:56:28.281 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:28.281 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:56:28.282 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:56:28.282 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:56:28.283 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:28.283 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:56:28Z" 2022-09-13T07:56:28.283 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:56:28.285 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.953175_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.953175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:56:28Z" } 2022-09-13T07:56:28.286 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:56:28.287 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:28.288 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:28.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.953175_216.1 2022-09-13T07:56:33.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:33.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:33.293 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:56:33.378 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:56:33.379 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:56:33.435 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:33.427+0000 7f2def545600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:33.435 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:33.429+0000 7f2def545600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:33.450 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:56:33.450 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:56:33.450 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:56:33.451 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:56:33.451 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:56:33.451 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:56:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:56:33.452 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:33.453 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:56:33.453 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:33.453 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:56:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:33.454 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:56:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:56:33.455 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:33.456 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:56:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:33.457 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:56:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:33.458 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:56:33.459 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:33.460 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:33.460 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:33.461 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:33.461 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:56:33.462 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:33.462 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:33.462 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.861945_210.1", 2022-09-13T07:56:33.463 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:33.463 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:33.463 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:33.463 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T07:56:33.464 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:33.464 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:33.464 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:33.465 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:33.465 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:56:33.465 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:33.465 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:33.466 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:56:33.466 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:33.466 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:33.467 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:33.467 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:56:33.467 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:33.468 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:33.468 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:33.468 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:56:33.468 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:33.469 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:56:33.469 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:56:33.469 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:56:33.470 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:33.470 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:56:33Z" 2022-09-13T07:56:33.470 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:56:33.479 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.861945_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.861945Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:56:33Z" } 2022-09-13T07:56:33.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:56:33.480 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:33.481 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:33.481 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.861945_210.1 2022-09-13T07:56:38.482 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:38.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:38.484 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:56:38.535 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:56:38.536 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:56:38.579 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:38.574+0000 7f5cf5137600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:38.580 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:38.575+0000 7f5cf5137600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:38.594 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:56:38.595 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:56:38.595 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:56:38.595 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:56:38.596 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:56:38.596 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:56:38.597 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:38.598 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:38.598 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:56:38.599 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:38.599 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:56:38.600 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:38.600 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:38.601 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:56:38.601 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:38.602 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:56:38.602 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:38.602 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:56:38.603 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:38.603 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:38.604 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:38.604 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:38.605 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:56:38.605 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:38.606 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:38.606 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:56:38.607 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:38.607 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:38.608 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:38.608 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:56:38.609 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:38.609 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:38.610 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:38.610 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:38.611 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:56:38.611 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:38.612 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:38.612 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T07:56:38.613 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:38.613 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:38.614 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:38.614 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T07:56:38.615 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:38.615 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:38.615 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:38.616 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:38.616 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:56:38.617 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:38.617 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:38.618 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:56:38.618 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:38.619 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:38.619 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:38.620 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:56:38.620 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:38.621 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:38.621 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:38.622 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:56:38.622 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:38.623 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:56:38.623 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:56:38.624 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:56:38.624 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:38.625 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:56:38Z" 2022-09-13T07:56:38.625 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:56:38.627 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:56:38Z" } 2022-09-13T07:56:38.629 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:56:38.630 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:38.630 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:38.631 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.832045_205.1 2022-09-13T07:56:43.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:43.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:43.635 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:56:43.685 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:56:43.711 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:56:43.738 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:43.733+0000 7f6fff467600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:43.739 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:43.734+0000 7f6fff467600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:43.761 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:56:43.762 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:56:43.762 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:56:43.763 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:56:43.763 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:56:43.763 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:56:43.764 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:43.764 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:43.764 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:56:43.765 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:43.765 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:56:43.765 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:43.765 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:43.766 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:56:43.766 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:43.766 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:56:43.767 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:43.767 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:56:43.768 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:43.768 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:43.769 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:43.769 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:43.769 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:56:43.770 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:43.770 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:43.770 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:56:43.770 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:43.771 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:43.771 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:43.771 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:56:43.772 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:43.772 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:43.773 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:43.773 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:43.773 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:56:43.773 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:43.774 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:43.774 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:56:43.774 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:43.774 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:43.775 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:43.775 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:56:43.775 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:43.776 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:43.776 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:43.777 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:43.777 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:56:43.777 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:43.778 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:43.778 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:56:43.778 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:43.778 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:43.779 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:43.779 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:56:43.779 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:43.779 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:43.780 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:43.780 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:56:43.780 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:43.780 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:56:43.781 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:56:43.781 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:56:43.781 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:43.781 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:56:43Z" 2022-09-13T07:56:43.782 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:56:43.783 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:56:43Z" } 2022-09-13T07:56:43.784 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:56:43.784 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:43.785 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:43.785 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.834293_206.1 2022-09-13T07:56:48.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:48.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:48.790 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:56:48.841 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:56:48.842 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:56:48.888 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:48.884+0000 7fe11ca14600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:48.889 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:48.885+0000 7fe11ca14600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:48.907 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:56:48.907 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:56:48.908 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:56:48.908 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:56:48.908 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:56:48.908 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:56:48.909 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:48.909 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:48.909 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:56:48.910 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:48.911 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:56:48.911 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:48.912 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:48.912 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:56:48.913 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:48.913 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:56:48.913 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:48.914 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:56:48.914 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:48.914 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:48.914 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:48.915 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:48.915 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:56:48.915 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:48.915 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:48.916 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:56:48.916 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:48.916 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:48.916 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:48.917 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:56:48.917 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:48.917 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:48.917 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:48.918 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:48.918 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:56:48.918 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:48.918 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:48.919 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:56:48.919 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:48.919 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:48.920 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:48.920 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:56:48.920 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:48.921 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:48.921 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:48.922 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:48.922 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:56:48.923 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:48.923 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:48.924 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:56:48.924 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:48.925 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:48.925 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:48.925 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:56:48.926 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:48.926 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:48.927 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:48.927 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:56:48.927 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:48.927 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:56:48.928 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:56:48.928 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:56:48.928 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:48.929 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:56:48Z" 2022-09-13T07:56:48.929 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:56:48.930 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:56:48Z" } 2022-09-13T07:56:48.931 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:56:48.931 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:48.932 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:48.932 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.834293_206.1 2022-09-13T07:56:53.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:53.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:53.944 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:56:54.000 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:56:54.001 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:56:54.031 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:54.027+0000 7f81abae3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:54.032 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:54.028+0000 7f81abae3600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:54.049 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:56:54.049 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:56:54.050 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:56:54.050 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:56:54.050 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:56:54.051 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:56:54.051 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:54.052 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:54.052 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:56:54.052 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:54.052 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:56:54.053 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:54.053 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:54.053 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:56:54.053 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:54.054 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:56:54.054 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:54.054 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:56:54.054 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:54.055 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:54.055 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:54.055 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:54.056 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:56:54.056 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:54.056 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:54.057 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:56:54.057 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:54.057 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:54.058 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:54.058 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:56:54.058 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:54.058 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:54.059 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:54.059 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:54.059 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:56:54.059 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:54.060 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:54.060 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.107994_222.1", 2022-09-13T07:56:54.060 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:54.060 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:54.061 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:54.061 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.107994Z", 2022-09-13T07:56:54.061 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:54.062 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:54.062 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:54.062 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:54.062 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:56:54.063 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:54.063 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:54.063 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:56:54.063 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:54.064 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:54.064 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:54.064 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:56:54.064 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:54.065 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:54.065 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:54.065 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:56:54.065 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:54.065 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:56:54.066 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:56:54.066 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:56:54.066 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:54.067 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:56:54Z" 2022-09-13T07:56:54.067 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:56:54.068 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.107994_222.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.107994Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:56:54Z" } 2022-09-13T07:56:54.069 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.107994_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.107994Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:56:54.069 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:54.070 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.107994_222.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:54.070 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055099.107994_222.1 2022-09-13T07:56:59.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:59.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:56:59.076 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:56:59.136 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:56:59.137 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:56:59.183 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:59.179+0000 7f0fed8c5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:59.184 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:56:59.180+0000 7f0fed8c5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:56:59.237 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:56:59.237 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:56:59.237 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:56:59.238 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:56:59.238 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:56:59.238 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:56:59.239 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:59.239 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:59.239 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:56:59.239 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:59.240 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:56:59.240 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:59.240 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:59.240 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:56:59.241 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:59.241 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:56:59.241 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:59.241 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:56:59.242 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:59.242 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:59.242 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:59.242 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:59.243 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:56:59.243 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:59.243 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:59.244 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:56:59.244 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:59.244 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:59.244 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:59.245 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:56:59.245 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:59.245 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:59.245 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:59.246 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:59.246 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:56:59.246 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:59.246 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:59.247 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.994620_218.1", 2022-09-13T07:56:59.247 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:59.247 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:59.247 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:59.248 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.994620Z", 2022-09-13T07:56:59.248 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:59.248 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:59.248 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:59.249 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:56:59.249 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:56:59.249 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:56:59.250 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:56:59.250 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:56:59.250 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:56:59.251 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:56:59.251 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:56:59.251 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:56:59.252 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:56:59.252 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:59.252 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:56:59.252 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:56:59.253 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:59.253 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:56:59.253 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:56:59.253 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:56:59.254 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:56:59.254 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:56:59Z" 2022-09-13T07:56:59.254 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:56:59.255 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.994620_218.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.994620Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:56:59Z" } 2022-09-13T07:56:59.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.994620_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.994620Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:56:59.257 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:59.257 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.994620_218.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:56:59.258 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.994620_218.1 2022-09-13T07:57:04.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:04.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:04.263 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:57:04.350 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:57:04.350 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:57:04.389 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:04.385+0000 7f0ce3252600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:04.390 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:04.386+0000 7f0ce3252600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:04.422 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:57:04.423 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:57:04.423 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:57:04.424 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:57:04.424 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:57:04.425 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:57:04.425 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:04.425 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:04.426 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:57:04.426 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:04.427 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:57:04.427 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:04.428 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:04.428 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:57:04.429 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:04.429 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:57:04.430 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:04.430 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:57:04.431 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:04.431 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:04.432 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:04.432 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:04.433 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:57:04.433 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:04.433 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:04.433 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:57:04.434 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:04.434 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:04.434 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:04.434 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:57:04.435 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:04.435 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:04.435 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:04.435 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:04.436 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:57:04.436 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:04.436 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:04.436 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:57:04.437 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:04.437 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:04.437 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:04.437 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:57:04.438 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:04.438 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:04.438 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:04.439 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:04.439 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:57:04.439 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:04.439 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:04.440 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:57:04.440 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:04.440 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:04.440 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:04.441 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:57:04.441 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:04.441 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:04.442 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:04.442 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:57:04.442 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:04.443 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:57:04.443 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:57:04.443 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:57:04.443 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:04.444 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:57:04Z" 2022-09-13T07:57:04.444 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:57:04.445 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:57:04Z" } 2022-09-13T07:57:04.446 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:57:04.447 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:04.447 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:04.448 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.831134_204.1 2022-09-13T07:57:09.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:09.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:09.685 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:57:09.750 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:57:09.751 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:57:09.777 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:09.773+0000 7f7225c01600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:09.778 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:09.774+0000 7f7225c01600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:09.800 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:57:09.801 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:57:09.801 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:57:09.802 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:57:09.802 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:57:09.803 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:57:09.803 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:09.804 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:09.804 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:57:09.804 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:09.804 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:57:09.805 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:09.805 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:09.806 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:57:09.806 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:09.806 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:57:09.807 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:09.807 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:57:09.807 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:09.808 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:09.808 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:09.808 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:09.809 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:57:09.809 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:09.809 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:09.809 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:57:09.810 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:09.810 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:09.811 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:09.811 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:57:09.811 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:09.812 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:09.812 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:09.812 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:09.812 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:57:09.813 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:09.814 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:09.814 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.111513_224.1", 2022-09-13T07:57:09.814 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:09.815 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:09.815 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:09.816 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.111513Z", 2022-09-13T07:57:09.816 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:09.816 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:09.817 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:09.817 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:09.831 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:57:09.832 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:09.832 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:09.842 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:57:09.842 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:09.843 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:09.843 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:09.843 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:57:09.843 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:09.844 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:09.844 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:09.844 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:57:09.844 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:09.845 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:57:09.845 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:57:09.845 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:57:09.845 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:09.846 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:57:09Z" 2022-09-13T07:57:09.846 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:57:09.847 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.111513_224.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.111513Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:57:09Z" } 2022-09-13T07:57:09.848 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.111513_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.111513Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:57:09.849 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:09.849 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.111513_224.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:09.849 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055099.111513_224.1 2022-09-13T07:57:14.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:14.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:14.852 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:57:14.903 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:57:14.904 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:57:14.934 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:14.930+0000 7fed400f0600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:14.935 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:14.931+0000 7fed400f0600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:14.948 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:57:14.949 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:57:14.949 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:57:14.949 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:57:14.950 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:57:14.950 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:57:14.950 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:14.951 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:14.951 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:57:14.951 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:14.951 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:57:14.952 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:14.952 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:14.952 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:57:14.952 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:14.953 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:57:14.953 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:14.953 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:57:14.953 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:14.954 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:14.954 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:14.954 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:14.955 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:57:14.955 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:14.955 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:14.955 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:57:14.956 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:14.956 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:14.956 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:14.956 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:57:14.957 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:14.957 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:14.957 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:14.958 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:14.958 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:57:14.958 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:14.958 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:14.959 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:57:14.959 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:14.959 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:14.959 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:14.960 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:57:14.960 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:14.960 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:14.960 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:14.961 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:14.961 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:57:14.961 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:14.962 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:14.962 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:57:14.962 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:14.963 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:14.963 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:14.963 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:57:14.963 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:14.964 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:14.964 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:14.964 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:57:14.964 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:14.965 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:57:14.965 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:57:14.965 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:57:14.965 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:14.966 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:57:14Z" 2022-09-13T07:57:14.966 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:57:14.967 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:57:14Z" } 2022-09-13T07:57:14.968 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:57:14.968 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:14.969 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:14.969 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.834293_206.1 2022-09-13T07:57:19.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:19.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:19.973 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:57:20.077 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:57:20.078 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:57:20.181 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:20.167+0000 7f7bb6a73600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:20.181 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:20.168+0000 7f7bb6a73600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:20.205 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:57:20.205 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:57:20.206 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:57:20.206 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:57:20.207 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:57:20.207 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:57:20.208 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:20.208 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:20.208 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:57:20.208 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:20.209 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:57:20.209 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:20.209 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:20.209 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:57:20.210 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:20.210 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:57:20.211 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:20.211 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:57:20.211 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:20.211 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:20.212 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:20.212 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:20.212 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:57:20.212 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:20.213 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:20.213 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:57:20.213 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:20.213 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:20.214 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:20.215 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:57:20.215 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:20.215 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:20.216 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:20.216 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:20.216 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:57:20.217 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:20.217 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:20.217 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T07:57:20.217 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:20.218 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:20.218 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:20.218 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T07:57:20.219 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:20.219 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:20.219 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:20.220 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:20.220 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:57:20.220 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:20.220 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:20.221 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:57:20.221 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:20.221 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:20.221 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:20.223 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:57:20.223 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:20.223 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:20.223 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:20.224 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:57:20.224 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:20.224 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:57:20.224 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:57:20.225 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:57:20.225 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:20.225 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:57:20Z" 2022-09-13T07:57:20.225 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:57:20.227 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:57:20Z" } 2022-09-13T07:57:20.227 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:57:20.228 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:20.228 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:20.229 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.949151_215.1 2022-09-13T07:57:25.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:25.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:25.233 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:57:25.285 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:57:25.286 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:57:25.328 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:25.324+0000 7f15766fc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:25.329 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:25.325+0000 7f15766fc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:57:25.345 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:57:25.346 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:57:25.346 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:57:25.346 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:57:25.347 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:57:25.347 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:25.348 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:25.348 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:57:25.348 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:25.349 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:57:25.349 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:25.350 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:25.350 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:57:25.351 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:25.351 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:57:25.351 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:25.352 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:57:25.352 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:25.353 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:25.353 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:25.353 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:25.354 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:57:25.354 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:25.355 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:25.355 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:57:25.356 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:25.356 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:25.356 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:25.357 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:57:25.357 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:25.357 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:25.358 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:25.358 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:25.359 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:57:25.359 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:25.359 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:25.360 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.857076_208.1", 2022-09-13T07:57:25.360 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:25.361 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:25.361 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:25.362 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T07:57:25.362 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:25.362 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:25.363 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:25.363 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:25.364 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:57:25.364 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:25.364 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:25.365 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:57:25.365 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:25.366 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:25.367 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:25.367 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:57:25.368 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:25.368 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:25.369 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:25.369 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:57:25.370 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:25.370 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:57:25.371 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:57:25.371 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:57:25.371 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:25.372 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:57:25Z" 2022-09-13T07:57:25.372 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:57:25.373 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.857076_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.857076Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:57:25Z" } 2022-09-13T07:57:25.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:57:25.375 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:25.376 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:25.377 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.857076_208.1 2022-09-13T07:57:30.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:30.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:30.380 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:57:30.430 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:57:30.431 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:57:30.482 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:30.478+0000 7fc23cfbc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:30.482 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:30.479+0000 7fc23cfbc600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:30.501 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:57:30.502 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:57:30.502 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:57:30.503 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:57:30.503 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:57:30.503 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:57:30.504 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:30.504 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:30.504 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:57:30.505 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:30.505 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:57:30.506 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:30.506 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:30.507 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:57:30.507 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:30.508 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:57:30.508 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:30.509 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:57:30.509 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:30.510 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:30.510 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:30.511 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:30.511 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:57:30.512 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:30.512 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:30.512 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:57:30.513 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:30.514 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:30.514 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:30.514 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:57:30.515 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:30.515 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:30.515 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:30.516 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:30.516 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:57:30.516 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:30.517 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:30.517 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.907374_213.1", 2022-09-13T07:57:30.518 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:30.518 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:30.518 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:30.519 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.907374Z", 2022-09-13T07:57:30.519 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:30.519 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:30.519 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:30.520 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:30.520 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:57:30.520 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:30.521 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:30.521 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:57:30.521 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:30.521 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:30.522 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:30.522 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:57:30.522 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:30.522 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:30.523 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:30.523 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:57:30.523 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:30.525 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:57:30.526 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:57:30.526 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:57:30.527 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:30.527 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:57:30Z" 2022-09-13T07:57:30.528 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:57:30.530 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.907374_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.907374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:57:30Z" } 2022-09-13T07:57:30.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.907374_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.907374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:57:30.532 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:30.533 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.907374_213.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:30.533 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.907374_213.1 2022-09-13T07:57:35.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:35.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:35.537 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:57:35.589 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:57:35.590 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:57:35.636 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:35.632+0000 7fd710660600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:35.637 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:35.633+0000 7fd710660600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:35.660 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:57:35.661 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:57:35.661 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:57:35.662 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:57:35.662 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:57:35.662 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:57:35.663 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:35.663 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:35.663 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:57:35.664 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:35.664 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:57:35.664 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:35.664 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:35.665 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:57:35.665 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:35.666 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:57:35.666 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:35.666 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:57:35.666 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:35.667 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:35.667 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:35.667 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:35.668 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:57:35.668 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:35.668 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:35.669 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:57:35.669 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:35.669 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:35.670 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:35.670 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:57:35.670 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:35.671 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:35.671 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:35.671 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:35.671 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:57:35.672 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:35.672 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:35.672 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.187352_231.1", 2022-09-13T07:57:35.672 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:35.673 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:35.673 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:35.673 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T07:57:35.674 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:35.674 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:35.674 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:57:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:57:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:35.677 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:57:35.677 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:35.677 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:35.678 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:35.678 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:57:35.678 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:57:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:57:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:57:35.680 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:35.680 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:57:35Z" 2022-09-13T07:57:35.680 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:57:35.682 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.187352_231.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.187352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:57:35Z" } 2022-09-13T07:57:35.683 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.187352_231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.187352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:57:35.683 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:35.684 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.187352_231.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:35.684 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055099.187352_231.1 2022-09-13T07:57:40.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:40.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:40.693 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:57:40.747 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:57:40.747 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:57:40.778 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:40.774+0000 7f5ad9c50600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:40.780 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:40.775+0000 7f5ad9c50600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:40.798 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:57:40.799 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:57:40.800 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:57:40.800 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:57:40.801 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:57:40.801 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:57:40.802 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:40.802 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:40.803 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:57:40.803 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:40.804 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:57:40.804 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:40.804 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:40.805 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:57:40.805 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:40.806 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:57:40.806 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:40.807 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:57:40.808 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:40.808 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:40.809 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:40.810 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:40.810 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:57:40.811 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:40.811 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:40.812 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:57:40.812 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:40.812 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:40.813 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:40.813 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:57:40.814 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:40.814 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:40.815 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:40.815 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:40.816 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:57:40.816 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:40.817 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:40.817 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:57:40.818 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:40.819 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:40.819 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:40.820 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:57:40.821 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:40.821 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:40.822 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:40.822 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:40.823 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:57:40.823 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:40.824 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:40.824 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:57:40.825 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:40.825 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:40.826 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:40.826 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:57:40.827 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:40.827 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:40.828 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:40.828 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:57:40.829 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:40.829 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:57:40.830 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:57:40.830 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:57:40.831 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:40.832 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:57:40Z" 2022-09-13T07:57:40.832 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:57:40.833 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:57:40Z" } 2022-09-13T07:57:40.835 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:57:40.836 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:40.837 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:40.838 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.831134_204.1 2022-09-13T07:57:45.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:45.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:45.842 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:57:45.905 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:57:45.906 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:57:45.946 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:45.940+0000 7f1ebec82600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:45.947 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:45.942+0000 7f1ebec82600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:45.985 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:57:45.986 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:57:45.986 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:57:45.987 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:57:45.987 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:57:45.988 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:57:45.988 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:45.988 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:45.989 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:57:45.989 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:45.990 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:57:45.990 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:45.991 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:45.991 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:57:45.992 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:45.993 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:57:45.993 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:45.994 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:57:45.994 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:45.995 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:45.995 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:45.996 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:45.997 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:57:45.997 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:45.998 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:45.999 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:57:45.999 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:45.999 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:46.000 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:46.000 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:57:46.001 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:46.001 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:46.002 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:46.002 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:46.002 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:57:46.003 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:46.003 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:46.004 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.854520_207.1", 2022-09-13T07:57:46.004 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:46.004 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:46.005 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:46.005 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T07:57:46.006 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:46.006 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:46.007 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:57:46.009 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:46.009 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:46.010 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:57:46.010 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:46.010 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:46.011 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:46.011 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:57:46.012 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:46.012 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:46.013 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:46.013 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:57:46.013 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:46.014 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:57:46.014 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:57:46.015 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:57:46.015 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:46.016 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:57:45Z" 2022-09-13T07:57:46.017 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:57:46.017 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.854520_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.854520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:57:45Z" } 2022-09-13T07:57:46.019 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:57:46.020 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:46.021 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:46.021 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.854520_207.1 2022-09-13T07:57:51.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:51.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:51.025 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:57:51.111 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:57:51.112 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:57:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:51.157+0000 7fe6749b5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:51.161 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:51.158+0000 7fe6749b5600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:51.179 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:57:51.180 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:57:51.180 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:57:51.181 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:57:51.181 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:57:51.181 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:57:51.182 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:51.182 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:51.182 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:57:51.182 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:51.183 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:57:51.183 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:51.183 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:51.184 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:57:51.184 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:51.184 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:57:51.185 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:51.185 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:57:51.185 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:51.185 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:51.186 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:51.186 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:51.186 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:57:51.187 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:51.187 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:51.187 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:57:51.188 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:51.188 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:51.188 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:51.188 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:57:51.189 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:51.189 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:51.189 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:51.189 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:51.190 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:57:51.190 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:51.190 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:51.191 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.906479_212.1", 2022-09-13T07:57:51.191 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:51.191 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:51.191 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:51.192 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.906479Z", 2022-09-13T07:57:51.192 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:51.192 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:51.192 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:51.193 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:51.193 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:57:51.194 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:51.194 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:51.195 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:57:51.195 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:51.195 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:51.195 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:51.196 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:57:51.196 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:51.196 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:51.196 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:51.197 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:57:51.197 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:51.197 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:57:51.198 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:57:51.198 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:57:51.198 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:51.199 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:57:51Z" 2022-09-13T07:57:51.199 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:57:51.201 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.906479_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.906479Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:57:51Z" } 2022-09-13T07:57:51.201 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.906479_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.906479Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:57:51.202 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:51.203 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.906479_212.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:51.204 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.906479_212.1 2022-09-13T07:57:56.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:56.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:57:56.207 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:57:56.258 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:57:56.258 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:57:56.287 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:56.284+0000 7f2b2632c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:56.288 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:57:56.285+0000 7f2b2632c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:57:56.304 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:57:56.304 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:57:56.305 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:57:56.305 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:57:56.306 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:57:56.306 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:57:56.307 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:56.307 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:56.308 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:57:56.308 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:56.309 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:57:56.309 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:56.310 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:56.310 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:57:56.311 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:56.311 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:57:56.312 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:56.312 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:57:56.313 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:56.314 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:56.314 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:56.315 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:56.315 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:57:56.315 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:56.316 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:56.316 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:57:56.316 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:56.317 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:56.317 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:56.317 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:57:56.318 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:56.318 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:56.319 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:56.319 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:56.319 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:57:56.319 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:56.320 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:56.320 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.220845_233.1", 2022-09-13T07:57:56.320 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:56.321 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:56.321 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:56.321 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T07:57:56.322 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:56.322 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:56.322 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:56.323 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:57:56.323 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:57:56.323 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:57:56.324 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:57:56.324 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:57:56.324 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:57:56.324 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:57:56.325 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:57:56.325 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:57:56.325 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:57:56.326 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:56.326 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:57:56.326 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:57:56.327 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:56.327 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:57:56.327 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:57:56.328 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:57:56.328 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:57:56.328 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:57:56Z" 2022-09-13T07:57:56.329 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:57:56.332 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.220845_233.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.220845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:57:56Z" } 2022-09-13T07:57:56.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:57:56.334 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:56.335 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:57:56.335 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055099.220845_233.1 2022-09-13T07:58:01.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:01.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:01.339 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:58:01.389 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:58:01.390 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:58:01.424 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:01.419+0000 7f4462cea600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:01.425 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:01.420+0000 7f4462cea600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:01.448 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:58:01.448 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:58:01.449 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:58:01.449 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:58:01.450 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:58:01.450 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:58:01.451 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:01.451 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:01.451 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:58:01.452 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:01.452 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:58:01.453 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:01.453 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:01.453 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:58:01.454 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:01.454 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:58:01.455 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:01.455 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:58:01.456 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:01.456 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:01.457 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:01.457 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:01.457 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:58:01.458 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:01.458 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:01.459 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:58:01.459 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:01.459 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:01.460 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:01.460 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:58:01.461 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:01.461 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:01.461 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:01.462 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:01.462 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:58:01.463 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:01.463 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:01.464 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.854520_207.1", 2022-09-13T07:58:01.464 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:01.464 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:01.465 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:01.465 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T07:58:01.466 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:01.467 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:01.467 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:01.468 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:01.468 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:58:01.468 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:01.469 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:01.469 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:58:01.470 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:01.471 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:01.471 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:01.472 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:58:01.472 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:01.473 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:01.473 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:01.474 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:58:01.475 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:01.475 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:58:01.476 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:58:01.476 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:58:01.476 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:01.477 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:58:01Z" 2022-09-13T07:58:01.477 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:58:01.479 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.854520_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.854520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:58:01Z" } 2022-09-13T07:58:01.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:58:01.481 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:01.482 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:01.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.854520_207.1 2022-09-13T07:58:06.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:06.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:06.486 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:58:06.541 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:58:06.541 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:58:06.578 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:06.566+0000 7f4ec501a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:06.578 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:06.567+0000 7f4ec501a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:06.588 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:58:06.589 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:58:06.589 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:58:06.590 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:58:06.590 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:58:06.591 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:58:06.591 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:06.592 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:06.592 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:58:06.593 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:06.593 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:58:06.593 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:06.593 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:06.594 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:58:06.594 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:06.595 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:58:06.595 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:06.595 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:58:06.596 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:06.596 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:06.597 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:06.597 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:06.598 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:58:06.598 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:06.599 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:06.599 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:58:06.600 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:06.600 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:06.601 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:06.601 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:58:06.602 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:06.602 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:06.602 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:06.603 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:06.603 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:58:06.604 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:06.604 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:06.605 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.861945_210.1", 2022-09-13T07:58:06.606 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:06.606 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:06.607 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:06.607 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T07:58:06.607 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:06.608 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:06.608 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:06.609 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:06.609 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:58:06.610 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:06.610 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:06.610 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:58:06.611 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:06.611 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:06.612 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:06.612 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:58:06.612 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:06.613 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:06.613 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:06.614 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:58:06.614 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:06.615 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:58:06.615 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:58:06.615 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:58:06.616 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:06.616 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:58:06Z" 2022-09-13T07:58:06.617 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:58:06.618 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.861945_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.861945Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:58:06Z" } 2022-09-13T07:58:06.619 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:58:06.620 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:06.621 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:06.621 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.861945_210.1 2022-09-13T07:58:11.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:11.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:11.624 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:58:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:58:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:58:11.714 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:11.709+0000 7f7a135a6600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:11.715 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:11.710+0000 7f7a135a6600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:11.734 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:58:11.735 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:58:11.735 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:58:11.735 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:58:11.736 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:58:11.736 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:58:11.736 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:11.736 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:11.737 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:58:11.737 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:11.737 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:58:11.737 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:11.738 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:11.738 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:58:11.738 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:11.739 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:58:11.739 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:11.739 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:58:11.740 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:11.740 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:11.741 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:11.741 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:11.741 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:58:11.741 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:11.742 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:11.742 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:58:11.742 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:11.743 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:11.744 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:11.744 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:58:11.744 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:11.745 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:11.745 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:11.745 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:11.745 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:58:11.746 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:11.746 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:11.746 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.907374_213.1", 2022-09-13T07:58:11.746 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:11.747 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:11.747 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:11.748 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.907374Z", 2022-09-13T07:58:11.748 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:11.749 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:11.749 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:11.749 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:11.750 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:58:11.750 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:11.751 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:11.751 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:58:11.751 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:11.752 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:11.752 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:11.753 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:58:11.753 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:11.753 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:11.754 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:11.755 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:58:11.755 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:11.755 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:58:11.755 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:58:11.756 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:58:11.756 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:11.756 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:58:11Z" 2022-09-13T07:58:11.757 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:58:11.758 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.907374_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.907374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:58:11Z" } 2022-09-13T07:58:11.759 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.907374_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.907374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:58:11.759 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:11.760 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.907374_213.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:11.760 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.907374_213.1 2022-09-13T07:58:16.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:16.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:16.764 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:58:16.821 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:58:16.822 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:58:16.863 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:16.859+0000 7f5d78caa600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:16.864 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:16.860+0000 7f5d78caa600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:16.881 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:58:16.881 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:58:16.882 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:58:16.882 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:58:16.883 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:58:16.883 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:58:16.884 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:16.884 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:16.885 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:58:16.885 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:16.886 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:58:16.886 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:16.887 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:16.887 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:58:16.887 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:16.887 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:58:16.888 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:16.888 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:58:16.888 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:16.889 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:16.889 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:16.889 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:16.889 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:58:16.890 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:16.890 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:16.890 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:58:16.890 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:16.891 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:16.891 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:16.891 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:58:16.891 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:16.892 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:16.892 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:16.892 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:16.892 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:58:16.893 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:16.893 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:16.893 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T07:58:16.893 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:16.894 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:16.894 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:16.894 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T07:58:16.894 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:16.895 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:16.896 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:16.896 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:16.896 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:58:16.897 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:16.897 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:16.897 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:58:16.898 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:16.898 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:16.898 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:16.899 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:58:16.899 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:16.899 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:16.899 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:16.900 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:58:16.900 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:16.900 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:58:16.900 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:58:16.901 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:58:16.901 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:16.901 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:58:16Z" 2022-09-13T07:58:16.901 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:58:16.903 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:58:16Z" } 2022-09-13T07:58:16.903 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:58:16.904 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:16.905 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:16.905 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.832045_205.1 2022-09-13T07:58:21.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:21.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:21.914 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:58:21.972 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:58:21.973 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:58:22.006 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:22.002+0000 7f80c0920600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:22.007 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:22.003+0000 7f80c0920600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:22.022 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:58:22.023 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:58:22.023 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:58:22.024 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:58:22.024 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:58:22.025 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:58:22.025 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:22.026 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:22.026 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:58:22.027 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:22.027 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:58:22.028 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:22.028 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:22.029 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:58:22.029 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:22.030 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:58:22.031 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:22.031 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:58:22.031 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:22.032 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:22.032 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:22.033 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:22.033 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:58:22.033 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:22.034 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:22.035 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:58:22.035 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:22.036 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:22.036 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:22.036 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:58:22.037 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:22.037 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:22.037 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:22.043 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:22.043 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:58:22.043 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:22.044 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:22.044 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.861945_210.1", 2022-09-13T07:58:22.044 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:22.045 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:22.045 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:22.046 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T07:58:22.047 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:22.047 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:22.048 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:22.048 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:22.049 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:58:22.049 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:22.050 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:22.050 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:58:22.050 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:22.051 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:22.051 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:22.052 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:58:22.052 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:22.053 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:22.053 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:22.054 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:58:22.054 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:22.055 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:58:22.055 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:58:22.056 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:58:22.056 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:22.057 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:58:22Z" 2022-09-13T07:58:22.057 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:58:22.058 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.861945_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.861945Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:58:22Z" } 2022-09-13T07:58:22.059 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:58:22.060 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:22.060 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:22.061 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.861945_210.1 2022-09-13T07:58:27.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:27.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:27.066 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:58:27.164 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:58:27.165 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:58:27.254 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:27.246+0000 7fb4b924c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:27.254 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:27.247+0000 7fb4b924c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:27.272 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:58:27.272 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:58:27.273 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:58:27.273 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:58:27.274 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:58:27.274 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:58:27.275 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:27.275 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:27.275 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:58:27.277 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:27.277 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:58:27.278 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:27.278 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:27.279 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:58:27.279 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:27.279 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:58:27.280 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:27.280 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:58:27.280 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:27.281 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:27.281 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:27.282 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:27.282 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:58:27.282 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:27.283 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:27.283 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:58:27.283 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:27.284 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:27.284 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:27.284 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:58:27.285 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:27.285 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:27.286 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:27.286 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:27.286 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:58:27.287 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:27.287 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:27.287 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.858000_209.1", 2022-09-13T07:58:27.288 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:27.288 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:27.288 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:27.289 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T07:58:27.289 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:27.290 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:27.290 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:27.291 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:27.291 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:58:27.291 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:27.292 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:27.292 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:58:27.293 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:27.293 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:27.294 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:27.294 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:58:27.294 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:27.295 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:27.295 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:27.296 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:58:27.296 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:27.297 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:58:27.297 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:58:27.297 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:58:27.298 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:27.298 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:58:27Z" 2022-09-13T07:58:27.299 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:58:27.301 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.858000_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.858000Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:58:27Z" } 2022-09-13T07:58:27.302 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:58:27.303 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:27.304 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:27.305 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.858000_209.1 2022-09-13T07:58:32.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:32.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:32.308 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:58:32.357 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:58:32.358 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:58:32.388 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:32.384+0000 7fc5b3afd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:32.389 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:32.385+0000 7fc5b3afd600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:32.404 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:58:32.404 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:58:32.405 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:58:32.405 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:58:32.406 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:58:32.406 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:58:32.407 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:32.407 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:32.408 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:58:32.408 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:32.409 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:58:32.409 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:32.410 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:32.410 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:58:32.411 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:32.411 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:58:32.412 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:32.412 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:58:32.413 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:32.413 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:32.414 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:32.414 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:32.415 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:58:32.415 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:32.416 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:32.416 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:58:32.417 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:32.417 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:32.418 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:32.418 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:58:32.419 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:32.419 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:32.420 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:32.421 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:32.421 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:58:32.422 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:32.422 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:32.423 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T07:58:32.423 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:32.424 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:32.424 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:32.425 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T07:58:32.425 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:32.426 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:32.426 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:32.427 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:32.427 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:58:32.428 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:32.428 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:32.429 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:58:32.429 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:32.430 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:32.430 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:32.431 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:58:32.431 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:32.432 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:32.433 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:32.433 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:58:32.433 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:32.434 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:58:32.434 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:58:32.435 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:58:32.435 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:32.436 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:58:32Z" 2022-09-13T07:58:32.436 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:58:32.439 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:58:32Z" } 2022-09-13T07:58:32.440 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:58:32.441 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:32.442 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:32.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.834293_206.1 2022-09-13T07:58:37.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:37.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:37.449 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:58:37.501 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:58:37.502 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:58:37.545 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:37.541+0000 7ffa7a586600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:37.546 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:37.542+0000 7ffa7a586600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:37.571 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:58:37.572 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:58:37.572 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:58:37.572 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:58:37.573 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:58:37.573 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:58:37.573 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:37.573 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:58:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:58:37.574 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:37.575 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:37.575 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:58:37.575 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:37.575 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:58:37.576 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:37.576 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:58:37.576 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:37.576 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:37.577 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:37.577 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:37.577 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:58:37.577 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:37.578 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:37.578 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:58:37.578 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:37.579 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:37.579 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:37.579 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:58:37.580 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:37.580 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:37.580 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:37.581 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:37.581 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:58:37.581 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:37.581 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:37.582 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.854520_207.1", 2022-09-13T07:58:37.582 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:37.582 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:37.582 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:37.583 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T07:58:37.583 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:37.583 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:37.584 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:37.584 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:37.584 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:58:37.584 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:37.585 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:37.585 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:58:37.585 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:37.585 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:37.586 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:37.586 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:58:37.586 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:37.587 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:37.587 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:37.587 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:58:37.587 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:58:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:58:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:58:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:37.589 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:58:37Z" 2022-09-13T07:58:37.589 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:58:37.589 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.854520_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.854520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:58:37Z" } 2022-09-13T07:58:37.590 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:58:37.591 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:37.591 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:37.592 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.854520_207.1 2022-09-13T07:58:42.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:42.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:42.595 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:58:42.647 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:58:42.648 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:58:42.675 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:42.671+0000 7f77e532c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:42.676 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:42.672+0000 7f77e532c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:42.689 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:58:42.690 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:58:42.690 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:58:42.691 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:58:42.691 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:58:42.692 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:58:42.692 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:42.692 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:42.692 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:58:42.693 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:42.693 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:58:42.694 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:42.694 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:42.694 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:58:42.695 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:42.695 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:58:42.695 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:42.695 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:58:42.696 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:42.696 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:42.696 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:42.696 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:42.697 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:58:42.697 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:42.697 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:42.697 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:58:42.698 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:42.698 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:42.698 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:42.699 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:58:42.699 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:42.699 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:42.699 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:42.700 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:42.700 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:58:42.700 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:42.700 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:42.701 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.858000_209.1", 2022-09-13T07:58:42.701 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:42.701 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:42.702 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:42.702 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T07:58:42.702 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:42.702 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:42.703 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:42.703 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:42.703 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:58:42.703 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:42.704 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:42.704 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:58:42.704 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:42.704 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:42.705 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:42.705 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:58:42.705 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:42.705 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:42.706 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:42.706 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:58:42.706 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:42.707 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:58:42.707 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:58:42.707 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:58:42.707 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:42.708 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:58:42Z" 2022-09-13T07:58:42.708 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:58:42.709 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.858000_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.858000Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:58:42Z" } 2022-09-13T07:58:42.710 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:58:42.710 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:42.711 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:42.712 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.858000_209.1 2022-09-13T07:58:47.713 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:47.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:47.716 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:58:47.812 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:58:47.813 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:58:47.913 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:47.908+0000 7fc4bb400600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:47.913 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:47.909+0000 7fc4bb400600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:47.933 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:58:47.933 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:58:47.933 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:58:47.934 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:58:47.934 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:58:47.934 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:58:47.935 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:47.935 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:47.935 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:58:47.935 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:47.936 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:58:47.936 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:47.936 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:47.937 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:58:47.937 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:47.937 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:58:47.938 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:47.938 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:58:47.938 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:47.938 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:47.939 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:47.939 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:47.939 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:58:47.940 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:47.940 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:47.940 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:58:47.940 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:47.941 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:47.941 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:47.941 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:58:47.941 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:47.942 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:47.942 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:47.942 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:47.942 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:58:47.943 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:47.943 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:47.943 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T07:58:47.944 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:47.944 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:47.944 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:47.944 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T07:58:47.945 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:47.945 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:47.945 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:47.945 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:47.946 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:58:47.946 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:47.946 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:47.947 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:58:47.947 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:47.947 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:47.947 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:47.948 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:58:47.948 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:47.948 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:47.949 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:47.949 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:58:47.949 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:47.950 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:58:47.950 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:58:47.950 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:58:47.951 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:47.952 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:58:47Z" 2022-09-13T07:58:47.952 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:58:47.955 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:58:47Z" } 2022-09-13T07:58:47.956 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:58:47.957 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:47.958 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:47.959 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.832045_205.1 2022-09-13T07:58:52.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:52.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:52.964 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:58:53.014 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:58:53.015 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:58:53.049 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:53.045+0000 7fdccb159600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:53.050 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:53.046+0000 7fdccb159600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:53.067 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:58:53.067 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:58:53.068 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:58:53.068 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:58:53.069 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:58:53.069 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:58:53.069 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:53.070 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:53.070 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:58:53.071 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:53.071 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:58:53.072 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:53.072 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:53.073 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:58:53.073 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:53.074 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:58:53.074 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:53.075 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:58:53.075 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:53.076 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:53.076 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:53.077 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:53.077 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:58:53.077 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:53.078 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:53.078 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:58:53.079 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:53.079 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:53.080 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:53.080 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:58:53.080 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:53.080 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:53.081 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:53.081 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:53.081 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:58:53.081 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:53.082 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:53.082 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:58:53.082 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:53.082 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:53.083 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:53.083 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:58:53.083 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:53.084 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:53.084 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:53.085 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:53.085 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:58:53.086 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:53.086 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:53.087 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:58:53.087 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:53.088 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:53.088 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:53.088 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:58:53.089 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:53.089 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:53.090 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:53.091 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:58:53.091 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:53.092 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:58:53.092 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:58:53.092 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:58:53.093 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:53.093 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:58:53Z" 2022-09-13T07:58:53.093 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:58:53.095 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:58:53Z" } 2022-09-13T07:58:53.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:58:53.097 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:53.098 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:53.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.831134_204.1 2022-09-13T07:58:58.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:58.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:58:58.104 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:58:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:58:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:58:58.189 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:58.185+0000 7fbb40c1c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:58.191 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:58:58.186+0000 7fbb40c1c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:58:58.204 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:58:58.205 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:58:58.205 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:58:58.205 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:58:58.206 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:58:58.206 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:58:58.207 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:58.207 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:58.208 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:58:58.208 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:58.209 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:58:58.209 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:58.210 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:58.210 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:58:58.211 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:58.211 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:58:58.212 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:58.213 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:58:58.213 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:58.214 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:58.214 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:58.215 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:58.215 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:58:58.216 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:58.216 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:58.217 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:58:58.217 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:58.218 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:58.218 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:58.219 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:58:58.219 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:58.220 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:58.220 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:58.221 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:58.221 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:58:58.222 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:58.222 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:58.223 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.990623_217.1", 2022-09-13T07:58:58.223 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:58.224 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:58.224 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:58.225 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T07:58:58.225 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:58.226 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:58.226 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:58.227 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:58:58.227 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:58:58.228 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:58:58.228 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:58:58.229 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:58:58.229 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:58:58.230 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:58:58.231 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:58:58.231 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:58:58.231 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:58:58.232 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:58.232 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:58:58.233 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:58:58.233 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:58.234 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:58:58.234 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:58:58.235 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:58:58.235 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:58:58.236 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:58:58Z" 2022-09-13T07:58:58.236 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:58:58.239 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.990623_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.990623Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:58:58Z" } 2022-09-13T07:58:58.240 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:58:58.241 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:58.242 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:58:58.243 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.990623_217.1 2022-09-13T07:59:03.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:03.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:03.247 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:59:03.304 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:59:03.305 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:59:03.346 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:03.341+0000 7f4215bb2600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:03.347 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:03.342+0000 7f4215bb2600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:03.363 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:59:03.365 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:59:03.365 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:59:03.366 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:59:03.366 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:59:03.367 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:59:03.367 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:03.368 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:03.368 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:59:03.369 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:03.369 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:59:03.370 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:03.370 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:03.371 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:59:03.371 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:03.372 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:59:03.372 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:03.373 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:59:03.373 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:03.374 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:03.374 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:03.375 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:03.375 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:59:03.376 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:03.376 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:03.378 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:59:03.378 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:03.379 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:03.379 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:03.380 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:59:03.381 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:03.381 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:03.382 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:03.382 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:03.383 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:59:03.383 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:03.384 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:03.384 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.857076_208.1", 2022-09-13T07:59:03.385 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:03.385 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:03.386 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:03.386 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T07:59:03.387 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:03.387 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:03.388 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:03.388 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:03.389 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:59:03.389 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:03.390 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:03.390 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:59:03.391 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:03.391 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:03.392 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:03.392 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:59:03.393 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:03.393 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:03.394 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:03.394 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:59:03.395 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:03.395 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:59:03.396 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:59:03.396 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:59:03.397 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:03.397 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:59:03Z" 2022-09-13T07:59:03.398 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:59:03.400 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.857076_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.857076Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:59:03Z" } 2022-09-13T07:59:03.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:59:03.402 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:03.403 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:03.404 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.857076_208.1 2022-09-13T07:59:08.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:08.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:08.407 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:59:08.459 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:59:08.460 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:59:08.499 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:08.494+0000 7f163ca39600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:08.499 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:08.495+0000 7f163ca39600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:08.522 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:59:08.523 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:59:08.523 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:59:08.524 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:59:08.524 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:59:08.524 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:59:08.525 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:08.525 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:08.525 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:59:08.525 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:08.526 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:59:08.526 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:08.526 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:08.527 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:59:08.527 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:08.527 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:59:08.528 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:08.528 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:59:08.528 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:08.529 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:08.530 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:08.530 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:08.531 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:59:08.531 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:08.532 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:08.532 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:59:08.532 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:08.532 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:08.533 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:08.533 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:59:08.533 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:08.533 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:08.534 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:08.534 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:08.534 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:59:08.535 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:08.535 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:08.535 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T07:59:08.536 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:08.536 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:08.537 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:08.537 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T07:59:08.538 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:08.538 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:08.539 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:08.539 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:08.540 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:59:08.540 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:08.541 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:08.541 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:59:08.542 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:08.542 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:08.543 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:08.543 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:59:08.544 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:08.544 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:08.545 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:08.545 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:59:08.546 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:08.546 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:59:08.547 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:59:08.547 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:59:08.548 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:08.548 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:59:08Z" 2022-09-13T07:59:08.549 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:59:08.549 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:59:08Z" } 2022-09-13T07:59:08.551 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:59:08.552 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:08.552 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:08.553 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.949151_215.1 2022-09-13T07:59:13.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:13.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:13.558 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:59:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:59:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:59:13.647 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:13.642+0000 7f0d30239600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:13.648 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:13.643+0000 7f0d30239600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:13.661 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:59:13.662 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:59:13.663 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:59:13.663 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:59:13.663 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:59:13.664 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:59:13.664 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:13.664 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:13.665 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:59:13.665 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:13.665 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:59:13.666 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:13.666 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:13.666 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:59:13.667 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:13.667 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:59:13.667 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:13.667 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:59:13.668 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:13.668 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:13.669 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:13.669 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:13.670 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:59:13.671 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:13.671 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:13.672 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:59:13.672 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:13.673 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:13.673 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:13.673 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:59:13.673 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:13.674 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:13.674 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:13.674 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:13.675 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:59:13.675 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:13.678 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:13.678 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.857076_208.1", 2022-09-13T07:59:13.678 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:13.679 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:13.680 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:13.680 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T07:59:13.680 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:13.680 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:13.681 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:13.681 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:13.681 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:59:13.681 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:13.682 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:13.682 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:59:13.682 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:13.683 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:13.683 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:13.684 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:59:13.684 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:13.685 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:13.685 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:13.685 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:59:13.686 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:13.687 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:59:13.687 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:59:13.688 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:59:13.689 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:13.690 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:59:13Z" 2022-09-13T07:59:13.690 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:59:13.693 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.857076_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.857076Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:59:13Z" } 2022-09-13T07:59:13.694 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:59:13.695 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:13.699 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:13.700 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.857076_208.1 2022-09-13T07:59:18.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:18.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:18.703 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:59:18.801 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:59:18.803 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:59:18.879 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:18.876+0000 7f3867dbe600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:18.882 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:18.877+0000 7f3867dbe600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:18.894 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:59:18.895 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:59:18.895 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:59:18.895 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:59:18.896 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:59:18.896 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:59:18.896 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:18.896 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:18.897 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:59:18.897 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:18.897 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:59:18.898 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:18.898 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:18.898 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:59:18.899 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:18.899 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:59:18.899 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:18.900 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:59:18.900 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:18.900 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:18.900 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:18.901 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:18.901 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:59:18.901 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:18.901 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:18.902 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:59:18.902 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:18.902 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:18.902 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:18.903 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:59:18.903 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:18.903 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:18.904 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:18.904 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:18.904 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:59:18.904 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:18.905 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:18.905 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T07:59:18.905 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:18.905 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:18.906 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:18.906 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T07:59:18.906 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:18.906 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:18.907 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:18.907 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:18.907 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:59:18.907 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:18.908 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:18.908 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:59:18.908 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:18.909 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:18.909 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:18.909 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:59:18.909 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:18.910 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:18.910 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:18.910 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:59:18.910 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:18.911 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:59:18.911 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:59:18.911 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:59:18.911 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:18.912 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:59:18Z" 2022-09-13T07:59:18.912 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:59:18.913 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:59:18Z" } 2022-09-13T07:59:18.914 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:59:18.915 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:18.915 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:18.915 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.949151_215.1 2022-09-13T07:59:23.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:23.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:23.920 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:59:23.971 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:59:23.972 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:59:24.015 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:24.012+0000 7f5eb2365600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:24.016 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:24.013+0000 7f5eb2365600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:24.035 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:59:24.036 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:59:24.036 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:59:24.037 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:59:24.037 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:59:24.038 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:59:24.038 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:24.039 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:24.039 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:59:24.039 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:24.040 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:59:24.040 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:24.040 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:24.041 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:59:24.041 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:24.042 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:59:24.042 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:24.043 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:59:24.043 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:24.043 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:24.044 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:24.044 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:24.045 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:59:24.045 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:24.046 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:24.046 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:59:24.047 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:24.048 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:24.049 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:24.049 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:59:24.049 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:24.050 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:24.051 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:24.051 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:24.051 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:59:24.052 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:24.052 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:24.053 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.340682_237.1", 2022-09-13T07:59:24.053 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:24.054 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:24.054 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:24.054 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T07:59:24.055 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:24.055 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:24.055 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:24.055 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:24.056 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:59:24.056 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:24.056 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:24.056 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:59:24.057 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:24.057 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:24.057 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:24.058 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:59:24.058 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:24.058 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:24.058 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:24.059 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:59:24.059 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:24.059 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:59:24.059 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:59:24.060 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:59:24.060 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:24.060 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:59:24Z" 2022-09-13T07:59:24.061 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:59:24.062 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.340682_237.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.340682Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:59:24Z" } 2022-09-13T07:59:24.063 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.340682_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.340682Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:59:24.063 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:24.064 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.340682_237.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:24.064 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055099.340682_237.1 2022-09-13T07:59:29.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:29.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:29.068 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:59:29.119 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:59:29.120 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:59:29.168 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:29.164+0000 7f64e5c33600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:29.169 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:29.165+0000 7f64e5c33600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:29.202 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:59:29.203 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:59:29.204 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:59:29.204 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:59:29.204 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:59:29.205 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:59:29.205 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:29.206 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:29.207 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:59:29.208 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:29.209 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:59:29.210 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:29.210 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:29.211 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:59:29.211 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:29.211 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:59:29.212 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:29.212 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:59:29.213 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:29.213 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:29.214 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:29.215 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:29.215 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:59:29.216 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:29.216 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:29.216 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:59:29.217 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:29.217 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:29.218 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:29.219 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:59:29.219 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:29.219 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:29.220 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:29.220 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:29.220 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:59:29.220 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:29.221 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:29.221 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.911619_214.1", 2022-09-13T07:59:29.222 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:29.222 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:29.222 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:29.223 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T07:59:29.223 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:29.223 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:29.223 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:29.224 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:29.224 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:59:29.224 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:29.225 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:29.225 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:59:29.225 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:29.225 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:29.226 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:29.226 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:59:29.226 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:29.226 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:29.227 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:29.227 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:59:29.227 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:29.228 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:59:29.228 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:59:29.228 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:59:29.228 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:29.229 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:59:29Z" 2022-09-13T07:59:29.229 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:59:29.229 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.911619_214.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.911619Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:59:29Z" } 2022-09-13T07:59:29.230 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:59:29.230 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:29.231 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:29.231 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.911619_214.1 2022-09-13T07:59:34.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:34.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:34.235 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:59:34.286 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:59:34.287 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:59:34.333 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:34.329+0000 7fef8be6a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:34.334 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:34.330+0000 7fef8be6a600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:34.352 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:59:34.352 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:59:34.353 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:59:34.353 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:59:34.354 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:59:34.354 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:59:34.355 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:34.355 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:34.356 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:59:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:59:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:59:34.359 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:34.359 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:59:34.359 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:34.360 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:59:34.360 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:34.361 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:34.361 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:34.361 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:34.361 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:59:34.362 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:34.362 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:34.362 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:59:34.363 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:34.363 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:34.363 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:34.363 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:59:34.364 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:34.364 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:34.364 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:34.364 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:34.365 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:59:34.365 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:34.365 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:34.365 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T07:59:34.366 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:34.366 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:34.366 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:34.367 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T07:59:34.367 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:34.367 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:34.367 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:34.368 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:34.368 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:59:34.368 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:34.368 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:34.369 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:59:34.369 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:34.369 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:34.369 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:34.370 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:59:34.370 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:34.370 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:34.371 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:34.371 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:59:34.371 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:34.371 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:59:34.372 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:59:34.372 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:59:34.372 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:34.372 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:59:34Z" 2022-09-13T07:59:34.373 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:59:34.374 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:59:34Z" } 2022-09-13T07:59:34.375 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:59:34.375 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:34.376 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:34.376 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.949151_215.1 2022-09-13T07:59:39.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:39.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:39.379 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:59:39.432 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:59:39.432 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:59:39.475 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:39.471+0000 7f45e6d0c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:39.476 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:39.472+0000 7f45e6d0c600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:39.494 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:59:39.495 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:59:39.496 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:59:39.496 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:59:39.497 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:59:39.498 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:59:39.498 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:39.498 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:39.499 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:59:39.499 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:39.499 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:59:39.499 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:39.500 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:39.500 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:59:39.500 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:39.500 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:59:39.501 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:39.501 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:59:39.502 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:39.502 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:39.503 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:39.503 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:39.504 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:59:39.504 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:39.505 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:39.505 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:59:39.506 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:39.506 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:39.507 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:39.507 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:59:39.508 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:39.508 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:39.509 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:39.509 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:39.510 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:59:39.510 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:39.511 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:39.511 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.831134_204.1", 2022-09-13T07:59:39.512 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:39.512 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:39.513 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:39.513 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T07:59:39.513 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:39.513 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:39.514 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:39.514 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:39.515 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:59:39.515 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:39.515 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:39.516 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:59:39.516 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:39.517 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:39.517 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:39.518 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:59:39.518 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:39.519 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:39.519 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:39.520 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:59:39.520 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:39.521 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:59:39.521 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:59:39.522 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:59:39.522 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:39.523 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:59:39Z" 2022-09-13T07:59:39.523 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:59:39.525 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.831134_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.831134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:59:39Z" } 2022-09-13T07:59:39.526 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:59:39.527 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:39.528 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:39.528 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.831134_204.1 2022-09-13T07:59:44.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:44.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:44.531 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:59:44.597 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:59:44.597 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:59:44.644 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:44.639+0000 7fcc897e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:44.645 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:44.640+0000 7fcc897e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:44.702 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:59:44.702 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:59:44.702 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:59:44.703 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:59:44.703 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:59:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:59:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:59:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:44.706 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:59:44.706 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:44.707 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:44.707 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:59:44.708 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:44.708 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:59:44.708 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:44.708 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:59:44.711 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:44.713 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:59:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:44.722 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:44.722 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:59:44.723 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:59:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:44.728 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:44.729 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:44.730 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:44.730 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:59:44.731 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:44.731 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:44.732 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.105395_221.1", 2022-09-13T07:59:44.732 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:44.733 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:44.733 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:44.734 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T07:59:44.734 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:44.734 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:44.735 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:44.735 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:59:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:44.737 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:44.737 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:59:44.738 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:44.738 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:44.738 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:44.739 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:59:44.740 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:44.741 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:44.741 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:44.742 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:59:44.742 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:44.743 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:59:44.743 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:59:44.744 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:59:44.744 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:44.744 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:59:44Z" 2022-09-13T07:59:44.745 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:59:44.747 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.105395_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.105395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:59:44Z" } 2022-09-13T07:59:44.748 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.105395_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.105395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:59:44.749 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:44.750 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.105395_221.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:44.751 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055099.105395_221.1 2022-09-13T07:59:49.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:49.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:49.756 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:59:49.812 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:59:49.812 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:59:49.850 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:49.846+0000 7f900d702600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:49.851 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:49.847+0000 7f900d702600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:49.865 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:59:49.866 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:59:49.866 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:59:49.867 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:59:49.867 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:59:49.867 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:59:49.868 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:49.868 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:49.868 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:59:49.868 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:49.869 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:59:49.869 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:49.869 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:49.870 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:59:49.870 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:49.870 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:59:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:59:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:59:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:49.873 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:49.873 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:59:49.873 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:49.874 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:49.874 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:49.874 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:59:49.875 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:49.876 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:49.876 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:49.877 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:49.877 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:59:49.877 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:49.878 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:49.878 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.858000_209.1", 2022-09-13T07:59:49.879 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:49.879 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:49.880 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:49.881 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T07:59:49.881 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:49.882 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:49.882 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:49.882 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:49.883 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:59:49.883 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:49.893 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:49.893 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:59:49.893 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:49.894 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:49.894 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:49.895 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:59:49.895 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:49.895 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:49.896 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:49.896 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:59:49.896 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:49.897 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:59:49.897 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:59:49.897 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:59:49.900 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:49.900 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:59:49Z" 2022-09-13T07:59:49.901 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:59:49.903 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.858000_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.858000Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:59:49Z" } 2022-09-13T07:59:49.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:59:49.905 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:49.906 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:49.907 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.858000_209.1 2022-09-13T07:59:54.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:54.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T07:59:54.911 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T07:59:54.962 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T07:59:54.963 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T07:59:54.994 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:54.987+0000 7f3fa4741600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:54.995 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T07:59:54.988+0000 7f3fa4741600 20 RGW-SYNC:meta: read sync status 2022-09-13T07:59:55.011 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T07:59:55.012 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T07:59:55.012 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T07:59:55.012 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T07:59:55.013 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T07:59:55.014 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T07:59:55.014 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:55.014 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:55.015 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T07:59:55.015 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:55.016 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T07:59:55.017 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:55.017 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:55.018 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T07:59:55.019 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:55.019 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T07:59:55.020 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:55.020 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T07:59:55.021 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:55.022 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:55.022 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:55.023 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:55.023 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T07:59:55.024 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:55.024 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:55.025 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T07:59:55.025 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:55.025 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:55.026 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:55.027 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T07:59:55.028 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:55.028 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:55.029 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:55.029 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:55.030 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T07:59:55.030 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:55.031 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:55.031 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.861945_210.1", 2022-09-13T07:59:55.032 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:55.032 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:55.032 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:55.033 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T07:59:55.033 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:55.033 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:55.033 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:55.034 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T07:59:55.034 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T07:59:55.034 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T07:59:55.034 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T07:59:55.035 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T07:59:55.035 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T07:59:55.035 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T07:59:55.036 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T07:59:55.036 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T07:59:55.036 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T07:59:55.036 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:55.037 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T07:59:55.037 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T07:59:55.037 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:55.037 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T07:59:55.038 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T07:59:55.038 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T07:59:55.039 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T07:59:55.039 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T07:59:54Z" 2022-09-13T07:59:55.039 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T07:59:55.040 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.861945_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.861945Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T07:59:54Z" } 2022-09-13T07:59:55.040 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T07:59:55.041 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:55.041 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T07:59:55.042 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.861945_210.1 2022-09-13T08:00:00.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:00:00.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:00:00.046 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:00:00.106 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:00:00.107 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:00:00.146 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:00.141+0000 7f2382c0b600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:00:00.146 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:00.142+0000 7f2382c0b600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:00:00.165 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:00:00.166 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:00:00.167 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:00:00.167 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:00:00.167 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:00:00.168 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:00:00.168 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:00.168 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:00.169 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:00:00.169 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:00.169 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:00:00.170 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:00.170 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:00.171 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:00:00.171 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:00.172 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:00:00.172 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:00.173 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:00:00.173 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:00.174 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:00.174 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:00.175 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:00.175 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:00:00.176 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:00.177 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:00.177 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:00:00.177 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:00.177 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:00.178 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:00.178 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:00:00.178 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:00.179 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:00.179 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:00.179 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:00.179 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:00:00.180 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:00.180 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:00.181 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.070128_219.1", 2022-09-13T08:00:00.181 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:00.181 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:00.181 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:00.182 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T08:00:00.182 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:00.182 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:00.183 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:00.183 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:00.183 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:00:00.183 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:00.184 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:00.184 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:00:00.184 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:00.185 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:00.185 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:00.185 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:00:00.185 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:00.186 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:00.186 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:00.186 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:00:00.186 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:00.187 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:00:00.187 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:00:00.187 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:00:00.187 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:00.188 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:00:00Z" 2022-09-13T08:00:00.188 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:00:00.189 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.070128_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.070128Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:00:00Z" } 2022-09-13T08:00:00.190 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:00:00.191 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:00:00.191 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:00:00.192 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055099.070128_219.1 2022-09-13T08:00:05.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:00:05.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:00:05.204 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:00:05.255 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:00:05.255 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:00:05.293 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:05.290+0000 7f8f6a35b600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:00:05.293 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:05.291+0000 7f8f6a35b600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:00:05.308 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:00:05.308 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:00:05.308 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:00:05.309 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:00:05.309 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:00:05.309 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:00:05.310 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:05.310 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:05.310 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:00:05.310 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:05.311 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:00:05.311 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:05.311 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:05.312 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:00:05.312 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:05.312 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:00:05.312 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:05.313 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:00:05.313 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:05.313 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:05.313 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:05.315 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:05.316 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:00:05.316 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:05.316 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:05.318 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:00:05.319 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:05.319 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:05.320 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:05.320 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:00:05.321 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:05.321 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:05.322 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:05.322 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:05.323 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:00:05.323 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:05.324 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:05.324 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.858000_209.1", 2022-09-13T08:00:05.325 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:05.325 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:05.326 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:05.327 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T08:00:05.327 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:05.328 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:05.328 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:05.329 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:05.329 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:00:05.330 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:05.331 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:05.331 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:00:05.332 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:05.332 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:05.333 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:05.333 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:00:05.334 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:05.334 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:05.335 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:05.335 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:00:05.336 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:05.336 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:00:05.337 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:00:05.337 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:00:05.337 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:05.346 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:00:05Z" 2022-09-13T08:00:05.346 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:00:05.348 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.858000_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.858000Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:00:05Z" } 2022-09-13T08:00:05.349 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:00:05.350 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:00:05.351 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:00:05.352 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.858000_209.1 2022-09-13T08:00:10.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:00:10.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:00:10.356 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:00:10.414 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:00:10.415 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:00:10.448 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:10.444+0000 7f4676abd600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:00:10.449 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:10.445+0000 7f4676abd600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:00:10.467 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:00:10.467 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:00:10.468 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:00:10.468 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:00:10.469 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:00:10.469 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:00:10.470 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:10.470 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:10.471 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:00:10.472 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:10.472 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:00:10.472 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:10.473 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:10.473 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:00:10.473 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:10.474 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:00:10.474 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:10.476 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:00:10.477 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:10.477 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:10.478 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:10.479 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:10.479 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:00:10.480 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:10.480 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:10.480 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:00:10.481 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:10.481 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:10.482 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:10.482 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:00:10.483 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:10.483 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:10.484 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:10.484 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:10.485 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:00:10.485 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:10.486 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:10.487 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T08:00:10.487 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:10.487 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:10.488 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:10.488 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T08:00:10.488 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:10.488 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:10.489 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:10.489 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:10.489 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:00:10.489 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:10.490 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:10.490 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:00:10.490 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:10.491 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:10.491 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:10.491 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:00:10.491 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:10.492 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:10.492 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:10.492 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:00:10.492 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:10.493 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:00:10.493 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:00:10.493 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:00:10.493 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:10.494 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:00:10Z" 2022-09-13T08:00:10.494 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:00:10.495 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:00:10Z" } 2022-09-13T08:00:10.496 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:00:10.497 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:00:10.498 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:00:10.498 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.832045_205.1 2022-09-13T08:00:15.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:00:15.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:00:15.501 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:00:15.553 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:00:15.553 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:00:15.589 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:15.581+0000 7f9f24394600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:00:15.590 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:15.582+0000 7f9f24394600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:00:15.601 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:00:15.602 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:00:15.602 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:00:15.603 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:00:15.604 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:00:15.604 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:00:15.605 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:15.606 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:15.606 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:00:15.607 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:15.608 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:00:15.608 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:15.609 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:15.611 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:00:15.611 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:15.611 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:00:15.612 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:15.612 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:00:15.612 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:15.613 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:15.614 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:15.614 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:15.614 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:00:15.615 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:15.615 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:15.615 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:00:15.615 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:15.616 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:15.616 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:15.616 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:00:15.616 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:15.617 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:15.617 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:15.617 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:15.617 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:00:15.618 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:15.618 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:15.618 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.834293_206.1", 2022-09-13T08:00:15.618 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:15.619 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:15.620 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:15.620 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T08:00:15.621 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:15.621 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:15.622 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:15.622 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:15.622 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:00:15.623 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:15.623 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:15.623 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:00:15.624 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:15.624 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:15.624 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:15.624 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:00:15.625 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:15.625 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:15.625 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:15.625 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:00:15.625 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:15.626 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:00:15.626 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:00:15.626 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:00:15.627 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:15.627 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:00:15Z" 2022-09-13T08:00:15.629 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:00:15.630 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.834293_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.834293Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:00:15Z" } 2022-09-13T08:00:15.631 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:00:15.632 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:00:15.632 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:00:15.633 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.834293_206.1 2022-09-13T08:00:20.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:00:20.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:00:20.635 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:00:20.685 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:00:20.686 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:00:20.716 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:20.711+0000 7f9f42b43600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:00:20.717 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:20.712+0000 7f9f42b43600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:00:20.736 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:00:20.736 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:00:20.737 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:00:20.737 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:00:20.737 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:00:20.737 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:00:20.738 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:20.738 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:20.738 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:00:20.739 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:20.739 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:00:20.739 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:20.739 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:20.740 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:00:20.740 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:20.740 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:00:20.740 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:20.741 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:00:20.741 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:20.741 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:20.741 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:20.742 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:20.742 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:00:20.742 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:20.742 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:20.743 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:00:20.743 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:20.743 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:20.743 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:20.744 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:00:20.744 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:20.744 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:20.744 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:20.745 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:20.745 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:00:20.746 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:20.746 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:20.746 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.854520_207.1", 2022-09-13T08:00:20.747 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:20.747 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:20.747 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:20.747 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T08:00:20.747 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:20.748 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:20.748 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:20.748 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:20.749 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:00:20.749 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:20.749 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:20.749 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:00:20.750 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:20.750 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:20.750 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:20.750 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:00:20.751 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:20.751 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:20.751 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:20.751 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:00:20.752 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:20.752 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:00:20.752 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:00:20.752 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:00:20.753 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:20.753 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:00:20Z" 2022-09-13T08:00:20.753 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:00:20.754 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.854520_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.854520Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:00:20Z" } 2022-09-13T08:00:20.754 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:00:20.755 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:00:20.756 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:00:20.756 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.854520_207.1 2022-09-13T08:00:25.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:00:25.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:00:25.760 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:00:25.856 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:00:25.857 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:00:25.949 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:25.944+0000 7fa6e1ca7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:00:25.950 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:25.946+0000 7fa6e1ca7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:00:25.965 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:00:25.966 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:00:25.966 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:00:25.967 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:00:25.967 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:00:25.968 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:00:25.968 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:25.969 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:25.969 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:00:25.969 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:25.970 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:00:25.970 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:25.971 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:25.971 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:00:25.972 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:25.972 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:00:25.973 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:25.973 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:00:25.973 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:25.974 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:25.974 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:25.975 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:25.975 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:00:25.976 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:25.976 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:25.977 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:00:25.977 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:25.978 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:25.978 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:25.979 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:00:25.979 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:25.980 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:25.980 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:25.981 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:25.981 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:00:25.982 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:25.982 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:25.983 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.832045_205.1", 2022-09-13T08:00:25.983 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:25.983 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:25.984 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:25.984 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T08:00:25.985 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:25.985 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:25.986 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:25.986 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:25.987 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:00:25.987 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:25.987 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:00:25.988 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:00:25.988 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:00:25.988 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:00:25.989 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:00:25.989 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:00:25.990 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:25.990 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:25.991 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:25.992 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:00:25.992 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:25.993 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:00:25.993 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:00:25.994 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:00:25.995 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:25.995 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:00:25Z" 2022-09-13T08:00:25.996 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:00:25.998 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.832045_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.832045Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:00:25Z" } 2022-09-13T08:00:25.999 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:00:26.000 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:00:26.001 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:00:26.002 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.832045_205.1 2022-09-13T08:00:31.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2022-09-13T08:00:31.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:00:31.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:00:31.020 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-13T08:00:31.070 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:00:31.070 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:00:31.092 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:31.086+0000 7f13e6059600 0 failed reading obj info from .rgw.root:zone_info.4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9: (2) No such file or directory 2022-09-13T08:00:31.093 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:31.086+0000 7f13e6059600 0 WARNING: could not read zone params for zone id=4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9 name=test-zone3 2022-09-13T08:00:31.093 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:31.086+0000 7f13e6059600 0 failed reading obj info from .rgw.root:zone_info.7103daa4-1b45-47a1-bb49-5d9877f6d137: (2) No such file or directory 2022-09-13T08:00:31.093 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:00:31.086+0000 7f13e6059600 0 WARNING: could not read zone params for zone id=7103daa4-1b45-47a1-bb49-5d9877f6d137 name=test-zone1 2022-09-13T08:00:31.094 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:00:31.094 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T08:00:31.095 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "test-zone2", 2022-09-13T08:00:31.095 INFO:teuthology.orchestra.run.smithi083.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-13T08:00:31.095 INFO:teuthology.orchestra.run.smithi083.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-13T08:00:31.096 INFO:teuthology.orchestra.run.smithi083.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-13T08:00:31.096 INFO:teuthology.orchestra.run.smithi083.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-13T08:00:31.096 INFO:teuthology.orchestra.run.smithi083.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-13T08:00:31.097 INFO:teuthology.orchestra.run.smithi083.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-13T08:00:31.098 INFO:teuthology.orchestra.run.smithi083.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-13T08:00:31.098 INFO:teuthology.orchestra.run.smithi083.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-13T08:00:31.099 INFO:teuthology.orchestra.run.smithi083.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-13T08:00:31.099 INFO:teuthology.orchestra.run.smithi083.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-13T08:00:31.100 INFO:teuthology.orchestra.run.smithi083.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-13T08:00:31.100 INFO:teuthology.orchestra.run.smithi083.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-13T08:00:31.100 INFO:teuthology.orchestra.run.smithi083.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-13T08:00:31.101 INFO:teuthology.orchestra.run.smithi083.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-13T08:00:31.101 INFO:teuthology.orchestra.run.smithi083.stdout: "system_key": { 2022-09-13T08:00:31.102 INFO:teuthology.orchestra.run.smithi083.stdout: "access_key": "G693TB5XEJ3206QY", 2022-09-13T08:00:31.102 INFO:teuthology.orchestra.run.smithi083.stdout: "secret_key": "QS5zddodwd26OlSDjoxox3W0BzgFaGzV" 2022-09-13T08:00:31.103 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:31.103 INFO:teuthology.orchestra.run.smithi083.stdout: "placement_pools": [ 2022-09-13T08:00:31.104 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:31.104 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "default-placement", 2022-09-13T08:00:31.105 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:00:31.105 INFO:teuthology.orchestra.run.smithi083.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-13T08:00:31.105 INFO:teuthology.orchestra.run.smithi083.stdout: "storage_classes": { 2022-09-13T08:00:31.106 INFO:teuthology.orchestra.run.smithi083.stdout: "STANDARD": { 2022-09-13T08:00:31.106 INFO:teuthology.orchestra.run.smithi083.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-13T08:00:31.106 INFO:teuthology.orchestra.run.smithi083.stdout: "compression_type": "random" 2022-09-13T08:00:31.107 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:31.107 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:31.107 INFO:teuthology.orchestra.run.smithi083.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-13T08:00:31.107 INFO:teuthology.orchestra.run.smithi083.stdout: "index_type": 0 2022-09-13T08:00:31.108 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:31.108 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:31.108 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T08:00:31.109 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T08:00:31.109 INFO:teuthology.orchestra.run.smithi083.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-13T08:00:31.110 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:00:31.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:00:31.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:00:31.113 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-13T08:00:31.162 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:00:31.163 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:00:31.202 INFO:teuthology.orchestra.run.smithi083.stderr:Sending period to new master zone 7103daa4-1b45-47a1-bb49-5d9877f6d137 2022-09-13T08:00:31.232 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:00:31.233 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:00:31.233 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 4, 2022-09-13T08:00:31.233 INFO:teuthology.orchestra.run.smithi083.stdout: "predecessor_uuid": "0f1e9404-34a5-47ad-8196-42a4e6e2ff9c", 2022-09-13T08:00:31.234 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": [], 2022-09-13T08:00:31.234 INFO:teuthology.orchestra.run.smithi083.stdout: "period_map": { 2022-09-13T08:00:31.234 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:00:31.235 INFO:teuthology.orchestra.run.smithi083.stdout: "zonegroups": [ 2022-09-13T08:00:31.235 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:31.235 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "test-zonegroup", 2022-09-13T08:00:31.235 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "test-zonegroup", 2022-09-13T08:00:31.236 INFO:teuthology.orchestra.run.smithi083.stdout: "api_name": "", 2022-09-13T08:00:31.236 INFO:teuthology.orchestra.run.smithi083.stdout: "is_master": true, 2022-09-13T08:00:31.236 INFO:teuthology.orchestra.run.smithi083.stdout: "endpoints": [ 2022-09-13T08:00:31.237 INFO:teuthology.orchestra.run.smithi083.stdout: "http://smithi036.front.sepia.ceph.com:8000" 2022-09-13T08:00:31.237 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T08:00:31.237 INFO:teuthology.orchestra.run.smithi083.stdout: "hostnames": [], 2022-09-13T08:00:31.238 INFO:teuthology.orchestra.run.smithi083.stdout: "hostnames_s3website": [], 2022-09-13T08:00:31.238 INFO:teuthology.orchestra.run.smithi083.stdout: "master_zone": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T08:00:31.238 INFO:teuthology.orchestra.run.smithi083.stdout: "zones": [ 2022-09-13T08:00:31.238 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:31.239 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T08:00:31.239 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "test-zone2", 2022-09-13T08:00:31.239 INFO:teuthology.orchestra.run.smithi083.stdout: "endpoints": [ 2022-09-13T08:00:31.240 INFO:teuthology.orchestra.run.smithi083.stdout: "http://smithi083.front.sepia.ceph.com:8000" 2022-09-13T08:00:31.240 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T08:00:31.240 INFO:teuthology.orchestra.run.smithi083.stdout: "log_meta": false, 2022-09-13T08:00:31.241 INFO:teuthology.orchestra.run.smithi083.stdout: "log_data": true, 2022-09-13T08:00:31.241 INFO:teuthology.orchestra.run.smithi083.stdout: "bucket_index_max_shards": 11, 2022-09-13T08:00:31.241 INFO:teuthology.orchestra.run.smithi083.stdout: "read_only": false, 2022-09-13T08:00:31.242 INFO:teuthology.orchestra.run.smithi083.stdout: "tier_type": "", 2022-09-13T08:00:31.242 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_from_all": false, 2022-09-13T08:00:31.242 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_from": [], 2022-09-13T08:00:31.243 INFO:teuthology.orchestra.run.smithi083.stdout: "redirect_zone": "", 2022-09-13T08:00:31.243 INFO:teuthology.orchestra.run.smithi083.stdout: "supported_features": [ 2022-09-13T08:00:31.243 INFO:teuthology.orchestra.run.smithi083.stdout: "resharding" 2022-09-13T08:00:31.244 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:00:31.244 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:31.245 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:31.245 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9", 2022-09-13T08:00:31.246 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "test-zone3", 2022-09-13T08:00:31.246 INFO:teuthology.orchestra.run.smithi083.stdout: "endpoints": [ 2022-09-13T08:00:31.247 INFO:teuthology.orchestra.run.smithi083.stdout: "http://smithi036.front.sepia.ceph.com:8001" 2022-09-13T08:00:31.247 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T08:00:31.248 INFO:teuthology.orchestra.run.smithi083.stdout: "log_meta": false, 2022-09-13T08:00:31.248 INFO:teuthology.orchestra.run.smithi083.stdout: "log_data": true, 2022-09-13T08:00:31.249 INFO:teuthology.orchestra.run.smithi083.stdout: "bucket_index_max_shards": 11, 2022-09-13T08:00:31.249 INFO:teuthology.orchestra.run.smithi083.stdout: "read_only": false, 2022-09-13T08:00:31.249 INFO:teuthology.orchestra.run.smithi083.stdout: "tier_type": "", 2022-09-13T08:00:31.250 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_from_all": true, 2022-09-13T08:00:31.251 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_from": [], 2022-09-13T08:00:31.251 INFO:teuthology.orchestra.run.smithi083.stdout: "redirect_zone": "", 2022-09-13T08:00:31.252 INFO:teuthology.orchestra.run.smithi083.stdout: "supported_features": [ 2022-09-13T08:00:31.252 INFO:teuthology.orchestra.run.smithi083.stdout: "resharding" 2022-09-13T08:00:31.253 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:00:31.253 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:31.253 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:31.254 INFO:teuthology.orchestra.run.smithi083.stdout: "id": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T08:00:31.255 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "test-zone1", 2022-09-13T08:00:31.255 INFO:teuthology.orchestra.run.smithi083.stdout: "endpoints": [ 2022-09-13T08:00:31.256 INFO:teuthology.orchestra.run.smithi083.stdout: "http://smithi036.front.sepia.ceph.com:8000" 2022-09-13T08:00:31.256 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T08:00:31.257 INFO:teuthology.orchestra.run.smithi083.stdout: "log_meta": false, 2022-09-13T08:00:31.258 INFO:teuthology.orchestra.run.smithi083.stdout: "log_data": true, 2022-09-13T08:00:31.258 INFO:teuthology.orchestra.run.smithi083.stdout: "bucket_index_max_shards": 11, 2022-09-13T08:00:31.259 INFO:teuthology.orchestra.run.smithi083.stdout: "read_only": false, 2022-09-13T08:00:31.260 INFO:teuthology.orchestra.run.smithi083.stdout: "tier_type": "", 2022-09-13T08:00:31.260 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_from_all": true, 2022-09-13T08:00:31.260 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_from": [], 2022-09-13T08:00:31.261 INFO:teuthology.orchestra.run.smithi083.stdout: "redirect_zone": "", 2022-09-13T08:00:31.262 INFO:teuthology.orchestra.run.smithi083.stdout: "supported_features": [ 2022-09-13T08:00:31.262 INFO:teuthology.orchestra.run.smithi083.stdout: "resharding" 2022-09-13T08:00:31.263 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:00:31.263 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:31.264 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T08:00:31.264 INFO:teuthology.orchestra.run.smithi083.stdout: "placement_targets": [ 2022-09-13T08:00:31.265 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:31.265 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "default-placement", 2022-09-13T08:00:31.266 INFO:teuthology.orchestra.run.smithi083.stdout: "tags": [], 2022-09-13T08:00:31.266 INFO:teuthology.orchestra.run.smithi083.stdout: "storage_classes": [ 2022-09-13T08:00:31.267 INFO:teuthology.orchestra.run.smithi083.stdout: "STANDARD" 2022-09-13T08:00:31.267 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:00:31.268 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:31.268 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T08:00:31.269 INFO:teuthology.orchestra.run.smithi083.stdout: "default_placement": "default-placement", 2022-09-13T08:00:31.269 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T08:00:31.270 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_policy": { 2022-09-13T08:00:31.270 INFO:teuthology.orchestra.run.smithi083.stdout: "groups": [] 2022-09-13T08:00:31.271 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:31.271 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled_features": [ 2022-09-13T08:00:31.272 INFO:teuthology.orchestra.run.smithi083.stdout: "resharding" 2022-09-13T08:00:31.272 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:00:31.273 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:31.273 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2022-09-13T08:00:31.274 INFO:teuthology.orchestra.run.smithi083.stdout: "short_zone_ids": [ 2022-09-13T08:00:31.274 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:31.275 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T08:00:31.275 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 2827250327 2022-09-13T08:00:31.276 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:31.276 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:31.277 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9", 2022-09-13T08:00:31.277 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 1723930414 2022-09-13T08:00:31.277 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:31.278 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:00:31.278 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T08:00:31.279 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 2837352077 2022-09-13T08:00:31.279 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:31.280 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:00:31.280 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:31.282 INFO:teuthology.orchestra.run.smithi083.stdout: "master_zonegroup": "test-zonegroup", 2022-09-13T08:00:31.282 INFO:teuthology.orchestra.run.smithi083.stdout: "master_zone": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T08:00:31.282 INFO:teuthology.orchestra.run.smithi083.stdout: "period_config": { 2022-09-13T08:00:31.283 INFO:teuthology.orchestra.run.smithi083.stdout: "bucket_quota": { 2022-09-13T08:00:31.283 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled": false, 2022-09-13T08:00:31.283 INFO:teuthology.orchestra.run.smithi083.stdout: "check_on_raw": false, 2022-09-13T08:00:31.284 INFO:teuthology.orchestra.run.smithi083.stdout: "max_size": -1, 2022-09-13T08:00:31.284 INFO:teuthology.orchestra.run.smithi083.stdout: "max_size_kb": 0, 2022-09-13T08:00:31.284 INFO:teuthology.orchestra.run.smithi083.stdout: "max_objects": -1 2022-09-13T08:00:31.285 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:31.285 INFO:teuthology.orchestra.run.smithi083.stdout: "user_quota": { 2022-09-13T08:00:31.286 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled": false, 2022-09-13T08:00:31.286 INFO:teuthology.orchestra.run.smithi083.stdout: "check_on_raw": false, 2022-09-13T08:00:31.287 INFO:teuthology.orchestra.run.smithi083.stdout: "max_size": -1, 2022-09-13T08:00:31.287 INFO:teuthology.orchestra.run.smithi083.stdout: "max_size_kb": 0, 2022-09-13T08:00:31.287 INFO:teuthology.orchestra.run.smithi083.stdout: "max_objects": -1 2022-09-13T08:00:31.288 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:31.288 INFO:teuthology.orchestra.run.smithi083.stdout: "user_ratelimit": { 2022-09-13T08:00:31.288 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_ops": 0, 2022-09-13T08:00:31.288 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_ops": 0, 2022-09-13T08:00:31.289 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_bytes": 0, 2022-09-13T08:00:31.289 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_bytes": 0, 2022-09-13T08:00:31.289 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled": false 2022-09-13T08:00:31.289 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:31.290 INFO:teuthology.orchestra.run.smithi083.stdout: "bucket_ratelimit": { 2022-09-13T08:00:31.290 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_ops": 0, 2022-09-13T08:00:31.291 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_ops": 0, 2022-09-13T08:00:31.291 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_bytes": 0, 2022-09-13T08:00:31.292 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_bytes": 0, 2022-09-13T08:00:31.292 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled": false 2022-09-13T08:00:31.293 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:31.293 INFO:teuthology.orchestra.run.smithi083.stdout: "anonymous_ratelimit": { 2022-09-13T08:00:31.294 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_ops": 0, 2022-09-13T08:00:31.294 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_ops": 0, 2022-09-13T08:00:31.295 INFO:teuthology.orchestra.run.smithi083.stdout: "max_read_bytes": 0, 2022-09-13T08:00:31.295 INFO:teuthology.orchestra.run.smithi083.stdout: "max_write_bytes": 0, 2022-09-13T08:00:31.296 INFO:teuthology.orchestra.run.smithi083.stdout: "enabled": false 2022-09-13T08:00:31.296 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:00:31.297 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:00:31.297 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T08:00:31.298 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_name": "test-realm", 2022-09-13T08:00:31.298 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:00:31.299 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:00:31.300 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2022-09-13T08:01:31.301 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-45 2022-09-13T08:01:31.303 DEBUG:boto:path=/yxvvuh-45/ 2022-09-13T08:01:31.303 DEBUG:boto:auth_path=/yxvvuh-45/ 2022-09-13T08:01:31.304 DEBUG:boto:Method: PUT 2022-09-13T08:01:31.304 DEBUG:boto:Path: /yxvvuh-45/ 2022-09-13T08:01:31.305 DEBUG:boto:Data: 2022-09-13T08:01:31.305 DEBUG:boto:Headers: {} 2022-09-13T08:01:31.306 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T08:01:31.306 DEBUG:boto:Port: 8000 2022-09-13T08:01:31.307 DEBUG:boto:Params: {} 2022-09-13T08:01:31.308 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:01:31.309 DEBUG:boto:Token: None 2022-09-13T08:01:31.309 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:01:31 GMT /yxvvuh-45/ 2022-09-13T08:01:31.310 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rGQ51Cjt3r7RJwjnDr1kkXsulRw= 2022-09-13T08:01:31.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:01:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rGQ51Cjt3r7RJwjnDr1kkXsulRw=', 'Content-Length': '0'} 2022-09-13T08:01:31.344 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000c407a3c89309934-00632038db-140b-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:01:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:01:31.345 DEBUG:boto:path=/yxvvuh-45/testredirect 2022-09-13T08:01:31.346 DEBUG:boto:auth_path=/yxvvuh-45/testredirect 2022-09-13T08:01:31.346 DEBUG:boto:Method: PUT 2022-09-13T08:01:31.347 DEBUG:boto:Path: /yxvvuh-45/testredirect 2022-09-13T08:01:31.347 DEBUG:boto:Data: 2022-09-13T08:01:31.348 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-13T08:01:31.348 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T08:01:31.349 DEBUG:boto:Port: 8000 2022-09-13T08:01:31.349 DEBUG:boto:Params: {} 2022-09-13T08:01:31.350 DEBUG:boto:Token: None 2022-09-13T08:01:31.350 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Tue, 13 Sep 2022 08:01:31 GMT /yxvvuh-45/testredirect 2022-09-13T08:01:31.351 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:woq36g5w5nUY99CwthhyuzS4zyo= 2022-09-13T08:01:31.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 08:01:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:woq36g5w5nUY99CwthhyuzS4zyo='} 2022-09-13T08:01:31.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b9606f3beee4312-00632038db-140b-test-zone1'), ('Date', 'Tue, 13 Sep 2022 08:01:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:01:31.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:01:31.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:01:31.364 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:01:31.419 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:01:31.420 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:01:31.452 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T08:01:31.466 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T08:01:31.467 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:01:31.467 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:01:31.467 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T08:01:31.467 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:01:31.468 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:01:31.468 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T08:01:31.468 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T08:01:31.468 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:01:31.469 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:01:31.469 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663056091.338062_357.1", 2022-09-13T08:01:31.469 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T08:01:31.338062Z" 2022-09-13T08:01:31.470 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:01:31.470 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:01:31.470 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:01:31.470 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T08:01:31.471 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:01:31.471 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T08:01:31.472 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:01:31.473 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T08:01:31.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:01:31.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:01:31.474 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:01:31.563 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:01:31.564 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:01:31.639 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:01:31.636+0000 7f2591f61600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:01:31.640 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:01:31.637+0000 7f2591f61600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:01:31.666 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:01:31.667 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:01:31.667 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:01:31.668 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:01:31.668 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:01:31.668 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:01:31.669 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:31.669 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:31.670 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:01:31.670 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:31.671 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:01:31.672 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:31.672 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:31.673 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:01:31.673 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:31.674 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:01:31.674 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:31.674 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:01:31.675 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:31.675 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:31.675 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:31.675 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:31.676 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:01:31.676 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:31.676 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:31.676 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:01:31.677 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:31.677 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:31.677 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:31.677 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:01:31.678 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:31.678 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:31.678 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:31.678 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:31.679 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:01:31.679 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:31.679 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:31.679 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.220845_233.1", 2022-09-13T08:01:31.680 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:31.680 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:31.680 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:31.680 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T08:01:31.681 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:31.681 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:31.681 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:31.681 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:31.682 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:01:31.682 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:31.682 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:31.682 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:01:31.683 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:31.683 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:31.683 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:31.684 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:01:31.684 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:31.684 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:31.684 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:31.685 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:01:31.685 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:31.685 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:01:31.685 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:01:31.686 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:01:31.686 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:31.686 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:01:31Z" 2022-09-13T08:01:31.686 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:01:31.688 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.220845_233.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.220845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:01:31Z" } 2022-09-13T08:01:31.688 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:01:31.689 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:01:31.690 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:01:31.690 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T08:01:36.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:01:36.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:01:36.694 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:01:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:01:36.790 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:01:36.865 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:01:36.861+0000 7efff9ef5600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:01:36.866 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:01:36.862+0000 7efff9ef5600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:01:36.886 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:01:36.887 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:01:36.887 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:01:36.887 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:01:36.888 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:01:36.888 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:01:36.888 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:36.889 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:36.889 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:01:36.889 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:36.890 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:01:36.890 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:36.890 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:36.891 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:01:36.891 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:36.891 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:01:36.892 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:36.892 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:01:36.892 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:36.893 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:36.893 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:36.893 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:36.894 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:01:36.894 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:36.894 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:36.895 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:01:36.895 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:36.895 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:36.896 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:36.896 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:01:36.896 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:36.897 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:36.897 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:36.897 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:36.898 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:01:36.898 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:36.898 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:36.898 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.105395_221.1", 2022-09-13T08:01:36.899 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:36.899 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:36.900 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:36.900 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T08:01:36.900 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:36.900 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:36.901 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:36.901 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:36.901 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:01:36.902 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:36.902 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:36.902 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:01:36.902 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:36.903 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:36.903 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:36.903 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:01:36.904 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:36.904 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:36.904 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:36.904 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:01:36.905 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:36.905 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:01:36.905 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:01:36.905 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:01:36.906 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:36.906 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:01:36Z" 2022-09-13T08:01:36.906 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:01:36.908 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.105395_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.105395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:01:36Z" } 2022-09-13T08:01:36.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.105395_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.105395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:01:36.909 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:01:36.909 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.105395_221.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:01:36.910 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.105395_221.1 2022-09-13T08:01:41.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:01:41.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:01:41.914 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:01:41.964 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:01:41.965 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:01:42.007 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:01:42.001+0000 7fe74b018600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:01:42.008 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:01:42.002+0000 7fe74b018600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:01:42.026 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:01:42.027 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:01:42.028 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:01:42.028 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:01:42.029 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:01:42.029 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:01:42.030 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:42.030 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:42.031 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:01:42.031 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:42.032 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:01:42.032 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:42.033 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:42.033 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:01:42.034 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:42.034 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:01:42.035 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:42.035 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:01:42.036 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:42.036 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:42.036 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:42.037 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:42.037 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:01:42.038 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:42.038 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:42.040 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:01:42.041 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:42.041 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:42.041 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:42.042 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:01:42.042 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:42.042 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:42.042 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:42.043 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:42.043 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:01:42.043 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:42.044 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:42.044 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.340682_237.1", 2022-09-13T08:01:42.044 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:42.044 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:42.045 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:42.045 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T08:01:42.045 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:42.046 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:42.046 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:42.047 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:42.047 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:01:42.047 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:42.048 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:42.048 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:01:42.048 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:42.049 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:42.049 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:42.049 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:01:42.049 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:42.050 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:42.050 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:42.050 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:01:42.051 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:42.051 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:01:42.051 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:01:42.051 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:01:42.052 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:42.052 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:01:42Z" 2022-09-13T08:01:42.052 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:01:42.054 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.340682_237.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.340682Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:01:42Z" } 2022-09-13T08:01:42.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.340682_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.340682Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:01:42.056 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:01:42.057 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.340682_237.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:01:42.057 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.340682_237.1 2022-09-13T08:01:47.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:01:47.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:01:47.060 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:01:47.111 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:01:47.112 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:01:47.142 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:01:47.138+0000 7ff907561600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:01:47.146 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:01:47.142+0000 7ff907561600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:01:47.224 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:01:47.225 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:01:47.226 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:01:47.226 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:01:47.226 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:01:47.227 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:01:47.227 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:47.227 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:47.248 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:01:47.248 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:47.248 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:01:47.249 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:47.249 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:47.249 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:01:47.249 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:47.250 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:01:47.250 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:47.250 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:01:47.251 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:47.251 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:47.251 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:47.251 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:47.252 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:01:47.252 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:47.252 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:47.252 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:01:47.253 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:47.253 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:47.254 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:47.254 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:01:47.254 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:47.255 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:47.255 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:47.255 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:47.255 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:01:47.256 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:47.256 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:47.256 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T08:01:47.257 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:47.257 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:47.258 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:47.258 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T08:01:47.258 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:47.259 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:47.259 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:47.260 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:47.260 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:01:47.260 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:47.261 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:47.261 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:01:47.262 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:47.262 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:47.263 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:47.263 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:01:47.264 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:47.264 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:47.264 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:47.264 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:01:47.265 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:47.265 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:01:47.265 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:01:47.265 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:01:47.266 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:47.266 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:01:47Z" 2022-09-13T08:01:47.266 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:01:47.268 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:01:47Z" } 2022-09-13T08:01:47.269 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:01:47.270 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:01:47.270 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:01:47.272 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T08:01:52.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:01:52.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:01:52.276 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:01:52.365 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:01:52.366 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:01:52.470 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:01:52.465+0000 7f5b1f3d5600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:01:52.471 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:01:52.466+0000 7f5b1f3d5600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:01:52.497 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:01:52.498 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:01:52.499 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:01:52.499 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:01:52.500 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:01:52.500 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:01:52.501 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:52.502 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:52.502 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:01:52.503 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:52.503 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:01:52.504 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:52.504 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:52.505 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:01:52.505 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:52.506 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:01:52.506 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:52.506 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:01:52.507 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:52.507 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:52.507 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:52.508 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:52.508 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:01:52.508 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:52.508 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:52.509 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:01:52.509 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:52.509 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:52.510 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:52.510 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:01:52.510 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:52.511 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:52.511 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:52.511 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:52.511 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:01:52.512 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:52.512 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:52.512 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.146590_228.1", 2022-09-13T08:01:52.513 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:52.513 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:52.513 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:52.513 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.146590Z", 2022-09-13T08:01:52.514 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:52.514 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:52.514 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:52.515 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:52.515 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:01:52.515 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:52.515 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:52.516 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:01:52.516 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:52.516 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:52.517 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:52.517 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:01:52.517 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:52.518 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:52.518 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:52.518 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:01:52.519 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:52.519 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:01:52.519 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:01:52.519 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:01:52.520 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:52.520 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:01:52Z" 2022-09-13T08:01:52.520 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:01:52.521 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.146590_228.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.146590Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:01:52Z" } 2022-09-13T08:01:52.521 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.146590_228.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.146590Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:01:52.522 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:01:52.523 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.146590_228.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:01:52.523 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.146590_228.1 2022-09-13T08:01:57.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:01:57.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:01:57.527 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:01:57.577 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:01:57.578 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:01:57.615 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:01:57.611+0000 7fe899853600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:01:57.616 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:01:57.612+0000 7fe899853600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:01:57.630 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:01:57.631 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:01:57.631 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:01:57.631 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:01:57.631 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:01:57.632 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:01:57.632 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:57.632 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:57.633 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:01:57.633 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:57.633 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:01:57.633 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:57.634 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:57.634 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:01:57.634 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:57.634 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:01:57.635 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:57.635 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:01:57.635 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:57.636 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:57.636 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:57.636 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:57.637 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:01:57.637 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:57.637 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:57.637 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:01:57.638 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:57.638 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:57.638 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:57.639 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:01:57.639 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:57.639 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:57.639 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:57.640 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:57.640 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:01:57.640 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:57.640 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:57.641 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T08:01:57.641 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:57.641 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:57.641 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:57.642 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T08:01:57.642 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:57.642 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:57.642 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:57.643 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:01:57.643 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:01:57.643 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:01:57.644 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:01:57.644 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:01:57.644 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:01:57.644 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:01:57.645 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:01:57.645 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:01:57.645 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:01:57.645 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:57.646 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:01:57.646 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:01:57.646 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:57.646 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:01:57.647 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:01:57.647 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:01:57.647 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:01:57.647 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:01:57Z" 2022-09-13T08:01:57.648 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:01:57.649 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:01:57Z" } 2022-09-13T08:01:57.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:01:57.650 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:01:57.651 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:01:57.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T08:02:02.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:02.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:02.654 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:02:02.700 INFO:tasks.rgw.c1.client.1.smithi036.stdout:2022-09-13T08:02:02.696+0000 7f8f5e7a1700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-09-13T08:02:02.706 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:02:02.707 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:02:02.737 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:02.733+0000 7fb2b9ba8600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:02.738 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:02.734+0000 7fb2b9ba8600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:02.752 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:02:02.752 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:02:02.753 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:02:02.753 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:02:02.754 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:02:02.754 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:02:02.755 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:02.755 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:02.756 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:02:02.757 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:02.757 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:02:02.758 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:02.758 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:02.759 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:02:02.759 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:02.760 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:02:02.760 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:02.761 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:02:02.761 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:02.762 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:02.762 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:02.763 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:02.763 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:02:02.764 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:02.764 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:02.765 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:02:02.766 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:02.766 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:02.767 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:02.767 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:02:02.768 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:02.768 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:02.769 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:02.769 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:02.770 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:02:02.770 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:02.771 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:02.771 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.107994_222.1", 2022-09-13T08:02:02.772 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:02.772 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:02.773 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:02.773 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.107994Z", 2022-09-13T08:02:02.774 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:02.774 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:02.775 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:02.775 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:02.776 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:02:02.777 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:02.777 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:02.777 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:02:02.778 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:02.779 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:02.779 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:02.780 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:02:02.780 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:02.781 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:02.781 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:02.782 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:02:02.783 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:02.783 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:02:02.784 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:02:02.784 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:02:02.785 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:02.785 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:02:02Z" 2022-09-13T08:02:02.786 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:02:02.788 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.107994_222.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.107994Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:02:02Z" } 2022-09-13T08:02:02.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.107994_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.107994Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:02:02.790 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:02.791 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.107994_222.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:02.792 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.107994_222.1 2022-09-13T08:02:07.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:07.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:07.795 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:02:07.853 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:02:07.853 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:02:07.954 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:07.950+0000 7f1a24c9a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:07.956 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:07.953+0000 7f1a24c9a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:08.032 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:02:08.032 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:02:08.033 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:02:08.033 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:02:08.033 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:02:08.034 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:02:08.034 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:08.034 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:08.034 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:02:08.035 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:08.035 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:02:08.035 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:08.036 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:08.036 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:02:08.036 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:08.036 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:02:08.037 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:08.037 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:02:08.037 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:08.037 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:08.038 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:08.038 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:08.038 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:02:08.038 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:08.039 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:08.039 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:02:08.039 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:08.039 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:08.040 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:08.040 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:02:08.040 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:08.041 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:08.041 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:08.041 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:08.041 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:02:08.042 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:08.042 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:08.042 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T08:02:08.042 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:08.043 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:08.043 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:08.043 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T08:02:08.043 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:08.044 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:08.044 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:08.044 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:08.044 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:02:08.045 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:08.045 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:08.045 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:02:08.045 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:08.046 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:08.046 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:08.046 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:02:08.047 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:08.047 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:08.047 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:08.047 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:02:08.048 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:08.048 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:02:08.048 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:02:08.048 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:02:08.049 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:08.049 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:02:07Z" 2022-09-13T08:02:08.049 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:02:08.050 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:02:07Z" } 2022-09-13T08:02:08.051 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:02:08.052 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:08.052 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:08.053 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T08:02:13.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:13.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:13.056 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:02:13.114 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:02:13.115 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:02:13.155 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:13.152+0000 7fceec64e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:13.156 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:13.153+0000 7fceec64e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:13.176 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:02:13.177 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:02:13.177 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:02:13.178 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:02:13.178 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:02:13.179 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:02:13.179 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:13.180 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:13.181 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:02:13.182 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:13.182 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:02:13.182 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:13.183 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:13.183 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:02:13.184 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:13.184 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:02:13.185 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:13.185 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:02:13.185 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:13.186 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:13.186 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:13.187 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:13.187 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:02:13.187 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:13.187 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:13.188 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:02:13.188 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:13.188 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:13.189 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:13.189 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:02:13.191 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:13.191 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:13.192 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:13.192 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:13.193 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:02:13.194 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:13.194 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:13.195 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.340682_237.1", 2022-09-13T08:02:13.195 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:13.196 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:13.196 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:13.197 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T08:02:13.197 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:13.197 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:13.198 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:13.198 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:13.199 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:02:13.199 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:13.200 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:13.200 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:02:13.201 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:13.201 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:13.202 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:13.202 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:02:13.202 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:13.203 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:13.204 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:13.204 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:02:13.204 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:13.205 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:02:13.205 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:02:13.206 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:02:13.206 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:13.207 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:02:13Z" 2022-09-13T08:02:13.207 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:02:13.209 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.340682_237.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.340682Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:02:13Z" } 2022-09-13T08:02:13.210 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.340682_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.340682Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:02:13.211 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:13.212 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.340682_237.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:13.212 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.340682_237.1 2022-09-13T08:02:18.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:18.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:18.217 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:02:18.274 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:02:18.274 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:02:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:18.312+0000 7f4c454e6600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:18.313+0000 7f4c454e6600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:18.335 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:02:18.335 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:02:18.336 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:02:18.336 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:02:18.336 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:02:18.336 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:02:18.337 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:18.337 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:18.338 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:02:18.342 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:18.342 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:02:18.343 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:18.343 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:18.344 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:02:18.344 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:18.345 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:02:18.345 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:18.347 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:02:18.347 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:18.347 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:18.349 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:18.349 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:18.349 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:02:18.349 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:18.350 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:18.350 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:02:18.351 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:18.351 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:18.352 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:18.353 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:02:18.353 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:18.353 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:18.354 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:18.354 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:18.354 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:02:18.355 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:18.355 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:18.355 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.990623_217.1", 2022-09-13T08:02:18.356 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:18.356 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:18.357 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:18.357 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T08:02:18.357 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:18.358 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:18.359 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:18.359 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:18.360 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:02:18.360 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:18.360 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:18.361 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:02:18.362 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:18.362 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:18.362 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:18.363 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:02:18.363 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:18.364 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:18.364 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:18.364 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:02:18.365 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:18.365 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:02:18.366 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:02:18.366 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:02:18.367 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:18.367 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:02:18Z" 2022-09-13T08:02:18.368 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:02:18.369 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.990623_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.990623Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:02:18Z" } 2022-09-13T08:02:18.370 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:02:18.371 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:18.371 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:18.372 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.990623_217.1 2022-09-13T08:02:23.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:23.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:23.376 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:02:23.440 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:02:23.441 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:02:23.491 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:23.487+0000 7f5e914df600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:23.492 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:23.488+0000 7f5e914df600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:23.513 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:02:23.514 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:02:23.514 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:02:23.515 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:02:23.515 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:02:23.515 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:02:23.516 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:23.516 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:23.517 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:02:23.517 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:23.517 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:02:23.518 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:23.518 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:23.518 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:02:23.519 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:23.520 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:02:23.521 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:23.521 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:02:23.522 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:23.522 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:23.522 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:23.523 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:23.523 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:02:23.523 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:23.523 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:23.524 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:02:23.524 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:23.524 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:23.525 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:23.525 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:02:23.525 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:23.526 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:23.526 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:23.527 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:23.527 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:02:23.527 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:23.528 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:23.528 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.182451_229.1", 2022-09-13T08:02:23.529 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:23.529 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:23.530 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:23.530 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T08:02:23.531 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:23.532 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:23.532 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:23.533 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:23.533 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:02:23.534 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:23.534 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:23.535 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:02:23.535 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:23.535 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:23.536 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:23.536 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:02:23.537 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:23.538 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:23.538 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:23.539 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:02:23.539 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:23.540 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:02:23.540 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:02:23.541 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:02:23.541 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:23.541 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:02:23Z" 2022-09-13T08:02:23.542 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:02:23.543 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.182451_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.182451Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:02:23Z" } 2022-09-13T08:02:23.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.182451_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.182451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:02:23.545 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:23.546 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.182451_229.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:23.547 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.182451_229.1 2022-09-13T08:02:28.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:28.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:28.550 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:02:28.602 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:02:28.603 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:02:28.636 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:28.632+0000 7fe6174b7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:28.637 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:28.633+0000 7fe6174b7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:28.651 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:02:28.652 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:02:28.653 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:02:28.653 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:02:28.653 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:02:28.654 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:02:28.654 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:28.655 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:28.655 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:02:28.655 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:28.656 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:02:28.656 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:28.656 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:28.657 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:02:28.657 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:28.660 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:02:28.660 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:28.661 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:02:28.662 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:28.662 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:28.663 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:28.663 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:28.664 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:02:28.664 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:28.665 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:28.665 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:02:28.666 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:28.666 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:28.667 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:28.667 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:02:28.668 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:28.668 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:28.668 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:28.669 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:28.669 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:02:28.669 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:28.670 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:28.670 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.187352_231.1", 2022-09-13T08:02:28.670 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:28.671 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:28.671 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:28.671 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T08:02:28.672 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:28.672 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:28.672 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:28.672 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:28.672 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:02:28.673 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:28.673 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:28.673 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:02:28.674 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:28.674 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:28.675 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:28.675 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:02:28.675 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:28.676 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:28.676 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:28.676 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:02:28.676 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:28.677 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:02:28.677 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:02:28.677 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:02:28.677 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:28.678 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:02:28Z" 2022-09-13T08:02:28.678 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:02:28.680 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.187352_231.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.187352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:02:28Z" } 2022-09-13T08:02:28.680 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.187352_231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.187352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:02:28.681 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:28.681 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.187352_231.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:28.682 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.187352_231.1 2022-09-13T08:02:33.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:33.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:33.685 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:02:33.736 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:02:33.736 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:02:33.765 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:33.761+0000 7fcf35a6c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:33.766 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:33.762+0000 7fcf35a6c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:33.780 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:02:33.780 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:02:33.780 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:02:33.781 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:02:33.782 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:02:33.782 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:02:33.782 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:33.783 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:33.783 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:02:33.783 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:33.783 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:02:33.784 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:33.784 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:33.784 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:02:33.784 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:33.785 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:02:33.785 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:33.785 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:02:33.785 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:33.786 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:33.786 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:33.786 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:33.786 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:02:33.787 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:33.787 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:33.787 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:02:33.787 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:33.788 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:33.788 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:33.788 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:02:33.788 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:33.789 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:33.789 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:33.790 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:33.790 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:02:33.790 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:33.791 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:33.791 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T08:02:33.791 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:33.792 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:33.792 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:33.792 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T08:02:33.793 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:33.793 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:33.793 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:33.793 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:33.794 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:02:33.794 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:33.795 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:33.795 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:02:33.795 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:33.796 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:33.796 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:33.797 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:02:33.797 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:33.797 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:33.800 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:33.800 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:02:33.801 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:33.802 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:02:33.802 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:02:33.803 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:02:33.803 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:33.804 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:02:33Z" 2022-09-13T08:02:33.804 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:02:33.806 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:02:33Z" } 2022-09-13T08:02:33.808 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:02:33.810 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:33.810 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:33.811 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T08:02:38.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:38.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:38.814 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:02:38.871 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:02:38.871 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:02:38.939 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:38.935+0000 7fc53bfa7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:38.942 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:38.936+0000 7fc53bfa7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:38.958 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:02:38.958 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:02:38.959 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:02:38.960 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:02:38.960 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:02:38.961 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:02:38.961 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:38.961 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:38.962 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:02:38.963 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:38.963 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:02:38.963 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:38.964 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:38.964 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:02:38.965 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:38.965 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:02:38.966 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:38.967 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:02:38.967 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:38.968 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:38.968 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:38.968 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:38.969 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:02:38.969 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:38.969 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:38.969 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:02:38.970 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:38.970 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:38.971 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:38.971 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:02:38.971 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:38.971 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:38.972 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:38.972 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:38.973 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:02:38.973 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:38.974 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:38.975 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.953175_216.1", 2022-09-13T08:02:38.975 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:38.976 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:38.976 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:38.977 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T08:02:38.977 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:38.978 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:38.978 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:38.979 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:38.980 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:02:38.980 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:38.980 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:38.981 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:02:38.981 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:38.982 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:38.982 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:38.983 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:02:38.983 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:38.983 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:38.983 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:38.984 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:02:38.984 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:38.984 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:02:38.984 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:02:38.985 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:02:38.985 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:38.986 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:02:38Z" 2022-09-13T08:02:38.986 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:02:38.988 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.953175_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.953175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:02:38Z" } 2022-09-13T08:02:38.989 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:02:38.989 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:38.990 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:38.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.953175_216.1 2022-09-13T08:02:43.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:43.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:43.995 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:02:44.046 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:02:44.046 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:02:44.084 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:44.080+0000 7f0e772b3600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:44.085 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:44.081+0000 7f0e772b3600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:44.099 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:02:44.100 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:02:44.100 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:02:44.101 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:02:44.101 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:02:44.102 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:02:44.102 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:44.102 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:44.103 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:02:44.103 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:44.104 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:02:44.104 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:44.104 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:44.105 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:02:44.105 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:44.105 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:02:44.106 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:44.106 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:02:44.106 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:44.107 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:44.107 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:44.107 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:44.108 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:02:44.108 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:44.108 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:44.108 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:02:44.109 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:44.109 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:44.109 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:44.109 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:02:44.110 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:44.112 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:44.112 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:44.113 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:44.113 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:02:44.114 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:44.115 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:44.115 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.109037_223.1", 2022-09-13T08:02:44.116 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:44.116 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:44.117 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:44.117 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.109037Z", 2022-09-13T08:02:44.118 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:44.118 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:44.119 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:44.119 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:44.120 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:02:44.120 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:44.121 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:44.121 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:02:44.124 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:44.125 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:44.125 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:44.126 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:02:44.127 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:44.127 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:44.128 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:44.128 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:02:44.129 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:44.129 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:02:44.131 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:02:44.131 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:02:44.132 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:44.132 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:02:44Z" 2022-09-13T08:02:44.133 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:02:44.135 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.109037_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.109037Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:02:44Z" } 2022-09-13T08:02:44.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.109037_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.109037Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:02:44.137 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:44.138 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.109037_223.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:44.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.109037_223.1 2022-09-13T08:02:49.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:49.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:49.143 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:02:49.200 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:02:49.201 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:02:49.236 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:49.233+0000 7f178b4dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:49.238 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:49.234+0000 7f178b4dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:49.252 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:02:49.253 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:02:49.253 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:02:49.254 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:02:49.254 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:02:49.254 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:02:49.254 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:49.255 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:49.255 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:02:49.255 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:49.256 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:02:49.256 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:49.256 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:49.256 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:02:49.257 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:49.257 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:02:49.257 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:49.258 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:02:49.258 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:49.258 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:49.259 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:49.259 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:49.259 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:02:49.259 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:49.260 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:49.260 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:02:49.260 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:49.260 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:49.261 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:49.261 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:02:49.261 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:49.262 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:49.262 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:49.262 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:49.263 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:02:49.263 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:49.263 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:49.264 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.142561_227.1", 2022-09-13T08:02:49.264 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:49.264 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:49.265 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:49.265 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T08:02:49.266 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:49.266 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:49.266 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:49.267 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:49.267 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:02:49.268 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:49.268 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:49.269 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:02:49.269 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:49.270 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:49.270 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:49.271 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:02:49.271 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:49.272 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:49.272 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:49.272 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:02:49.273 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:49.273 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:02:49.274 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:02:49.275 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:02:49.275 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:49.276 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:02:49Z" 2022-09-13T08:02:49.276 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:02:49.278 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.142561_227.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.142561Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:02:49Z" } 2022-09-13T08:02:49.280 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:02:49.280 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:49.281 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:49.281 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T08:02:54.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:54.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:54.287 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:02:54.341 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:02:54.342 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:02:54.372 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:54.369+0000 7f462a0ee600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:54.373 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:54.370+0000 7f462a0ee600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:54.388 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:02:54.388 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:02:54.389 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:02:54.389 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:02:54.390 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:02:54.391 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:02:54.391 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:54.392 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:54.393 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:02:54.393 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:54.394 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:02:54.395 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:54.395 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:54.396 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:02:54.396 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:54.397 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:02:54.397 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:54.398 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:02:54.398 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:54.399 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:54.399 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:54.399 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:54.400 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:02:54.400 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:54.401 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:54.402 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:02:54.402 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:54.403 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:54.403 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:54.403 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:02:54.404 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:54.404 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:54.404 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:54.405 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:54.405 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:02:54.406 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:54.406 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:54.407 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.070128_219.1", 2022-09-13T08:02:54.408 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:54.408 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:54.408 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:54.409 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T08:02:54.410 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:54.410 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:54.410 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:54.411 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:54.412 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:02:54.412 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:54.413 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:54.413 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:02:54.413 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:54.414 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:54.414 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:54.414 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:02:54.415 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:54.415 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:54.415 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:54.416 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:02:54.416 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:54.416 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:02:54.416 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:02:54.417 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:02:54.417 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:54.417 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:02:54Z" 2022-09-13T08:02:54.418 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:02:54.419 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.070128_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.070128Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:02:54Z" } 2022-09-13T08:02:54.420 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:02:54.421 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:54.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:54.422 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.070128_219.1 2022-09-13T08:02:59.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:59.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:02:59.425 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:02:59.477 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:02:59.478 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:02:59.520 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:59.517+0000 7fe4d8e2e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:59.522 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:02:59.518+0000 7fe4d8e2e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:02:59.535 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:02:59.536 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:02:59.537 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:02:59.537 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:02:59.538 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:02:59.538 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:02:59.538 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:59.538 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:59.539 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:02:59.539 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:59.539 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:02:59.540 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:59.540 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:59.541 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:02:59.541 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:59.542 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:02:59.543 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:59.543 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:02:59.544 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:59.544 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:59.545 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:59.545 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:59.546 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:02:59.547 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:59.547 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:59.548 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:02:59.548 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:59.548 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:59.549 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:59.549 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:02:59.549 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:59.549 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:59.550 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:59.550 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:59.550 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:02:59.550 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:59.551 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:59.551 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.434813_240.1", 2022-09-13T08:02:59.552 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:59.552 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:59.552 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:59.553 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.434813Z", 2022-09-13T08:02:59.553 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:59.553 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:59.554 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:59.554 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:02:59.555 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:02:59.555 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:02:59.556 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:02:59.556 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:02:59.556 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:02:59.557 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:02:59.557 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:02:59.558 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:02:59.559 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:02:59.559 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:59.559 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:02:59.560 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:02:59.561 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:59.561 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:02:59.562 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:02:59.562 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:02:59.563 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:02:59.563 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:02:59Z" 2022-09-13T08:02:59.564 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:02:59.566 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.434813_240.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.434813Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:02:59Z" } 2022-09-13T08:02:59.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.434813_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.434813Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:02:59.568 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:59.569 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.434813_240.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:02:59.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.434813_240.1 2022-09-13T08:03:04.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:04.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:04.572 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:03:04.624 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:03:04.625 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:03:04.656 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:04.652+0000 7f83f1510600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:04.657 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:04.653+0000 7f83f1510600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:04.672 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:03:04.672 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:03:04.672 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:03:04.673 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:03:04.673 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:03:04.673 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:03:04.674 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:04.674 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:04.674 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:03:04.674 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:04.675 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:03:04.675 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:04.675 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:04.675 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:03:04.676 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:04.676 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:03:04.676 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:04.676 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:03:04.677 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:04.677 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:04.677 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:04.677 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:04.678 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:03:04.678 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:04.678 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:04.678 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:03:04.679 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:04.679 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:04.679 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:04.679 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:03:04.680 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:04.680 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:04.681 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:04.681 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:04.681 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:03:04.681 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:04.682 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:04.682 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T08:03:04.682 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:04.682 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:04.683 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:04.683 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T08:03:04.683 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:04.683 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:04.684 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:04.684 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:04.684 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:03:04.684 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:04.685 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:04.685 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:03:04.685 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:04.685 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:04.686 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:04.686 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:03:04.686 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:04.686 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:04.687 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:04.687 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:03:04.687 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:04.687 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:03:04.688 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:03:04.688 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:03:04.688 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:04.688 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:03:04Z" 2022-09-13T08:03:04.689 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:03:04.690 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:03:04Z" } 2022-09-13T08:03:04.691 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:03:04.692 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:04.692 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:04.692 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T08:03:09.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:09.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:09.696 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:03:09.750 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:03:09.750 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:03:09.801 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:09.797+0000 7f32eedba600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:09.802 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:09.798+0000 7f32eedba600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:09.817 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:03:09.818 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:03:09.818 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:03:09.819 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:03:09.819 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:03:09.820 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:03:09.820 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:09.821 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:09.821 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:03:09.822 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:09.822 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:03:09.823 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:09.823 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:09.824 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:03:09.824 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:09.825 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:03:09.825 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:09.826 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:03:09.826 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:09.827 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:09.827 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:09.828 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:09.828 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:03:09.829 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:09.829 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:09.830 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:03:09.830 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:09.831 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:09.831 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:09.832 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:03:09.832 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:09.833 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:09.833 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:09.834 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:09.835 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:03:09.835 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:09.836 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:09.837 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.187352_231.1", 2022-09-13T08:03:09.837 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:09.838 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:09.838 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:09.839 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T08:03:09.839 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:09.840 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:09.840 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:09.841 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:09.841 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:03:09.842 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:09.842 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:09.843 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:03:09.843 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:09.844 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:09.844 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:09.845 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:03:09.845 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:09.846 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:09.846 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:09.847 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:03:09.847 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:09.848 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:03:09.848 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:03:09.849 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:03:09.849 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:09.850 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:03:09Z" 2022-09-13T08:03:09.851 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:03:09.853 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.187352_231.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.187352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:03:09Z" } 2022-09-13T08:03:09.854 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.187352_231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.187352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:03:09.855 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:09.856 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.187352_231.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:09.857 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.187352_231.1 2022-09-13T08:03:14.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:14.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:14.860 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:03:14.910 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:03:14.911 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:03:14.946 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:14.942+0000 7fcfef69b600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:14.947 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:14.943+0000 7fcfef69b600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:14.965 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:03:14.966 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:03:14.966 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:03:14.966 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:03:14.967 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:03:14.967 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:03:14.967 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:14.968 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:14.968 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:03:14.968 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:14.969 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:03:14.969 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:14.969 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:14.969 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:03:14.970 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:14.970 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:03:14.970 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:14.970 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:03:14.971 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:14.971 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:14.971 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:14.971 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:14.972 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:03:14.972 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:14.972 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:14.972 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:03:14.973 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:14.973 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:14.973 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:14.973 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:03:14.974 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:14.974 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:14.974 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:14.975 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:14.975 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:03:14.975 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:14.975 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:14.976 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.111513_224.1", 2022-09-13T08:03:14.976 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:14.976 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:14.976 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:14.977 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.111513Z", 2022-09-13T08:03:14.977 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:14.977 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:14.977 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:14.978 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:14.978 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:03:14.978 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:14.978 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:14.979 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:03:14.979 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:14.979 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:14.979 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:14.980 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:03:14.980 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:14.980 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:14.980 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:14.981 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:03:14.981 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:14.981 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:03:14.981 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:03:14.982 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:03:14.982 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:14.982 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:03:14Z" 2022-09-13T08:03:14.983 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:03:14.984 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.111513_224.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.111513Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:03:14Z" } 2022-09-13T08:03:14.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.111513_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.111513Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:03:14.985 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:14.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.111513_224.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:14.986 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.111513_224.1 2022-09-13T08:03:19.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:19.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:19.989 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:03:20.039 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:03:20.039 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:03:20.070 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:20.065+0000 7ff30888f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:20.071 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:20.066+0000 7ff30888f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:20.085 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:03:20.086 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:03:20.086 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:03:20.087 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:03:20.088 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:03:20.088 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:03:20.089 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:20.089 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:20.090 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:03:20.090 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:20.090 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:03:20.091 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:20.091 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:20.092 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:03:20.093 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:20.093 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:03:20.093 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:20.094 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:03:20.094 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:20.094 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:20.094 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:20.095 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:20.095 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:03:20.096 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:20.096 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:20.096 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:03:20.097 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:20.097 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:20.097 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:20.097 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:03:20.098 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:20.098 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:20.098 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:20.098 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:20.099 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:03:20.099 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:20.099 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:20.100 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.187352_231.1", 2022-09-13T08:03:20.100 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:20.100 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:20.100 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:20.101 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T08:03:20.101 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:20.101 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:20.101 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:20.102 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:20.102 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:03:20.102 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:20.103 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:20.103 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:03:20.103 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:20.104 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:20.104 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:20.105 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:03:20.105 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:20.105 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:20.106 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:20.106 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:03:20.106 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:20.106 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:03:20.107 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:03:20.107 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:03:20.107 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:20.107 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:03:20Z" 2022-09-13T08:03:20.108 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:03:20.109 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.187352_231.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.187352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:03:20Z" } 2022-09-13T08:03:20.110 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.187352_231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.187352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:03:20.111 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:20.111 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.187352_231.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:20.112 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.187352_231.1 2022-09-13T08:03:25.113 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:25.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:25.115 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:03:25.165 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:03:25.166 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:03:25.204 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:25.200+0000 7ff2cb06a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:25.205 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:25.201+0000 7ff2cb06a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:25.219 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:03:25.220 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:03:25.220 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:03:25.220 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:03:25.221 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:03:25.221 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:03:25.221 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:25.221 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:25.222 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:03:25.222 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:25.222 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:03:25.222 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:25.223 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:25.223 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:03:25.223 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:25.224 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:03:25.224 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:25.224 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:03:25.224 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:25.225 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:25.225 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:25.225 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:25.225 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:03:25.226 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:25.226 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:25.226 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:03:25.227 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:25.227 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:25.228 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:25.228 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:03:25.229 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:25.229 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:25.230 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:25.230 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:25.231 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:03:25.231 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:25.232 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:25.232 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T08:03:25.233 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:25.233 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:25.233 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:25.234 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T08:03:25.234 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:25.234 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:25.234 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:25.235 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:25.235 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:03:25.235 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:25.236 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:25.236 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:03:25.236 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:25.236 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:03:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:25.237 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:25.238 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:25.238 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:03:25.238 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:25.239 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:03:25.239 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:03:25.239 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:03:25.239 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:25.240 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:03:25Z" 2022-09-13T08:03:25.240 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:03:25.241 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:03:25Z" } 2022-09-13T08:03:25.242 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:03:25.242 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:25.243 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:25.243 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T08:03:30.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:30.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:30.247 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:03:30.302 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:03:30.303 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:03:30.333 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:30.329+0000 7f8c431b7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:30.334 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:30.330+0000 7f8c431b7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:30.349 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:03:30.350 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:03:30.350 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:03:30.351 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:03:30.351 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:03:30.351 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:03:30.351 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:30.352 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:30.352 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:03:30.352 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:30.352 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:03:30.353 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:30.353 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:30.353 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:03:30.354 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:30.354 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:03:30.354 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:30.355 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:03:30.355 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:30.355 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:30.355 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:30.356 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:30.356 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:03:30.356 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:30.357 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:30.357 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:03:30.357 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:30.358 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:30.358 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:30.358 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:03:30.358 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:30.359 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:30.359 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:30.359 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:30.359 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:03:30.360 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:30.360 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:30.360 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.070128_219.1", 2022-09-13T08:03:30.361 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:30.361 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:30.361 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:30.362 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T08:03:30.362 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:30.362 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:30.362 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:30.363 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:30.363 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:03:30.363 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:30.364 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:30.364 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:03:30.364 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:30.365 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:30.365 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:30.365 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:03:30.365 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:30.366 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:30.366 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:30.366 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:03:30.366 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:30.367 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:03:30.367 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:03:30.367 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:03:30.367 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:30.368 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:03:30Z" 2022-09-13T08:03:30.368 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:03:30.369 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.070128_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.070128Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:03:30Z" } 2022-09-13T08:03:30.370 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:03:30.370 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:30.371 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:30.371 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.070128_219.1 2022-09-13T08:03:35.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:35.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:35.374 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:03:35.431 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:03:35.431 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:03:35.464 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:35.461+0000 7f6dbb012600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:35.465 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:35.462+0000 7f6dbb012600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:35.480 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:03:35.480 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:03:35.481 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:03:35.481 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:03:35.481 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:03:35.482 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:03:35.482 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:35.482 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:35.483 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:03:35.483 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:35.483 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:03:35.483 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:35.484 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:35.484 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:03:35.484 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:35.484 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:03:35.485 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:35.485 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:03:35.485 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:35.485 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:35.486 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:35.486 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:35.486 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:03:35.486 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:35.487 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:35.487 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:03:35.487 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:35.487 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:35.488 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:35.488 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:03:35.488 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:35.488 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:35.488 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:35.489 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:35.489 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:03:35.489 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:35.490 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:35.490 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.911619_214.1", 2022-09-13T08:03:35.490 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:35.490 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:35.491 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:35.491 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T08:03:35.491 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:35.491 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:35.492 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:35.492 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:35.492 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:03:35.493 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:35.493 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:35.493 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:03:35.493 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:35.494 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:35.495 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:35.495 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:03:35.495 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:35.496 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:35.496 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:35.496 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:03:35.496 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:35.497 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:03:35.497 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:03:35.497 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:03:35.498 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:35.498 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:03:35Z" 2022-09-13T08:03:35.498 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:03:35.500 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.911619_214.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.911619Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:03:35Z" } 2022-09-13T08:03:35.500 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:03:35.501 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:35.501 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:35.502 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.911619_214.1 2022-09-13T08:03:40.505 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:40.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:40.507 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:03:40.557 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:03:40.557 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:03:40.589 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:40.586+0000 7ff9c5576600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:40.590 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:40.587+0000 7ff9c5576600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:40.609 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:03:40.610 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:03:40.611 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:03:40.611 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:03:40.611 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:03:40.611 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:03:40.612 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:40.612 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:40.612 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:03:40.613 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:40.613 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:03:40.613 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:40.614 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:40.614 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:03:40.614 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:40.614 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:03:40.615 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:40.615 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:03:40.616 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:40.616 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:40.616 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:40.617 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:40.617 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:03:40.618 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:40.618 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:40.618 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:03:40.618 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:40.619 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:40.619 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:40.619 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:03:40.620 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:40.620 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:40.620 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:40.621 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:40.621 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:03:40.621 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:40.622 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:40.622 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.134902_225.1", 2022-09-13T08:03:40.623 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:40.623 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:40.624 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:40.624 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.134902Z", 2022-09-13T08:03:40.624 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:40.625 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:40.625 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:40.626 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:40.626 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:03:40.626 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:40.627 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:40.627 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:03:40.628 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:40.628 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:40.629 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:40.629 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:03:40.629 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:40.630 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:40.630 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:40.630 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:03:40.630 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:40.631 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:03:40.631 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:03:40.632 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:03:40.632 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:40.632 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:03:40Z" 2022-09-13T08:03:40.633 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:03:40.635 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.134902_225.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.134902Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:03:40Z" } 2022-09-13T08:03:40.636 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.134902_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.134902Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:03:40.637 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:40.637 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.134902_225.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:40.638 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.134902_225.1 2022-09-13T08:03:45.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:45.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:45.641 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:03:45.691 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:03:45.692 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:03:45.740 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:45.736+0000 7f16223cc600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:45.741 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:45.737+0000 7f16223cc600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:45.755 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:03:45.755 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:03:45.755 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:03:45.756 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:03:45.756 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:03:45.756 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:03:45.756 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:45.757 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:45.757 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:03:45.757 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:45.757 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:03:45.758 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:45.758 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:45.758 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:03:45.758 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:45.759 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:03:45.759 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:45.759 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:03:45.759 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:45.760 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:45.760 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:45.760 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:45.760 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:03:45.761 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:45.761 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:45.761 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:03:45.761 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:45.761 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:45.762 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:45.762 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:03:45.762 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:45.763 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:45.763 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:45.763 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:45.763 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:03:45.764 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:45.764 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:45.764 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.186446_230.1", 2022-09-13T08:03:45.764 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:45.765 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:45.765 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:45.766 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.186446Z", 2022-09-13T08:03:45.766 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:45.767 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:45.767 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:45.767 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:45.768 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:03:45.768 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:45.768 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:45.768 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:03:45.769 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:45.769 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:45.769 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:45.769 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:03:45.770 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:45.770 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:45.770 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:45.770 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:03:45.771 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:45.771 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:03:45.771 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:03:45.771 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:03:45.772 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:45.772 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:03:45Z" 2022-09-13T08:03:45.772 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:03:45.773 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.186446_230.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.186446Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:03:45Z" } 2022-09-13T08:03:45.774 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.186446_230.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.186446Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:03:45.775 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:45.775 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.186446_230.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:45.776 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.186446_230.1 2022-09-13T08:03:50.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:50.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:50.778 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:03:50.830 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:03:50.831 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:03:50.876 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:50.871+0000 7fc13963a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:50.877 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:50.873+0000 7fc13963a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:50.899 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:03:50.899 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:03:50.899 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:03:50.900 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:03:50.900 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:03:50.900 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:03:50.900 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:50.901 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:50.901 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:03:50.901 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:50.901 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:03:50.902 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:50.902 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:50.902 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:03:50.902 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:50.903 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:03:50.903 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:50.904 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:03:50.904 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:50.905 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:50.905 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:50.905 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:50.905 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:03:50.906 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:50.906 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:50.907 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:03:50.907 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:50.908 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:50.908 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:50.908 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:03:50.909 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:50.909 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:50.910 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:50.910 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:50.910 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:03:50.911 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:50.911 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:50.912 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.072712_220.1", 2022-09-13T08:03:50.912 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:50.913 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:50.913 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:50.914 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.072712Z", 2022-09-13T08:03:50.914 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:50.915 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:50.915 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:50.916 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:50.916 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:03:50.916 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:50.917 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:50.917 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:03:50.917 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:50.918 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:50.918 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:50.918 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:03:50.919 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:50.919 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:50.919 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:50.920 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:03:50.920 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:50.920 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:03:50.920 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:03:50.921 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:03:50.921 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:50.922 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:03:50Z" 2022-09-13T08:03:50.922 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:03:50.924 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.072712_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.072712Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:03:50Z" } 2022-09-13T08:03:50.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.072712_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.072712Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:03:50.925 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:50.925 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.072712_220.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:50.926 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.072712_220.1 2022-09-13T08:03:55.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:55.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:03:55.928 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:03:56.019 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:03:56.019 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:03:56.087 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:56.083+0000 7f87118ad600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:56.088 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:03:56.084+0000 7f87118ad600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:03:56.111 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:03:56.111 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:03:56.112 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:03:56.112 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:03:56.113 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:03:56.113 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:03:56.114 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:56.114 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:56.115 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:03:56.115 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:56.115 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:03:56.116 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:56.116 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:56.117 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:03:56.117 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:56.117 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:03:56.118 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:56.118 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:03:56.119 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:56.119 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:56.120 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:56.120 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:56.121 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:03:56.121 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:56.122 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:56.122 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:03:56.123 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:56.124 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:56.124 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:56.125 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:03:56.125 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:56.125 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:56.126 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:56.126 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:56.127 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:03:56.127 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:56.127 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:56.128 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.911619_214.1", 2022-09-13T08:03:56.128 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:56.129 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:56.129 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:56.130 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T08:03:56.130 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:56.131 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:56.131 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:56.132 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:03:56.132 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:03:56.133 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:03:56.133 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:03:56.133 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:03:56.134 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:03:56.134 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:03:56.134 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:03:56.134 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:03:56.135 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:03:56.135 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:56.135 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:03:56.136 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:03:56.136 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:56.137 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:03:56.137 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:03:56.137 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:03:56.138 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:03:56.138 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:03:56Z" 2022-09-13T08:03:56.139 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:03:56.141 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.911619_214.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.911619Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:03:56Z" } 2022-09-13T08:03:56.141 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:03:56.143 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:56.143 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:03:56.144 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.911619_214.1 2022-09-13T08:04:01.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:01.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:01.147 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:04:01.196 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:04:01.197 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:04:01.233 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:01.229+0000 7f6194770600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:01.234 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:01.230+0000 7f6194770600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:01.249 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:04:01.250 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:04:01.250 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:04:01.250 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:04:01.251 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:04:01.251 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:04:01.251 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:01.251 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:01.252 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:04:01.252 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:01.252 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:04:01.253 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:01.253 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:01.253 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:04:01.253 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:01.254 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:04:01.254 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:01.254 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:04:01.254 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:01.255 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:01.255 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:01.255 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:01.255 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:04:01.256 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:01.256 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:01.256 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:04:01.256 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:01.257 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:01.257 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:01.257 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:04:01.257 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:01.258 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:01.258 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:01.258 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:01.258 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:04:01.259 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:01.259 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:01.259 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.105395_221.1", 2022-09-13T08:04:01.260 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:01.260 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:01.260 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:01.261 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T08:04:01.261 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:01.261 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:01.262 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:01.262 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:01.262 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:04:01.262 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:01.263 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:01.263 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:04:01.263 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:01.263 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:01.263 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:01.264 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:04:01.264 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:01.265 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:01.265 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:01.266 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:04:01.267 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:01.267 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:04:01.267 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:04:01.268 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:04:01.268 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:01.269 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:04:01Z" 2022-09-13T08:04:01.269 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:04:01.271 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.105395_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.105395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:04:01Z" } 2022-09-13T08:04:01.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.105395_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.105395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:04:01.274 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:01.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.105395_221.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:01.275 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.105395_221.1 2022-09-13T08:04:06.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:06.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:06.278 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:04:06.330 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:04:06.331 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:04:06.368 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:06.364+0000 7f840dce5600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:06.369 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:06.365+0000 7f840dce5600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:06.383 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:04:06.384 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:04:06.384 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:04:06.384 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:04:06.385 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:04:06.385 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:04:06.385 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:06.385 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:06.386 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:04:06.386 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:06.386 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:04:06.386 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:06.387 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:06.387 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:04:06.387 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:06.388 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:04:06.388 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:06.388 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:04:06.388 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:06.389 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:06.389 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:06.389 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:06.390 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:04:06.390 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:06.390 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:06.390 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:04:06.391 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:06.391 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:06.391 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:06.391 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:04:06.392 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:06.392 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:06.392 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:06.393 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:06.393 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:04:06.393 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:06.393 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:06.394 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T08:04:06.394 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:06.394 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:06.395 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:06.395 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T08:04:06.396 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:06.396 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:06.396 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:06.397 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:06.397 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:04:06.397 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:06.398 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:06.398 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:04:06.398 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:06.398 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:06.399 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:06.399 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:04:06.399 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:06.399 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:06.400 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:06.400 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:04:06.400 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:06.400 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:04:06.401 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:04:06.401 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:04:06.401 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:06.401 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:04:06Z" 2022-09-13T08:04:06.402 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:04:06.403 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:04:06Z" } 2022-09-13T08:04:06.404 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:04:06.405 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:06.405 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:06.406 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T08:04:11.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:11.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:11.408 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:04:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:04:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:04:11.491 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:11.487+0000 7fb2b6e68600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:11.492 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:11.488+0000 7fb2b6e68600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:11.507 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:04:11.508 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:04:11.508 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:04:11.508 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:04:11.509 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:04:11.509 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:04:11.509 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:11.510 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:11.510 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:04:11.511 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:11.511 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:04:11.512 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:11.512 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:11.512 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:04:11.513 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:11.513 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:04:11.513 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:11.513 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:04:11.514 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:11.514 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:11.514 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:11.515 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:11.515 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:04:11.515 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:11.515 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:11.516 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:04:11.516 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:11.516 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:11.516 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:11.517 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:04:11.517 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:11.517 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:11.517 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:11.518 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:11.518 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:04:11.518 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:11.519 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:11.519 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T08:04:11.520 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:11.520 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:11.521 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:11.521 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T08:04:11.521 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:11.522 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:11.522 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:11.522 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:11.523 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:04:11.523 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:11.523 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:11.523 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:04:11.524 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:11.524 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:11.524 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:11.524 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:04:11.525 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:11.525 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:11.525 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:11.526 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:04:11.526 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:11.526 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:04:11.527 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:04:11.527 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:04:11.527 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:11.527 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:04:11Z" 2022-09-13T08:04:11.528 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:04:11.529 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:04:11Z" } 2022-09-13T08:04:11.529 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:04:11.530 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:11.531 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:11.532 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T08:04:16.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:16.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:16.536 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:04:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:04:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:04:16.641 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:16.637+0000 7f262bb5c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:16.641 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:16.638+0000 7f262bb5c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:16.657 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:04:16.658 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:04:16.658 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:04:16.660 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:04:16.661 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:04:16.662 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:04:16.664 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:16.664 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:16.665 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:04:16.665 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:16.665 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:04:16.666 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:16.666 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:16.666 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:04:16.666 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:16.667 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:04:16.667 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:16.667 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:04:16.668 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:16.668 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:16.668 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:16.669 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:16.669 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:04:16.669 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:16.669 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:16.670 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:04:16.670 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:16.670 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:16.670 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:16.671 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:04:16.671 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:16.671 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:16.671 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:16.672 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:16.672 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:04:16.672 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:16.673 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:16.673 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.134902_225.1", 2022-09-13T08:04:16.673 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:16.673 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:16.674 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:16.674 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.134902Z", 2022-09-13T08:04:16.674 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:16.674 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:16.675 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:16.675 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:16.675 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:04:16.676 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:16.676 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:16.676 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:04:16.676 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:16.677 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:16.677 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:16.677 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:04:16.677 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:16.678 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:16.678 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:16.678 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:04:16.678 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:16.679 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:04:16.679 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:04:16.679 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:04:16.679 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:16.680 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:04:16Z" 2022-09-13T08:04:16.680 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:04:16.681 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.134902_225.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.134902Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:04:16Z" } 2022-09-13T08:04:16.682 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.134902_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.134902Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:04:16.682 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:16.683 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.134902_225.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:16.683 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.134902_225.1 2022-09-13T08:04:21.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:21.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:21.687 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:04:21.767 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:04:21.768 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:04:21.820 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:21.817+0000 7f9b0fd0f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:21.821 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:21.817+0000 7f9b0fd0f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:21.837 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:04:21.838 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:04:21.838 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:04:21.838 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:04:21.839 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:04:21.839 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:04:21.840 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:21.840 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:21.840 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:04:21.840 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:21.841 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:04:21.841 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:21.841 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:21.841 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:04:21.842 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:21.842 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:04:21.842 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:21.843 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:04:21.844 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:21.844 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:21.845 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:21.845 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:21.845 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:04:21.845 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:21.846 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:21.847 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:04:21.847 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:21.847 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:21.848 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:21.848 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:04:21.849 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:21.849 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:21.849 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:21.850 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:21.850 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:04:21.850 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:21.851 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:21.851 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.911619_214.1", 2022-09-13T08:04:21.851 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:21.852 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:21.852 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:21.853 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T08:04:21.853 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:21.854 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:21.854 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:21.854 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:21.855 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:04:21.855 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:21.855 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:21.856 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:04:21.856 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:21.857 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:21.857 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:21.858 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:04:21.858 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:21.858 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:21.859 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:21.859 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:04:21.859 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:21.860 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:04:21.860 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:04:21.860 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:04:21.860 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:21.861 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:04:21Z" 2022-09-13T08:04:21.861 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:04:21.862 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.911619_214.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.911619Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:04:21Z" } 2022-09-13T08:04:21.863 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:04:21.864 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:21.864 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:21.865 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.911619_214.1 2022-09-13T08:04:26.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:26.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:26.868 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:04:26.918 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:04:26.919 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:04:26.964 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:26.961+0000 7f5684bf3600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:26.965 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:26.962+0000 7f5684bf3600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:26.982 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:04:26.983 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:04:26.983 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:04:26.984 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:04:26.984 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:04:26.984 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:04:26.985 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:26.985 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:26.986 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:04:26.987 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:26.987 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:04:26.988 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:26.988 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:26.989 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:04:26.989 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:26.990 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:04:26.990 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:26.990 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:04:26.991 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:26.991 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:26.992 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:26.992 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:26.993 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:04:26.993 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:26.993 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:26.994 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:04:26.994 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:26.994 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:26.995 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:26.995 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:04:26.995 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:26.995 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:26.996 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:26.996 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:26.996 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:04:26.997 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:26.997 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:26.997 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T08:04:26.998 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:26.998 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:26.998 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:26.998 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T08:04:26.999 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:26.999 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:26.999 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:27.000 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:27.000 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:04:27.000 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:27.001 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:27.001 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:04:27.001 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:27.002 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:27.002 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:27.002 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:04:27.003 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:27.003 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:27.003 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:27.004 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:04:27.004 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:27.005 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:04:27.005 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:04:27.006 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:04:27.006 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:27.007 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:04:26Z" 2022-09-13T08:04:27.007 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:04:27.009 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:04:26Z" } 2022-09-13T08:04:27.010 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:04:27.011 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:27.012 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:27.013 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T08:04:32.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:32.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:32.017 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:04:32.068 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:04:32.068 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:04:32.114 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:32.110+0000 7f4a538ad600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:32.115 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:32.111+0000 7f4a538ad600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:32.135 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:04:32.136 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:04:32.136 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:04:32.137 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:04:32.137 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:04:32.137 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:04:32.137 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:32.138 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:32.138 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:04:32.138 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:32.138 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:04:32.139 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:32.139 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:32.139 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:04:32.140 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:32.140 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:04:32.140 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:32.140 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:04:32.141 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:32.141 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:32.141 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:32.141 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:32.142 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:04:32.142 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:32.143 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:32.143 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:04:32.143 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:32.144 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:32.144 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:32.144 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:04:32.144 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:32.145 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:32.145 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:32.145 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:32.145 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:04:32.146 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:32.146 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:32.146 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.070128_219.1", 2022-09-13T08:04:32.146 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:32.147 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:32.147 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:32.147 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T08:04:32.147 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:32.148 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:32.148 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:32.148 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:32.148 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:04:32.149 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:32.149 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:32.149 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:04:32.149 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:32.149 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:32.150 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:32.150 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:04:32.150 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:32.151 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:32.151 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:32.151 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:04:32.151 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:32.151 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:04:32.152 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:04:32.152 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:04:32.153 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:32.153 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:04:32Z" 2022-09-13T08:04:32.153 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:04:32.155 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.070128_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.070128Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:04:32Z" } 2022-09-13T08:04:32.156 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:04:32.156 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:32.157 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:32.157 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.070128_219.1 2022-09-13T08:04:37.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:37.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:37.160 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:04:37.219 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:04:37.219 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:04:37.252 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:37.247+0000 7fe506ca2600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:37.253 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:37.249+0000 7fe506ca2600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:37.273 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:04:37.273 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:04:37.273 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:04:37.274 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:04:37.274 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:04:37.297 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:04:37.298 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:37.298 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:37.299 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:04:37.299 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:37.299 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:04:37.300 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:37.300 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:37.300 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:04:37.300 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:37.300 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:04:37.301 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:37.301 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:04:37.302 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:37.302 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:37.302 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:37.302 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:37.303 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:04:37.303 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:37.303 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:37.303 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:04:37.304 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:37.304 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:37.304 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:37.304 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:04:37.305 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:37.305 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:37.305 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:37.305 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:37.307 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:04:37.308 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:37.308 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:37.309 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T08:04:37.309 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:37.310 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:37.310 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:37.311 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T08:04:37.312 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:37.312 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:37.313 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:37.313 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:37.314 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:04:37.315 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:37.315 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:37.316 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:04:37.316 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:37.316 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:37.317 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:37.317 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:04:37.318 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:37.318 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:37.320 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:37.320 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:04:37.320 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:37.321 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:04:37.321 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:04:37.321 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:04:37.321 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:37.322 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:04:37Z" 2022-09-13T08:04:37.322 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:04:37.323 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:04:37Z" } 2022-09-13T08:04:37.324 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:04:37.325 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:37.325 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:37.325 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T08:04:42.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:42.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:42.328 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:04:42.383 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:04:42.383 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:04:42.437 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:42.432+0000 7f902a8ab600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:42.437 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:42.433+0000 7f902a8ab600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:42.490 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:04:42.491 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:04:42.491 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:04:42.492 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:04:42.492 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:04:42.493 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:04:42.493 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:42.494 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:42.494 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:04:42.495 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:42.495 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:04:42.496 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:42.496 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:42.497 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:04:42.497 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:42.499 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:04:42.499 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:42.500 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:04:42.500 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:42.501 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:42.501 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:42.502 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:42.502 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:04:42.503 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:42.503 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:42.505 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:04:42.505 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:42.506 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:42.506 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:42.507 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:04:42.507 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:42.508 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:42.508 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:42.508 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:42.509 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:04:42.510 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:42.510 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:42.510 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.351164_238.1", 2022-09-13T08:04:42.511 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:42.511 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:42.511 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:42.512 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.351164Z", 2022-09-13T08:04:42.512 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:42.513 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:42.513 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:42.514 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:42.514 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:04:42.515 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:42.516 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:42.516 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:04:42.517 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:42.517 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:42.518 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:42.518 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:04:42.519 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:42.519 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:42.520 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:42.520 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:04:42.521 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:42.521 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:04:42.521 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:04:42.522 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:04:42.522 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:42.523 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:04:42Z" 2022-09-13T08:04:42.523 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:04:42.524 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.351164_238.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.351164Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:04:42Z" } 2022-09-13T08:04:42.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.351164_238.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.351164Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:04:42.525 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:42.526 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.351164_238.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:42.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.351164_238.1 2022-09-13T08:04:47.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:47.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:47.533 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:04:47.595 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:04:47.596 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:04:47.626 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:47.622+0000 7f1a878a1600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:47.627 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:47.623+0000 7f1a878a1600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:47.643 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:04:47.644 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:04:47.644 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:04:47.645 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:04:47.645 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:04:47.646 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:04:47.646 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:47.647 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:47.647 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:04:47.648 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:47.648 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:04:47.648 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:47.649 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:47.650 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:04:47.650 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:47.650 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:04:47.651 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:47.651 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:04:47.651 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:47.652 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:47.652 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:47.653 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:47.653 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:04:47.654 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:47.654 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:47.655 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:04:47.655 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:47.656 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:47.656 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:47.657 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:04:47.657 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:47.658 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:47.658 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:47.659 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:47.659 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:04:47.660 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:47.661 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:47.661 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.990623_217.1", 2022-09-13T08:04:47.662 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:47.662 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:47.663 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:47.663 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T08:04:47.664 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:47.664 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:47.665 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:47.665 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:47.665 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:04:47.666 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:47.666 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:47.667 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:04:47.667 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:47.667 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:47.668 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:47.668 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:04:47.669 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:47.669 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:47.669 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:47.669 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:04:47.670 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:47.670 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:04:47.670 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:04:47.671 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:04:47.671 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:47.671 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:04:47Z" 2022-09-13T08:04:47.671 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:04:47.673 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.990623_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.990623Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:04:47Z" } 2022-09-13T08:04:47.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:04:47.674 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:47.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:47.675 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.990623_217.1 2022-09-13T08:04:52.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:52.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:52.678 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:04:52.729 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:04:52.729 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:04:52.768 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:52.764+0000 7f528a634600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:52.769 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:52.765+0000 7f528a634600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:52.784 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:04:52.785 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:04:52.785 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:04:52.785 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:04:52.786 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:04:52.786 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:04:52.787 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:52.787 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:52.788 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:04:52.788 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:52.788 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:04:52.789 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:52.789 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:52.790 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:04:52.790 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:52.791 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:04:52.791 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:52.791 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:04:52.792 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:52.792 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:52.793 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:52.793 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:52.794 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:04:52.794 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:52.795 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:52.796 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:04:52.796 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:52.796 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:52.797 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:52.797 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:04:52.798 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:52.798 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:52.798 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:52.799 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:52.799 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:04:52.800 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:52.800 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:52.800 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.070128_219.1", 2022-09-13T08:04:52.801 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:52.801 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:52.802 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:52.802 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T08:04:52.803 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:52.803 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:52.804 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:52.804 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:52.805 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:04:52.805 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:52.806 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:52.806 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:04:52.806 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:52.806 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:52.807 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:52.807 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:04:52.807 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:52.807 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:52.808 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:52.808 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:04:52.808 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:52.809 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:04:52.809 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:04:52.809 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:04:52.809 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:52.810 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:04:52Z" 2022-09-13T08:04:52.810 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:04:52.811 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.070128_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.070128Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:04:52Z" } 2022-09-13T08:04:52.812 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:04:52.813 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:52.813 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:52.813 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.070128_219.1 2022-09-13T08:04:57.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:57.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:04:57.817 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:04:57.867 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:04:57.868 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:04:57.900 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:57.895+0000 7fb80d415600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:57.901 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:04:57.896+0000 7fb80d415600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:04:57.920 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:04:57.921 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:04:57.921 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:04:57.922 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:04:57.922 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:04:57.922 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:04:57.923 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:57.923 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:57.923 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:04:57.924 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:57.924 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:04:57.924 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:57.925 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:57.925 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:04:57.925 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:57.926 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:04:57.926 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:57.926 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:04:57.926 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:57.927 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:57.927 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:57.927 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:57.927 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:04:57.928 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:57.928 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:57.928 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:04:57.929 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:57.929 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:57.930 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:57.930 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:04:57.931 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:57.931 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:57.932 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:57.932 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:57.933 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:04:57.933 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:57.933 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:57.933 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.146590_228.1", 2022-09-13T08:04:57.934 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:57.934 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:57.935 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:57.935 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.146590Z", 2022-09-13T08:04:57.935 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:57.936 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:57.936 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:57.937 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:04:57.937 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:04:57.938 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:04:57.938 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:04:57.939 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:04:57.939 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:04:57.940 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:04:57.942 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:04:57.943 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:04:57.943 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:04:57.944 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:57.944 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:04:57.944 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:04:57.944 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:57.945 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:04:57.945 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:04:57.946 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:04:57.946 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:04:57.946 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:04:57Z" 2022-09-13T08:04:57.947 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:04:57.949 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.146590_228.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.146590Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:04:57Z" } 2022-09-13T08:04:57.950 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.146590_228.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.146590Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:04:57.951 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:57.951 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.146590_228.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:04:57.952 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.146590_228.1 2022-09-13T08:05:02.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:02.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:02.956 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:05:03.036 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:05:03.037 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:05:03.087 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:03.084+0000 7f991facc600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:03.088 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:03.086+0000 7f991facc600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:03.114 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:05:03.114 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:05:03.115 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:05:03.115 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:05:03.116 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:05:03.116 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:05:03.117 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:03.117 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:03.118 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:05:03.118 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:03.118 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:05:03.119 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:03.119 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:03.120 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:05:03.120 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:03.121 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:05:03.121 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:03.121 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:05:03.122 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:03.122 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:03.123 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:03.123 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:03.124 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:05:03.124 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:03.124 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:03.124 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:05:03.125 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:03.125 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:03.125 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:03.125 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:05:03.126 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:03.126 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:03.126 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:03.126 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:03.127 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:05:03.127 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:03.127 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:03.128 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.953175_216.1", 2022-09-13T08:05:03.128 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:03.128 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:03.128 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:03.129 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T08:05:03.129 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:03.129 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:03.129 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:03.130 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:03.130 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:05:03.130 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:03.130 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:03.131 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:05:03.131 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:03.131 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:03.132 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:03.132 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:05:03.132 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:03.133 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:03.133 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:03.133 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:05:03.133 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:03.134 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:05:03.134 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:05:03.134 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:05:03.135 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:03.135 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:05:03Z" 2022-09-13T08:05:03.135 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:05:03.136 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.953175_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.953175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:05:03Z" } 2022-09-13T08:05:03.137 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:05:03.138 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:03.138 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:03.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.953175_216.1 2022-09-13T08:05:08.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:08.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:08.142 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:05:08.194 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:05:08.194 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:05:08.240 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:08.236+0000 7fe1a2a8d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:08.241 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:08.237+0000 7fe1a2a8d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:08.260 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:05:08.261 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:05:08.261 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:05:08.262 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:05:08.262 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:05:08.263 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:05:08.263 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:08.264 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:08.264 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:05:08.265 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:08.265 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:05:08.266 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:08.268 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:08.269 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:05:08.271 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:08.272 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:05:08.272 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:08.272 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:05:08.273 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:08.273 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:08.278 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:08.279 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:08.279 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:05:08.280 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:08.280 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:08.281 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:05:08.281 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:08.282 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:08.283 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:08.283 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:05:08.284 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:08.284 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:08.285 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:08.285 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:08.286 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:05:08.286 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:08.287 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:08.288 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.911619_214.1", 2022-09-13T08:05:08.288 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:08.289 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:08.289 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:08.289 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T08:05:08.290 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:08.291 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:08.291 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:08.292 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:08.292 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:05:08.292 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:08.292 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:08.293 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:05:08.293 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:08.293 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:08.294 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:08.294 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:05:08.294 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:08.295 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:08.295 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:08.296 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:05:08.296 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:08.297 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:05:08.297 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:05:08.297 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:05:08.297 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:08.298 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:05:08Z" 2022-09-13T08:05:08.298 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:05:08.300 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.911619_214.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.911619Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:05:08Z" } 2022-09-13T08:05:08.301 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:05:08.301 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:08.302 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:08.302 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.911619_214.1 2022-09-13T08:05:13.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:13.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:13.306 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:05:13.358 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:05:13.358 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:05:13.402 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:13.398+0000 7f02aed78600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:13.402 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:13.399+0000 7f02aed78600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:13.418 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:05:13.419 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:05:13.419 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:05:13.419 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:05:13.420 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:05:13.420 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:05:13.420 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:13.420 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:13.421 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:05:13.421 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:13.421 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:05:13.421 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:13.421 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:13.422 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:05:13.422 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:13.422 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:05:13.423 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:13.423 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:05:13.423 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:13.423 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:13.424 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:13.424 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:13.424 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:05:13.424 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:13.424 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:13.425 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:05:13.425 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:13.425 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:13.426 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:13.426 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:05:13.426 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:13.427 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:13.427 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:13.427 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:13.428 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:05:13.428 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:13.428 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:13.428 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.182451_229.1", 2022-09-13T08:05:13.429 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:13.429 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:13.429 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:13.430 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T08:05:13.430 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:13.430 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:13.430 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:13.431 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:13.431 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:05:13.431 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:13.432 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:13.432 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:05:13.432 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:13.432 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:13.433 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:13.433 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:05:13.434 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:13.435 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:13.435 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:13.436 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:05:13.436 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:13.437 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:05:13.437 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:05:13.438 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:05:13.438 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:13.439 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:05:13Z" 2022-09-13T08:05:13.439 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:05:13.442 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.182451_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.182451Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:05:13Z" } 2022-09-13T08:05:13.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.182451_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.182451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:05:13.444 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:13.444 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.182451_229.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:13.445 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.182451_229.1 2022-09-13T08:05:18.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:18.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:18.450 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:05:18.501 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:05:18.502 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:05:18.539 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:18.535+0000 7f9cced3f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:18.540 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:18.536+0000 7f9cced3f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:18.558 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:05:18.559 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:05:18.559 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:05:18.560 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:05:18.560 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:05:18.560 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:05:18.561 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:18.561 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:18.562 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:05:18.562 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:18.563 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:05:18.563 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:18.564 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:18.564 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:05:18.564 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:18.565 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:05:18.565 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:18.566 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:05:18.566 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:18.567 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:18.567 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:18.568 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:18.568 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:05:18.568 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:18.569 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:18.569 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:05:18.570 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:18.570 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:18.570 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:18.571 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:05:18.571 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:18.572 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:18.572 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:18.572 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:18.573 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:05:18.573 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:18.574 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:18.574 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.182451_229.1", 2022-09-13T08:05:18.575 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:18.575 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:18.575 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:18.576 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T08:05:18.576 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:18.577 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:18.577 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:18.577 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:18.577 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:05:18.578 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:18.578 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:18.579 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:05:18.579 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:18.580 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:18.580 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:18.581 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:05:18.581 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:18.581 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:18.581 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:18.582 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:05:18.582 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:18.582 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:05:18.582 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:05:18.583 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:05:18.583 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:18.583 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:05:18Z" 2022-09-13T08:05:18.583 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:05:18.585 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.182451_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.182451Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:05:18Z" } 2022-09-13T08:05:18.585 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.182451_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.182451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:05:18.586 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:18.586 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.182451_229.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:18.587 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.182451_229.1 2022-09-13T08:05:23.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:23.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:23.593 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:05:23.698 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:05:23.699 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:05:23.786 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:23.782+0000 7fc0df94d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:23.787 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:23.783+0000 7fc0df94d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:23.814 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:05:23.815 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:05:23.815 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:05:23.815 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:05:23.815 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:05:23.816 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:05:23.816 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:23.816 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:23.816 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:05:23.817 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:23.817 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:05:23.817 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:23.817 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:23.818 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:05:23.818 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:23.818 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:05:23.819 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:23.819 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:05:23.819 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:23.819 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:23.820 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:23.820 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:23.820 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:05:23.821 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:23.821 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:23.821 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:05:23.823 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:23.823 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:23.824 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:23.824 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:05:23.824 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:23.824 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:23.825 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:23.825 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:23.825 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:05:23.825 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:23.826 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:23.826 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T08:05:23.826 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:23.826 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:23.827 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:23.827 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T08:05:23.827 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:23.827 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:23.828 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:23.828 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:23.828 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:05:23.828 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:23.829 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:23.829 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:05:23.829 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:23.829 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:23.830 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:23.830 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:05:23.830 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:23.830 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:23.831 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:23.831 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:05:23.831 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:23.832 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:05:23.832 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:05:23.832 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:05:23.832 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:23.832 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:05:23Z" 2022-09-13T08:05:23.833 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:05:23.834 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:05:23Z" } 2022-09-13T08:05:23.835 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:05:23.835 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:23.836 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:23.836 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T08:05:28.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:28.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:28.841 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:05:28.891 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:05:28.891 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:05:28.924 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:28.920+0000 7fe388a17600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:28.924 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:28.921+0000 7fe388a17600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:28.939 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:05:28.940 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:05:28.940 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:05:28.941 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:05:28.942 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:05:28.943 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:05:28.943 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:28.944 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:28.944 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:05:28.945 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:28.945 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:05:28.946 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:28.946 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:28.947 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:05:28.947 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:28.948 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:05:28.948 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:28.949 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:05:28.950 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:28.951 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:28.951 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:28.952 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:28.952 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:05:28.953 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:28.953 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:28.954 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:05:28.954 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:28.955 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:28.955 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:28.955 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:05:28.956 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:28.956 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:28.957 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:28.957 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:28.958 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:05:28.958 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:28.959 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:28.959 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.990623_217.1", 2022-09-13T08:05:28.960 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:28.960 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:28.961 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:28.962 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T08:05:28.962 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:28.963 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:28.963 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:28.964 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:28.964 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:05:28.964 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:28.965 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:28.965 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:05:28.965 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:28.966 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:28.966 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:28.967 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:05:28.967 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:28.968 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:28.968 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:28.968 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:05:28.968 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:28.969 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:05:28.969 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:05:28.969 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:05:28.969 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:28.970 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:05:28Z" 2022-09-13T08:05:28.970 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:05:28.972 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.990623_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.990623Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:05:28Z" } 2022-09-13T08:05:28.973 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:05:28.974 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:28.975 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:28.976 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.990623_217.1 2022-09-13T08:05:33.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:33.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:33.981 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:05:34.037 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:05:34.038 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:05:34.078 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:34.074+0000 7f4ce1b85600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:34.079 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:34.075+0000 7f4ce1b85600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:34.095 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:05:34.095 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:05:34.096 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:05:34.096 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:05:34.096 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:05:34.097 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:05:34.097 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:34.097 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:34.098 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:05:34.098 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:34.098 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:05:34.098 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:34.099 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:34.099 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:05:34.099 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:34.100 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:05:34.100 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:34.100 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:05:34.101 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:34.101 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:34.101 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:34.101 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:34.102 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:05:34.102 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:34.102 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:34.103 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:05:34.103 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:34.103 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:34.103 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:34.104 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:05:34.104 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:34.104 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:34.104 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:34.105 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:34.105 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:05:34.105 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:34.106 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:34.106 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.070128_219.1", 2022-09-13T08:05:34.106 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:34.106 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:34.107 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:34.107 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T08:05:34.107 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:34.108 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:34.108 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:34.108 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:34.108 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:05:34.109 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:34.109 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:34.109 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:05:34.109 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:34.110 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:34.110 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:34.110 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:05:34.111 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:34.111 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:34.111 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:34.111 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:05:34.112 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:34.112 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:05:34.112 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:05:34.112 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:05:34.113 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:34.113 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:05:34Z" 2022-09-13T08:05:34.113 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:05:34.115 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.070128_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.070128Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:05:34Z" } 2022-09-13T08:05:34.115 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:05:34.116 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:34.116 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:34.117 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.070128_219.1 2022-09-13T08:05:39.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:39.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:39.120 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:05:39.179 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:05:39.180 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:05:39.216 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:39.212+0000 7f967da46600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:39.217 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:39.213+0000 7f967da46600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:39.232 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:05:39.233 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:05:39.233 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:05:39.233 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:05:39.234 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:05:39.234 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:05:39.234 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:39.234 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:39.235 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:05:39.235 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:39.235 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:05:39.235 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:39.236 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:39.236 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:05:39.236 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:39.237 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:05:39.237 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:39.237 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:05:39.237 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:39.238 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:39.238 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:39.238 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:39.238 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:05:39.239 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:39.239 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:39.239 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:05:39.239 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:39.240 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:39.240 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:39.240 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:05:39.240 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:39.241 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:39.242 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:39.242 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:39.242 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:05:39.242 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:39.243 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:39.243 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.296844_235.1", 2022-09-13T08:05:39.243 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:39.244 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:39.244 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:39.245 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.296844Z", 2022-09-13T08:05:39.245 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:39.245 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:39.245 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:39.246 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:39.246 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:05:39.246 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:39.247 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:39.247 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:05:39.248 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:39.249 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:39.249 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:39.250 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:05:39.250 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:39.250 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:39.251 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:39.251 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:05:39.252 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:39.252 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:05:39.253 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:05:39.253 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:05:39.254 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:39.254 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:05:39Z" 2022-09-13T08:05:39.255 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:05:39.256 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.296844_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.296844Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:05:39Z" } 2022-09-13T08:05:39.257 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.296844_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.296844Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:05:39.257 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:39.258 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.296844_235.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:39.258 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.296844_235.1 2022-09-13T08:05:44.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:44.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:44.263 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:05:44.347 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:05:44.348 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:05:44.382 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:44.379+0000 7fe22cdbf600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:44.383 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:44.381+0000 7fe22cdbf600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:44.398 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:05:44.399 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:05:44.399 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:05:44.399 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:05:44.400 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:05:44.400 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:05:44.400 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:44.401 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:44.401 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:05:44.401 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:44.401 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:05:44.402 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:44.402 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:44.402 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:05:44.402 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:44.403 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:05:44.403 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:44.403 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:05:44.403 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:44.404 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:44.404 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:44.404 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:44.404 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:05:44.405 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:44.405 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:44.405 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:05:44.405 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:44.406 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:44.406 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:44.406 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:05:44.406 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:44.406 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:44.407 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:44.407 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:44.407 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:05:44.408 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:44.408 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:44.408 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.990623_217.1", 2022-09-13T08:05:44.408 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:44.409 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:44.409 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:44.409 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T08:05:44.410 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:44.410 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:44.410 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:44.410 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:44.411 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:05:44.411 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:44.411 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:44.411 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:05:44.412 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:44.412 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:44.412 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:44.412 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:05:44.413 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:44.413 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:44.413 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:44.413 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:05:44.413 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:44.414 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:05:44.414 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:05:44.414 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:05:44.415 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:44.415 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:05:44Z" 2022-09-13T08:05:44.415 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:05:44.416 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.990623_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.990623Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:05:44Z" } 2022-09-13T08:05:44.417 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:05:44.418 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:44.419 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:44.419 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.990623_217.1 2022-09-13T08:05:49.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:49.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:49.423 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:05:49.473 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:05:49.473 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:05:49.520 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:49.517+0000 7f5002ed6600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:49.521 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:49.518+0000 7f5002ed6600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:49.544 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:05:49.544 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:05:49.545 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:05:49.545 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:05:49.545 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:05:49.545 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:05:49.546 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:49.546 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:49.546 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:05:49.546 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:49.547 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:05:49.547 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:49.547 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:49.548 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:05:49.548 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:49.548 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:05:49.549 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:49.549 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:05:49.550 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:49.550 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:49.550 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:49.550 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:49.551 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:05:49.551 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:49.551 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:49.551 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:05:49.552 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:49.552 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:49.552 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:49.553 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:05:49.553 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:49.553 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:49.553 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:49.554 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:49.554 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:05:49.554 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:49.555 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:49.555 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.142561_227.1", 2022-09-13T08:05:49.555 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:49.555 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:49.556 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:49.556 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T08:05:49.556 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:49.557 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:49.557 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:49.557 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:49.557 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:05:49.558 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:49.558 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:49.558 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:05:49.558 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:49.559 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:49.559 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:49.559 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:05:49.560 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:49.560 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:49.560 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:49.560 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:05:49.561 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:49.561 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:05:49.562 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:05:49.562 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:05:49.562 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:49.563 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:05:49Z" 2022-09-13T08:05:49.563 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:05:49.565 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.142561_227.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.142561Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:05:49Z" } 2022-09-13T08:05:49.566 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:05:49.566 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:49.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:49.567 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T08:05:54.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:54.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:54.570 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:05:54.618 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:05:54.619 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:05:54.650 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:54.646+0000 7f0861d2d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:54.651 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:54.647+0000 7f0861d2d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:54.668 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:05:54.668 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:05:54.669 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:05:54.669 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:05:54.669 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:05:54.670 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:05:54.670 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:54.670 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:54.671 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:05:54.671 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:54.671 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:05:54.672 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:54.672 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:54.672 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:05:54.673 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:54.673 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:05:54.673 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:54.673 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:05:54.674 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:54.674 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:54.674 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:54.674 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:54.675 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:05:54.675 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:54.675 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:54.675 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:05:54.676 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:54.676 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:54.676 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:54.676 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:05:54.677 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:54.677 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:54.677 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:54.678 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:54.678 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:05:54.678 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:54.679 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:54.679 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T08:05:54.679 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:54.679 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:54.680 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:54.680 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T08:05:54.680 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:54.681 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:54.681 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:54.681 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:54.681 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:05:54.682 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:54.682 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:54.682 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:05:54.683 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:54.683 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:54.683 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:54.684 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:05:54.684 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:54.684 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:54.684 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:54.685 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:05:54.685 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:54.686 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:05:54.686 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:05:54.687 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:05:54.687 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:54.687 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:05:54Z" 2022-09-13T08:05:54.687 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:05:54.689 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:05:54Z" } 2022-09-13T08:05:54.689 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:05:54.690 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:54.690 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:54.691 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T08:05:59.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:59.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:05:59.694 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:05:59.744 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:05:59.744 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:05:59.797 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:59.793+0000 7f7319f25600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:59.798 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:05:59.794+0000 7f7319f25600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:05:59.822 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:05:59.822 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:05:59.823 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:05:59.823 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:05:59.823 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:05:59.824 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:05:59.824 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:59.824 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:59.824 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:05:59.825 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:59.825 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:05:59.825 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:59.826 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:59.826 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:05:59.826 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:59.827 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:05:59.827 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:59.827 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:05:59.827 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:59.828 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:59.828 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:59.828 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:59.828 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:05:59.829 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:59.829 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:59.830 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:05:59.830 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:59.831 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:59.831 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:59.831 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:05:59.832 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:59.832 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:59.832 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:59.832 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:59.833 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:05:59.833 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:59.833 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:59.833 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.990623_217.1", 2022-09-13T08:05:59.834 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:59.834 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:59.834 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:59.834 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T08:05:59.835 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:59.835 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:59.835 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:59.835 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:05:59.836 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:05:59.836 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:05:59.837 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:05:59.837 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:05:59.837 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:05:59.837 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:05:59.838 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:05:59.838 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:05:59.838 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:05:59.839 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:59.839 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:05:59.839 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:05:59.839 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:59.840 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:05:59.840 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:05:59.840 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:05:59.840 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:05:59.841 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:05:59Z" 2022-09-13T08:05:59.841 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:05:59.841 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.990623_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.990623Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:05:59Z" } 2022-09-13T08:05:59.842 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:05:59.842 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:59.843 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:05:59.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.990623_217.1 2022-09-13T08:06:04.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:04.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:04.846 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:06:04.899 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:06:04.900 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:06:04.954 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:04.950+0000 7f19ddda2600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:04.955 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:04.951+0000 7f19ddda2600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:04.977 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:06:04.977 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:06:04.977 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:06:04.978 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:06:04.978 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:06:04.979 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:06:04.979 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:04.979 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:04.979 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:06:04.980 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:04.980 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:06:04.980 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:04.981 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:04.981 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:06:04.981 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:04.981 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:06:04.982 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:04.982 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:06:04.982 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:04.983 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:04.983 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:04.983 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:04.983 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:06:04.984 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:04.984 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:04.984 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:06:04.985 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:04.985 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:04.985 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:04.985 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:06:04.986 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:04.986 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:04.986 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:04.987 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:04.987 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:06:04.987 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:04.987 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:04.988 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.111513_224.1", 2022-09-13T08:06:04.988 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:04.988 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:04.988 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:04.989 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.111513Z", 2022-09-13T08:06:04.989 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:04.989 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:04.990 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:04.990 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:04.990 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:06:04.991 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:04.991 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:04.991 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:06:04.992 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:04.992 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:04.992 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:04.992 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:06:04.993 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:04.993 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:04.993 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:04.993 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:06:04.994 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:04.994 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:06:04.994 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:06:04.995 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:06:04.995 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:04.995 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:06:04Z" 2022-09-13T08:06:04.995 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:06:04.997 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.111513_224.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.111513Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:06:04Z" } 2022-09-13T08:06:04.997 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.111513_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.111513Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:06:04.998 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:04.998 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.111513_224.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:04.999 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.111513_224.1 2022-09-13T08:06:10.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:10.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:10.002 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:06:10.100 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:06:10.100 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:06:10.153 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:10.149+0000 7f74c4bf2600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:10.154 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:10.150+0000 7f74c4bf2600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:10.168 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:06:10.168 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:06:10.169 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:06:10.169 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:06:10.170 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:06:10.170 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:06:10.171 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:10.171 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:10.171 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:06:10.172 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:10.173 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:06:10.173 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:10.173 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:10.174 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:06:10.174 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:10.175 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:06:10.175 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:10.176 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:06:10.176 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:10.177 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:10.177 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:10.178 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:10.178 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:06:10.178 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:10.179 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:10.179 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:06:10.180 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:10.180 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:10.180 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:10.180 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:06:10.181 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:10.181 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:10.181 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:10.181 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:10.182 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:06:10.182 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:10.182 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:10.182 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.949151_215.1", 2022-09-13T08:06:10.183 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:10.183 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:10.183 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:10.184 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T08:06:10.184 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:10.184 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:10.185 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:10.185 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:10.185 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:06:10.185 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:10.186 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:10.186 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:06:10.186 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:10.187 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:10.187 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:10.188 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:06:10.188 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:10.189 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:10.189 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:10.190 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:06:10.190 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:10.191 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:06:10.191 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:06:10.192 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:06:10.192 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:10.193 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:06:10Z" 2022-09-13T08:06:10.193 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:06:10.195 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.949151_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.949151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:06:10Z" } 2022-09-13T08:06:10.196 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:06:10.197 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:10.197 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:10.198 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T08:06:15.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:15.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:15.202 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:06:15.252 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:06:15.252 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:06:15.290 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:15.286+0000 7f4f9c36a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:15.291 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:15.287+0000 7f4f9c36a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:15.305 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:06:15.306 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:06:15.306 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:06:15.307 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:06:15.307 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:06:15.308 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:06:15.308 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:15.309 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:15.309 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:06:15.309 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:15.310 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:06:15.310 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:15.311 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:15.311 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:06:15.312 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:15.312 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:06:15.313 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:15.313 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:06:15.313 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:15.314 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:15.314 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:15.315 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:15.315 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:06:15.315 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:15.316 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:15.316 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:06:15.316 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:15.317 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:15.317 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:15.318 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:06:15.318 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:15.318 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:15.319 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:15.319 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:15.320 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:06:15.320 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:15.320 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:15.322 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.109037_223.1", 2022-09-13T08:06:15.322 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:15.322 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:15.323 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:15.323 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.109037Z", 2022-09-13T08:06:15.324 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:15.324 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:15.324 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:15.325 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:15.325 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:06:15.327 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:15.327 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:15.327 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:06:15.328 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:15.328 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:15.328 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:15.329 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:06:15.329 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:15.329 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:15.330 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:15.330 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:06:15.330 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:15.331 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:06:15.331 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:06:15.331 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:06:15.331 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:15.332 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:06:15Z" 2022-09-13T08:06:15.332 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:06:15.334 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.109037_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.109037Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:06:15Z" } 2022-09-13T08:06:15.334 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.109037_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.109037Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:06:15.335 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:15.336 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.109037_223.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:15.336 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.109037_223.1 2022-09-13T08:06:20.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:20.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:20.340 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:06:20.394 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:06:20.395 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:06:20.423 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:20.419+0000 7f824e747600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:20.424 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:20.420+0000 7f824e747600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:20.438 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:06:20.439 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:06:20.439 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:06:20.440 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:06:20.440 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:06:20.441 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:06:20.441 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:20.441 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:20.441 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:06:20.442 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:20.442 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:06:20.442 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:20.442 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:20.443 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:06:20.443 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:20.444 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:06:20.444 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:20.444 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:06:20.445 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:20.445 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:20.445 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:20.446 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:20.446 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:06:20.447 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:20.448 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:20.448 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:06:20.448 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:20.449 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:20.449 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:20.449 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:06:20.449 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:20.450 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:20.450 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:20.450 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:20.451 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:06:20.451 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:20.451 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:20.451 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.296844_235.1", 2022-09-13T08:06:20.452 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:20.452 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:20.452 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:20.453 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.296844Z", 2022-09-13T08:06:20.453 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:20.453 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:20.453 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:20.454 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:20.455 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:06:20.455 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:20.455 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:20.456 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:06:20.456 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:20.457 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:20.457 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:20.457 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:06:20.458 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:20.458 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:20.459 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:20.459 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:06:20.459 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:20.460 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:06:20.460 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:06:20.460 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:06:20.461 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:20.461 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:06:20Z" 2022-09-13T08:06:20.462 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:06:20.463 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.296844_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.296844Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:06:20Z" } 2022-09-13T08:06:20.464 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.296844_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.296844Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:06:20.465 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:20.466 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.296844_235.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:20.467 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.296844_235.1 2022-09-13T08:06:25.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:25.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:25.471 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:06:25.522 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:06:25.523 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:06:25.555 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:25.552+0000 7f813b602600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:25.556 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:25.553+0000 7f813b602600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:25.571 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:06:25.572 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:06:25.572 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:06:25.572 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:06:25.572 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:06:25.573 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:06:25.573 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:25.573 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:25.574 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:06:25.574 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:25.575 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:06:25.575 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:25.576 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:25.576 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:06:25.577 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:25.577 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:06:25.577 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:25.578 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:06:25.578 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:25.579 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:25.579 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:25.580 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:25.580 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:06:25.581 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:25.581 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:25.582 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:06:25.582 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:25.582 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:25.583 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:25.585 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:06:25.585 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:25.585 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:25.586 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:25.586 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:25.587 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:06:25.587 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:25.587 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:25.588 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.953175_216.1", 2022-09-13T08:06:25.588 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:25.589 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:25.589 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:25.589 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T08:06:25.590 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:25.590 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:25.591 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:25.591 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:25.591 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:06:25.592 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:25.592 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:25.593 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:06:25.593 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:25.594 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:25.594 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:25.594 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:06:25.595 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:25.595 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:25.596 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:25.597 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:06:25.597 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:25.597 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:06:25.598 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:06:25.598 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:06:25.599 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:25.599 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:06:25Z" 2022-09-13T08:06:25.599 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:06:25.601 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.953175_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.953175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:06:25Z" } 2022-09-13T08:06:25.602 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:06:25.605 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:25.606 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:25.606 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.953175_216.1 2022-09-13T08:06:30.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:30.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:30.609 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:06:30.667 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:06:30.667 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:06:30.714 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:30.711+0000 7f370604e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:30.714 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:30.712+0000 7f370604e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:30.736 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:06:30.737 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:06:30.737 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:06:30.737 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:06:30.738 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:06:30.738 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:06:30.739 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:30.739 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:30.739 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:06:30.740 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:30.740 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:06:30.740 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:30.741 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:30.741 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:06:30.742 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:30.742 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:06:30.743 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:30.743 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:06:30.744 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:30.744 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:30.744 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:30.745 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:30.745 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:06:30.745 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:30.746 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:30.746 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:06:30.747 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:30.747 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:30.748 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:30.748 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:06:30.748 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:30.749 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:30.749 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:30.750 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:30.750 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:06:30.750 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:30.751 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:30.751 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.105395_221.1", 2022-09-13T08:06:30.751 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:30.752 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:30.752 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:30.752 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T08:06:30.752 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:30.753 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:30.753 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:30.754 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:30.754 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:06:30.755 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:30.755 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:30.756 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:06:30.757 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:30.757 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:30.758 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:30.758 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:06:30.759 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:30.759 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:30.760 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:30.760 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:06:30.761 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:30.761 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:06:30.762 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:06:30.762 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:06:30.763 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:30.763 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:06:30Z" 2022-09-13T08:06:30.764 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:06:30.766 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.105395_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.105395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:06:30Z" } 2022-09-13T08:06:30.767 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.105395_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.105395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:06:30.768 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:30.769 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.105395_221.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:30.770 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.105395_221.1 2022-09-13T08:06:35.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:35.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:35.774 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:06:35.825 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:06:35.826 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:06:35.868 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:35.865+0000 7ff51a19d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:35.869 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:35.866+0000 7ff51a19d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:35.885 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:06:35.885 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:06:35.885 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:06:35.886 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:06:35.886 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:06:35.886 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:06:35.887 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:35.887 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:35.887 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:06:35.888 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:35.888 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:06:35.888 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:35.888 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:35.889 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:06:35.889 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:35.889 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:06:35.889 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:35.893 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:06:35.893 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:35.893 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:35.893 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:35.894 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:35.894 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:06:35.894 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:35.894 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:35.895 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:06:35.896 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:35.896 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:35.897 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:35.897 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:06:35.897 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:35.897 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:35.898 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:35.898 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:35.899 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:06:35.899 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:35.900 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:35.900 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.070128_219.1", 2022-09-13T08:06:35.900 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:35.900 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:35.901 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:35.901 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T08:06:35.903 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:35.903 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:35.903 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:35.904 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:35.904 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:06:35.904 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:35.905 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:35.905 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:06:35.906 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:35.907 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:35.907 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:06:35.908 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:35.909 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:35.910 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:35.910 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:06:35.910 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:35.911 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:06:35.911 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:06:35.912 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:06:35.912 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:35.913 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:06:35Z" 2022-09-13T08:06:35.913 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:06:35.915 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.070128_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.070128Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:06:35Z" } 2022-09-13T08:06:35.915 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:06:35.916 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:35.916 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:35.917 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.070128_219.1 2022-09-13T08:06:40.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2022-09-13T08:06:40.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:40.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:40.934 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:06:40.992 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:06:40.992 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:06:41.029 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T08:06:41.038 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T08:06:41.039 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:06:41.039 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:06:41.040 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T08:06:41.040 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:06:41.041 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:06:41.041 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T08:06:41.042 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T08:06:41.042 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:06:41.043 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:06:41.043 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663056091.338062_357.1", 2022-09-13T08:06:41.045 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T08:01:31.338062Z" 2022-09-13T08:06:41.045 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:06:41.046 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:06:41.046 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:06:41.046 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T08:06:41.047 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:06:41.047 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T08:06:41.048 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:41.049 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T08:06:41.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:41.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:41.051 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:06:41.101 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:06:41.102 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:06:41.132 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:41.128+0000 7f013ca3d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:41.133 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:41.129+0000 7f013ca3d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:41.149 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:06:41.149 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:06:41.150 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:06:41.150 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:06:41.150 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:06:41.151 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:06:41.151 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:41.151 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:41.151 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:06:41.152 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:41.152 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:06:41.152 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:41.152 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:41.153 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:06:41.153 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:41.153 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:06:41.153 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:41.154 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:06:41.154 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:41.155 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:41.155 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:41.156 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:41.156 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:06:41.157 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:41.157 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:41.157 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:06:41.158 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:41.159 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:41.160 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:41.160 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:06:41.161 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:41.161 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:41.161 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:41.162 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:41.162 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:06:41.163 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:41.163 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:41.164 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.953175_216.1", 2022-09-13T08:06:41.164 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:41.164 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:41.165 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:41.165 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T08:06:41.166 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:41.166 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:41.166 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:41.167 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:41.167 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:06:41.167 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:41.167 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:41.168 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:06:41.168 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:41.168 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:41.169 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:41.169 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:06:41.169 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:41.170 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:41.170 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:41.171 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:06:41.171 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:41.172 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:06:41.172 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:06:41.172 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:06:41.173 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:41.173 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:06:41Z" 2022-09-13T08:06:41.174 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:06:41.176 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.953175_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.953175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:06:41Z" } 2022-09-13T08:06:41.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:06:41.178 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:41.179 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:41.179 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.953175_216.1 2022-09-13T08:06:46.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:46.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:46.184 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:06:46.236 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:06:46.237 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:06:46.277 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:46.273+0000 7f5f20e5e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:46.278 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:46.274+0000 7f5f20e5e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:46.294 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:06:46.295 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:06:46.295 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:06:46.295 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:06:46.295 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:06:46.296 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:06:46.296 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:46.296 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:46.297 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:06:46.297 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:46.297 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:06:46.300 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:46.300 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:46.300 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:06:46.300 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:46.302 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:06:46.311 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:46.312 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:06:46.313 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:46.313 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:46.314 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:46.314 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:46.314 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:06:46.314 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:46.315 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:46.315 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:06:46.315 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:46.316 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:46.316 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:46.318 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:06:46.318 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:46.318 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:46.319 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:46.319 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:46.319 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:06:46.320 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:46.320 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:46.320 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.953175_216.1", 2022-09-13T08:06:46.321 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:46.321 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:46.321 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:46.321 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T08:06:46.323 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:46.324 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:46.324 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:46.325 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:46.325 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:06:46.326 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:46.326 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:46.327 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:06:46.327 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:46.328 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:46.328 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:46.329 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:06:46.329 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:46.330 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:46.330 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:46.331 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:06:46.331 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:46.331 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:06:46.332 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:06:46.332 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:06:46.332 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:46.333 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:06:46Z" 2022-09-13T08:06:46.333 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:06:46.335 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.953175_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.953175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:06:46Z" } 2022-09-13T08:06:46.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:06:46.338 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:46.339 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:46.339 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.953175_216.1 2022-09-13T08:06:51.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:51.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:51.344 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:06:51.409 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:06:51.410 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:06:51.697 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:51.693+0000 7fbc4ae08600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:51.701 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:51.694+0000 7fbc4ae08600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:51.738 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:06:51.739 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:06:51.739 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:06:51.740 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:06:51.740 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:06:51.741 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:06:51.741 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:51.744 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:51.745 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:06:51.745 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:51.745 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:06:51.746 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:51.746 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:51.746 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:06:51.747 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:51.747 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:06:51.747 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:51.748 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:06:51.748 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:51.748 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:51.748 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:51.749 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:51.749 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:06:51.749 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:51.749 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:51.750 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:06:51.750 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:51.751 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:51.751 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:51.751 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:06:51.751 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:51.753 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:51.753 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:51.753 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:51.754 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:06:51.754 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:51.754 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:51.755 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.134902_225.1", 2022-09-13T08:06:51.755 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:51.756 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:51.756 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:51.756 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.134902Z", 2022-09-13T08:06:51.756 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:51.757 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:51.757 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:51.757 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:51.758 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:06:51.758 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:51.758 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:51.758 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:06:51.759 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:51.759 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:51.759 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:51.760 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:06:51.760 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:51.760 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:51.760 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:51.761 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:06:51.761 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:51.762 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:06:51.762 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:06:51.762 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:06:51.762 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:51.763 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:06:51Z" 2022-09-13T08:06:51.763 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:06:51.766 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.134902_225.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.134902Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:06:51Z" } 2022-09-13T08:06:51.766 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.134902_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.134902Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:06:51.767 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:51.767 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.134902_225.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:51.768 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.134902_225.1 2022-09-13T08:06:56.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:56.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:06:56.772 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:06:56.824 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:06:56.825 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:06:56.856 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:56.852+0000 7f367befa600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:56.857 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:06:56.853+0000 7f367befa600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:06:56.874 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:06:56.874 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:06:56.875 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:06:56.875 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:06:56.875 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:06:56.876 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:06:56.876 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:56.876 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:56.876 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:06:56.877 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:56.877 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:06:56.877 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:56.877 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:56.878 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:06:56.878 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:56.878 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:06:56.878 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:56.879 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:06:56.879 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:56.879 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:56.880 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:56.880 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:56.880 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:06:56.880 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:56.881 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:56.881 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:06:56.881 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:56.881 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:56.882 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:56.882 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:06:56.882 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:56.882 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:56.883 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:56.883 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:56.883 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:06:56.884 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:56.884 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:56.884 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.111513_224.1", 2022-09-13T08:06:56.885 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:56.885 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:56.885 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:56.885 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.111513Z", 2022-09-13T08:06:56.886 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:56.886 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:56.886 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:56.886 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:06:56.887 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:06:56.887 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:06:56.887 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:06:56.887 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:06:56.888 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:06:56.888 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:06:56.888 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:06:56.888 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:06:56.889 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:06:56.889 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:56.891 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:06:56.891 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:06:56.892 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:56.892 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:06:56.893 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:06:56.893 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:06:56.894 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:06:56.895 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:06:56Z" 2022-09-13T08:06:56.895 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:06:56.898 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.111513_224.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.111513Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:06:56Z" } 2022-09-13T08:06:56.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.111513_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.111513Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:06:56.900 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:56.901 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.111513_224.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:06:56.902 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.111513_224.1 2022-09-13T08:07:01.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:01.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:01.905 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:07:01.956 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:07:01.957 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:07:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:01.998+0000 7fc25a5d0600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:02.002 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:01.999+0000 7fc25a5d0600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:02.020 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:07:02.021 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:07:02.021 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:07:02.021 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:07:02.022 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:07:02.022 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:07:02.022 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:02.022 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:02.023 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:07:02.023 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:02.023 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:07:02.024 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:02.024 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:02.024 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:07:02.024 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:02.025 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:07:02.025 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:02.025 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:07:02.025 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:02.026 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:02.027 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:02.027 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:02.028 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:07:02.028 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:02.029 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:02.029 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:07:02.029 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:02.030 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:02.030 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:02.031 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:07:02.031 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:02.032 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:02.033 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:02.033 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:02.034 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:07:02.034 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:02.034 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:02.035 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.182451_229.1", 2022-09-13T08:07:02.035 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:02.036 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:02.036 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:02.036 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T08:07:02.037 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:02.038 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:02.038 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:02.039 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:02.039 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:07:02.039 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:02.039 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:02.040 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:07:02.040 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:02.040 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:02.040 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:02.041 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:07:02.041 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:02.042 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:02.042 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:02.043 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:07:02.043 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:02.044 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:07:02.046 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:07:02.046 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:07:02.047 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:02.047 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:07:02Z" 2022-09-13T08:07:02.047 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:07:02.050 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.182451_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.182451Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:07:02Z" } 2022-09-13T08:07:02.051 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.182451_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.182451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:07:02.051 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:02.052 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.182451_229.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:02.052 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.182451_229.1 2022-09-13T08:07:07.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:07.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:07.056 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:07:07.107 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:07:07.107 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:07:07.148 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:07.144+0000 7f0c29226600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:07.149 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:07.145+0000 7f0c29226600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:07.172 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:07:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:07:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:07:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:07:07.174 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:07:07.175 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:07:07.175 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:07.176 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:07.176 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:07:07.176 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:07.177 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:07:07.178 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:07.178 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:07.179 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:07:07.179 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:07.179 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:07:07.180 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:07.180 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:07:07.180 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:07.181 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:07.181 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:07.182 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:07.182 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:07:07.182 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:07.182 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:07.183 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:07:07.183 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:07.183 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:07.183 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:07.184 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:07:07.184 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:07.185 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:07.185 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:07.185 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:07.185 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:07:07.186 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:07.186 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:07.186 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.351164_238.1", 2022-09-13T08:07:07.187 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:07.187 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:07.187 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:07.187 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.351164Z", 2022-09-13T08:07:07.188 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:07.188 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:07.188 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:07.188 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:07.189 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:07:07.189 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:07.189 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:07.190 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:07:07.190 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:07.190 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:07.190 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:07.191 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:07:07.191 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:07.192 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:07.192 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:07.193 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:07:07.193 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:07.194 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:07:07.194 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:07:07.195 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:07:07.195 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:07.195 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:07:07Z" 2022-09-13T08:07:07.196 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:07:07.197 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.351164_238.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.351164Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:07:07Z" } 2022-09-13T08:07:07.198 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.351164_238.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.351164Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:07:07.199 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:07.199 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.351164_238.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:07.200 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.351164_238.1 2022-09-13T08:07:12.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:12.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:12.203 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:07:12.252 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:07:12.253 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:07:12.284 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:12.281+0000 7fb436a33600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:12.285 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:12.282+0000 7fb436a33600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:12.302 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:07:12.303 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:07:12.303 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:07:12.303 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:07:12.304 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:07:12.304 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:07:12.304 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:12.305 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:12.305 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:07:12.305 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:12.306 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:07:12.307 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:12.308 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:12.309 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:07:12.309 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:12.310 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:07:12.310 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:12.310 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:07:12.311 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:12.311 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:12.311 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:12.311 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:12.312 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:07:12.312 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:12.312 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:12.313 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:07:12.313 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:12.313 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:12.314 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:12.314 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:07:12.314 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:12.314 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:12.315 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:12.315 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:12.315 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:07:12.315 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:12.316 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:12.316 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.911619_214.1", 2022-09-13T08:07:12.316 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:12.317 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:12.317 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:12.317 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T08:07:12.317 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:12.318 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:12.318 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:12.318 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:12.319 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:07:12.319 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:12.319 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:12.319 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:07:12.320 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:12.320 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:12.320 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:12.321 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:07:12.321 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:12.321 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:12.322 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:12.322 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:07:12.322 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:12.323 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:07:12.323 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:07:12.324 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:07:12.324 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:12.325 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:07:12Z" 2022-09-13T08:07:12.325 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:07:12.327 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.911619_214.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.911619Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:07:12Z" } 2022-09-13T08:07:12.328 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:07:12.329 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:12.330 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:12.331 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.911619_214.1 2022-09-13T08:07:17.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:17.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:17.335 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:07:17.387 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:07:17.387 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:07:17.417 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:17.414+0000 7efe9bcdd600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:17.418 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:17.415+0000 7efe9bcdd600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:17.432 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:07:17.433 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:07:17.433 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:07:17.434 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:07:17.434 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:07:17.435 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:07:17.435 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:17.435 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:17.436 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:07:17.436 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:17.437 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:07:17.437 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:17.438 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:17.438 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:07:17.438 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:17.439 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:07:17.439 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:17.440 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:07:17.440 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:17.440 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:17.441 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:17.441 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:17.442 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:07:17.442 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:17.443 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:17.443 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:07:17.443 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:17.444 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:17.444 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:17.445 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:07:17.445 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:17.445 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:17.446 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:17.446 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:17.447 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:07:17.447 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:17.447 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:17.448 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.107994_222.1", 2022-09-13T08:07:17.448 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:17.449 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:17.449 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:17.449 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.107994Z", 2022-09-13T08:07:17.450 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:17.450 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:17.451 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:17.451 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:17.452 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:07:17.452 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:17.453 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:17.453 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:07:17.453 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:17.454 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:17.454 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:17.455 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:07:17.455 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:17.456 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:17.456 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:17.456 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:07:17.457 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:17.457 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:07:17.458 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:07:17.458 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:07:17.458 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:17.459 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:07:17Z" 2022-09-13T08:07:17.459 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:07:17.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.107994_222.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.107994Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:07:17Z" } 2022-09-13T08:07:17.462 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.107994_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.107994Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:07:17.463 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:17.464 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.107994_222.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:17.464 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.107994_222.1 2022-09-13T08:07:22.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:22.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:22.467 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:07:22.522 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:07:22.523 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:07:22.572 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:22.568+0000 7f0122568600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:22.573 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:22.570+0000 7f0122568600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:22.588 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:07:22.588 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:07:22.589 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:07:22.589 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:07:22.589 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:07:22.590 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:07:22.590 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:22.590 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:22.592 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:07:22.592 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:22.593 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:07:22.593 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:22.594 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:22.594 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:07:22.595 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:22.595 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:07:22.596 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:22.596 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:07:22.596 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:22.597 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:22.597 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:22.598 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:22.598 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:07:22.598 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:22.599 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:22.599 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:07:22.599 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:22.599 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:22.600 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:22.600 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:07:22.600 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:22.600 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:22.601 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:22.601 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:22.601 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:07:22.602 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:22.602 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:22.603 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.182451_229.1", 2022-09-13T08:07:22.603 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:22.603 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:22.603 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:22.604 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T08:07:22.604 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:22.604 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:22.604 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:22.605 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:22.605 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:07:22.605 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:22.605 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:22.606 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:07:22.606 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:22.606 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:22.606 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:22.607 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:07:22.607 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:22.607 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:22.608 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:22.608 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:07:22.608 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:22.608 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:07:22.609 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:07:22.609 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:07:22.609 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:22.610 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:07:22Z" 2022-09-13T08:07:22.610 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:07:22.611 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.182451_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.182451Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:07:22Z" } 2022-09-13T08:07:22.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.182451_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.182451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:07:22.613 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:22.613 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.182451_229.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:22.614 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.182451_229.1 2022-09-13T08:07:27.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:27.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:27.617 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:07:27.716 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:07:27.717 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:07:27.794 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:27.790+0000 7f7679ff6600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:27.794 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:27.791+0000 7f7679ff6600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:27.809 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:07:27.810 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:07:27.810 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:07:27.810 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:07:27.811 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:07:27.811 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:07:27.811 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:27.811 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:27.812 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:07:27.812 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:27.812 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:07:27.813 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:27.813 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:27.813 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:07:27.813 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:27.814 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:07:27.814 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:27.814 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:07:27.815 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:27.815 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:27.815 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:27.815 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:27.816 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:07:27.816 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:27.816 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:27.817 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:07:27.817 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:27.817 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:27.817 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:27.818 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:07:27.818 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:27.818 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:27.819 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:27.819 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:27.819 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:07:27.820 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:27.820 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:27.820 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055098.994620_218.1", 2022-09-13T08:07:27.820 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:27.821 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:27.821 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:27.822 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:58.994620Z", 2022-09-13T08:07:27.822 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:27.822 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:27.823 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:27.823 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:27.824 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:07:27.824 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:27.825 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:27.825 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:07:27.826 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:27.826 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:27.827 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:27.827 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:07:27.828 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:27.828 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:27.829 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:27.829 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:07:27.830 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:27.830 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:07:27.831 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:07:27.831 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:07:27.832 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:27.832 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:07:27Z" 2022-09-13T08:07:27.832 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:07:27.834 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055098.994620_218.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:58.994620Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:07:27Z" } 2022-09-13T08:07:27.835 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.994620_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.994620Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:07:27.836 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:27.836 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.994620_218.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:27.836 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.994620_218.1 2022-09-13T08:07:32.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:32.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:32.839 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:07:32.901 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:07:32.901 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:07:32.923 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:32.919+0000 7f19936c1600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:32.924 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:32.921+0000 7f19936c1600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:32.946 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:07:32.947 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:07:32.947 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:07:32.947 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:07:32.948 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:07:32.948 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:07:32.948 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:32.948 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:32.949 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:07:32.949 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:32.949 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:07:32.950 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:32.950 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:32.950 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:07:32.950 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:32.951 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:07:32.951 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:32.951 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:07:32.952 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:32.952 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:32.952 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:32.952 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:32.953 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:07:32.953 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:32.953 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:32.953 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:07:32.954 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:32.954 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:32.954 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:32.955 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:07:32.955 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:32.955 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:32.955 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:32.956 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:32.956 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:07:32.956 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:32.956 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:32.957 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.432486_239.1", 2022-09-13T08:07:32.957 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:32.957 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:32.957 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:32.958 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.432486Z", 2022-09-13T08:07:32.958 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:32.958 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:32.958 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:32.959 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:32.959 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:07:32.959 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:32.959 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:32.960 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:07:32.960 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:32.960 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:32.960 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:32.961 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:07:32.961 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:32.963 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:32.964 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:32.964 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:07:32.965 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:32.965 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:07:32.965 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:07:32.966 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:07:32.966 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:32.967 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:07:32Z" 2022-09-13T08:07:32.967 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:07:32.969 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.432486_239.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.432486Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:07:32Z" } 2022-09-13T08:07:32.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.432486_239.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.432486Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:07:32.971 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:32.972 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.432486_239.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:32.973 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.432486_239.1 2022-09-13T08:07:37.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:37.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:37.976 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:07:38.027 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:07:38.028 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:07:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:38.055+0000 7f6d8a2ab600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:38.060 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:38.056+0000 7f6d8a2ab600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:38.076 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:07:38.076 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:07:38.076 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:07:38.077 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:07:38.077 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:07:38.077 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:07:38.077 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:38.078 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:38.078 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:07:38.078 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:38.078 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:07:38.079 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:38.079 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:38.080 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:07:38.080 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:38.081 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:07:38.081 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:38.081 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:07:38.081 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:38.082 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:38.082 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:38.082 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:38.083 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:07:38.083 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:38.083 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:38.083 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:07:38.084 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:38.084 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:38.084 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:38.084 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:07:38.085 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:38.085 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:38.085 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:38.085 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:38.086 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:07:38.086 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:38.086 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:38.086 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.182451_229.1", 2022-09-13T08:07:38.087 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:38.087 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:38.087 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:38.088 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T08:07:38.088 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:38.088 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:38.088 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:38.089 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:38.089 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:07:38.089 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:38.089 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:38.090 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:07:38.091 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:38.091 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:38.091 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:38.092 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:07:38.092 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:38.092 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:38.092 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:38.093 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:07:38.093 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:38.093 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:07:38.094 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:07:38.094 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:07:38.094 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:38.094 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:07:38Z" 2022-09-13T08:07:38.095 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:07:38.096 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.182451_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.182451Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:07:38Z" } 2022-09-13T08:07:38.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.182451_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.182451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:07:38.097 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:38.098 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.182451_229.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:38.098 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.182451_229.1 2022-09-13T08:07:43.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:43.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:43.102 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:07:43.160 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:07:43.161 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:07:43.266 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:43.262+0000 7f2b4dad8600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:43.270 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:43.266+0000 7f2b4dad8600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:43.325 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:07:43.326 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:07:43.326 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:07:43.327 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:07:43.327 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:07:43.327 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:07:43.328 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:43.328 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:43.328 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:07:43.329 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:43.329 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:07:43.330 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:43.330 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:43.331 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:07:43.331 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:43.332 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:07:43.332 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:43.333 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:07:43.333 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:43.334 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:43.334 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:43.335 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:43.335 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:07:43.336 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:43.336 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:43.337 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:07:43.337 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:43.338 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:43.338 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:43.339 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:07:43.339 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:43.340 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:43.340 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:43.340 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:43.340 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:07:43.341 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:43.341 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:43.341 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.302317_236.1", 2022-09-13T08:07:43.342 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:43.342 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:43.342 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:43.342 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.302317Z", 2022-09-13T08:07:43.343 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:43.343 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:43.343 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:43.343 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:43.344 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:07:43.344 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:43.344 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:43.344 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:07:43.345 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:43.345 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:43.345 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:43.345 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:07:43.346 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:43.346 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:43.346 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:43.347 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:07:43.347 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:43.347 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:07:43.347 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:07:43.348 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:07:43.348 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:43.348 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:07:43Z" 2022-09-13T08:07:43.348 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:07:43.350 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.302317_236.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.302317Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:07:43Z" } 2022-09-13T08:07:43.350 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.302317_236.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.302317Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:07:43.351 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:43.352 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.302317_236.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:43.352 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.302317_236.1 2022-09-13T08:07:48.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:48.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:48.355 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:07:48.404 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:07:48.405 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:07:48.439 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:48.435+0000 7f7e4bf3b600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:48.440 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:48.436+0000 7f7e4bf3b600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:48.455 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:07:48.455 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:07:48.456 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:07:48.456 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:07:48.457 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:07:48.457 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:07:48.458 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:48.458 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:48.459 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:07:48.459 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:48.460 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:07:48.460 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:48.460 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:48.461 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:07:48.462 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:48.462 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:07:48.463 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:48.463 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:07:48.463 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:48.463 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:48.464 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:48.464 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:48.464 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:07:48.464 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:48.465 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:48.465 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:07:48.465 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:48.465 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:48.466 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:48.466 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:07:48.466 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:48.467 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:48.467 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:48.467 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:48.467 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:07:48.468 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:48.468 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:48.468 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.296844_235.1", 2022-09-13T08:07:48.469 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:48.469 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:48.469 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:48.469 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.296844Z", 2022-09-13T08:07:48.470 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:48.470 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:48.470 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:48.470 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:48.471 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:07:48.471 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:48.471 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:48.472 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:07:48.472 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:48.473 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:48.473 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:48.474 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:07:48.474 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:48.474 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:48.475 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:48.475 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:07:48.475 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:48.477 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:07:48.477 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:07:48.477 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:07:48.478 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:48.478 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:07:48Z" 2022-09-13T08:07:48.478 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:07:48.480 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.296844_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.296844Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:07:48Z" } 2022-09-13T08:07:48.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.296844_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.296844Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:07:48.481 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:48.482 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.296844_235.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:48.482 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.296844_235.1 2022-09-13T08:07:53.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:53.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:53.486 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:07:53.539 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:07:53.540 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:07:53.587 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:53.584+0000 7fde939dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:53.587 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:53.585+0000 7fde939dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:53.603 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:07:53.604 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:07:53.604 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:07:53.604 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:07:53.604 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:07:53.605 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:07:53.605 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:53.605 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:53.606 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:07:53.606 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:53.607 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:07:53.607 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:53.607 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:53.607 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:07:53.608 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:53.608 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:07:53.608 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:53.609 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:07:53.609 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:53.610 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:53.610 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:53.611 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:53.611 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:07:53.612 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:53.612 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:53.613 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:07:53.613 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:53.614 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:53.614 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:53.615 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:07:53.616 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:53.616 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:53.617 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:53.617 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:53.618 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:07:53.618 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:53.618 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:53.618 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.514523_248.1", 2022-09-13T08:07:53.619 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:53.619 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:53.619 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:53.619 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.514523Z", 2022-09-13T08:07:53.619 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:53.620 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:53.620 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:53.620 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:53.620 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:07:53.621 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:53.621 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:53.621 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:07:53.621 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:53.622 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:53.622 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:53.622 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:07:53.622 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:53.623 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:53.623 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:53.623 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:07:53.624 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:53.624 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:07:53.624 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:07:53.624 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:07:53.626 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:53.626 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:07:53Z" 2022-09-13T08:07:53.627 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:07:53.629 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.514523_248.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.514523Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:07:53Z" } 2022-09-13T08:07:53.630 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.514523_248.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.514523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:07:53.631 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:53.632 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.514523_248.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:53.632 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.514523_248.1 2022-09-13T08:07:58.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:58.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:07:58.637 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:07:58.694 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:07:58.695 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:07:58.732 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:58.729+0000 7f1bc2d3c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:58.733 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:07:58.730+0000 7f1bc2d3c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:07:58.754 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:07:58.755 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:07:58.755 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:07:58.755 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:07:58.756 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:07:58.756 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:07:58.756 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:58.756 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:58.757 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:07:58.757 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:58.757 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:07:58.758 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:58.758 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:58.759 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:07:58.759 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:58.759 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:07:58.759 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:58.760 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:07:58.760 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:58.760 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:58.760 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:58.761 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:58.761 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:07:58.761 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:58.762 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:58.762 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:07:58.762 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:58.762 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:58.763 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:58.763 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:07:58.763 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:58.764 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:58.764 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:58.764 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:58.765 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:07:58.765 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:58.765 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:58.766 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.340682_237.1", 2022-09-13T08:07:58.766 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:58.766 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:58.766 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:58.767 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T08:07:58.767 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:58.767 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:58.768 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:58.769 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:07:58.769 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:07:58.769 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:07:58.770 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:07:58.770 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:07:58.770 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:07:58.771 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:07:58.771 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:07:58.771 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:07:58.772 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:07:58.772 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:58.772 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:07:58.773 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:07:58.773 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:58.773 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:07:58.774 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:07:58.774 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:07:58.774 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:07:58.775 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:07:58Z" 2022-09-13T08:07:58.775 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:07:58.776 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.340682_237.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.340682Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:07:58Z" } 2022-09-13T08:07:58.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.340682_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.340682Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:07:58.778 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:58.779 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.340682_237.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:07:58.780 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.340682_237.1 2022-09-13T08:08:03.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:03.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:03.782 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:08:03.854 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:08:03.855 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:08:03.960 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:03.956+0000 7f8c02ce7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:03.960 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:03.957+0000 7f8c02ce7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:03.996 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:08:03.996 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:08:03.997 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:08:03.997 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:08:03.997 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:08:03.997 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:08:03.998 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:03.998 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:03.998 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:08:03.999 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:03.999 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:08:04.000 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:04.000 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:04.001 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:08:04.001 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:04.002 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:08:04.002 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:04.003 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:08:04.004 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:04.005 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:04.005 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:04.006 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:04.006 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:08:04.007 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:04.008 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:04.008 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:08:04.009 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:04.009 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:04.010 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:04.010 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:08:04.011 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:04.011 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:04.012 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:04.012 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:04.013 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:08:04.013 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:04.014 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:04.014 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.220845_233.1", 2022-09-13T08:08:04.015 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:04.016 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:04.016 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:04.017 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T08:08:04.017 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:04.017 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:04.018 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:04.018 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:04.018 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:08:04.019 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:04.019 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:04.019 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:08:04.019 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:04.020 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:04.020 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:04.020 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:08:04.021 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:04.021 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:04.021 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:04.022 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:08:04.022 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:04.023 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:08:04.023 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:08:04.023 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:08:04.023 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:04.024 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:08:03Z" 2022-09-13T08:08:04.024 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:08:04.025 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.220845_233.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.220845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:08:03Z" } 2022-09-13T08:08:04.026 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:08:04.027 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:04.027 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:04.028 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T08:08:09.029 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:09.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:09.031 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:08:09.085 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:08:09.085 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:08:09.121 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:09.117+0000 7fe139be3600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:09.121 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:09.118+0000 7fe139be3600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:09.137 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:08:09.137 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:08:09.137 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:08:09.138 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:08:09.138 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:08:09.138 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:08:09.139 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:09.139 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:09.139 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:08:09.140 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:09.140 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:08:09.140 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:09.141 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:09.141 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:08:09.141 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:09.142 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:08:09.142 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:09.142 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:08:09.142 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:09.143 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:09.143 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:09.143 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:09.144 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:08:09.144 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:09.144 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:09.144 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:08:09.145 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:09.145 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:09.145 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:09.145 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:08:09.146 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:09.146 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:09.146 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:09.147 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:09.147 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:08:09.147 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:09.148 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:09.148 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.142561_227.1", 2022-09-13T08:08:09.148 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:09.148 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:09.149 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:09.149 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T08:08:09.149 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:09.149 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:09.150 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:09.150 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:09.151 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:08:09.151 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:09.151 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:09.151 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:08:09.152 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:09.152 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:09.152 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:09.152 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:08:09.153 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:09.153 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:09.153 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:09.153 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:08:09.154 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:09.154 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:08:09.154 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:08:09.154 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:08:09.155 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:09.155 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:08:09Z" 2022-09-13T08:08:09.155 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:08:09.157 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.142561_227.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.142561Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:08:09Z" } 2022-09-13T08:08:09.157 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:08:09.158 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:09.159 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:09.159 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T08:08:14.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:14.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:14.162 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:08:14.214 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:08:14.215 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:08:14.246 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:14.242+0000 7f0fae10e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:14.247 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:14.243+0000 7f0fae10e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:14.262 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:08:14.262 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:08:14.262 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:08:14.263 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:08:14.263 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:08:14.263 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:08:14.264 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:14.264 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:14.264 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:08:14.264 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:14.265 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:08:14.265 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:14.265 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:14.265 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:08:14.266 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:14.266 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:08:14.266 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:14.267 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:08:14.267 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:14.267 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:14.267 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:14.268 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:14.268 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:08:14.268 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:14.268 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:14.269 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:08:14.269 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:14.269 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:14.269 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:14.270 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:08:14.270 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:14.270 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:14.271 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:14.271 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:14.271 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:08:14.271 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:14.272 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:14.272 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.220845_233.1", 2022-09-13T08:08:14.272 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:14.272 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:14.273 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:14.273 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T08:08:14.273 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:14.274 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:14.274 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:14.274 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:14.274 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:08:14.275 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:14.275 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:14.275 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:08:14.275 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:14.276 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:14.276 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:14.276 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:08:14.276 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:14.277 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:14.277 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:14.277 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:08:14.277 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:14.278 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:08:14.278 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:08:14.278 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:08:14.279 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:14.279 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:08:14Z" 2022-09-13T08:08:14.279 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:08:14.280 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.220845_233.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.220845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:08:14Z" } 2022-09-13T08:08:14.281 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:08:14.282 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:14.282 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:14.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T08:08:19.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:19.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:19.286 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:08:19.369 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:08:19.370 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:08:19.454 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:19.450+0000 7fe4b9a4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:19.457 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:19.453+0000 7fe4b9a4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:19.471 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:08:19.472 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:08:19.472 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:08:19.472 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:08:19.473 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:08:19.473 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:08:19.473 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:19.473 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:19.474 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:08:19.474 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:19.474 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:08:19.474 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:19.475 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:19.475 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:08:19.476 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:19.476 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:08:19.476 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:19.477 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:08:19.477 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:19.477 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:19.478 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:19.478 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:19.478 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:08:19.479 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:19.479 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:19.479 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:08:19.479 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:19.480 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:19.480 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:19.480 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:08:19.481 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:19.481 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:19.481 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:19.482 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:19.482 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:08:19.483 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:19.483 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:19.484 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.296844_235.1", 2022-09-13T08:08:19.484 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:19.485 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:19.485 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:19.485 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.296844Z", 2022-09-13T08:08:19.486 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:19.486 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:19.486 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:19.487 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:19.487 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:08:19.487 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:19.488 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:19.488 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:08:19.489 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:19.489 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:19.489 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:19.490 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:08:19.490 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:19.491 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:19.491 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:19.492 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:08:19.492 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:19.493 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:08:19.493 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:08:19.493 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:08:19.494 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:19.494 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:08:19Z" 2022-09-13T08:08:19.495 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:08:19.497 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.296844_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.296844Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:08:19Z" } 2022-09-13T08:08:19.498 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.296844_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.296844Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:08:19.499 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:19.500 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.296844_235.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:19.501 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.296844_235.1 2022-09-13T08:08:24.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:24.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:24.505 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:08:24.556 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:08:24.557 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:08:24.588 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:24.585+0000 7f31c7149600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:24.589 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:24.585+0000 7f31c7149600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:24.604 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:08:24.605 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:08:24.605 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:08:24.605 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:08:24.605 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:08:24.606 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:08:24.606 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:24.606 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:24.606 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:08:24.607 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:24.607 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:08:24.607 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:24.607 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:24.608 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:08:24.609 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:24.609 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:08:24.610 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:24.610 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:08:24.610 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:24.610 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:24.611 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:24.611 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:24.611 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:08:24.611 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:24.612 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:24.613 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:08:24.613 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:24.613 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:24.613 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:24.614 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:08:24.614 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:24.614 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:24.614 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:24.615 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:24.615 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:08:24.615 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:24.615 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:24.616 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.340682_237.1", 2022-09-13T08:08:24.616 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:24.616 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:24.616 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:24.617 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T08:08:24.617 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:24.617 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:24.617 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:24.618 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:24.618 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:08:24.618 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:24.618 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:24.620 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:08:24.620 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:24.620 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:24.621 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:24.621 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:08:24.621 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:24.621 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:24.622 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:24.622 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:08:24.623 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:24.624 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:08:24.624 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:08:24.624 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:08:24.625 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:24.625 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:08:24Z" 2022-09-13T08:08:24.626 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:08:24.628 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.340682_237.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.340682Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:08:24Z" } 2022-09-13T08:08:24.629 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.340682_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.340682Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:08:24.629 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:24.630 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.340682_237.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:24.630 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.340682_237.1 2022-09-13T08:08:29.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:29.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:29.633 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:08:29.691 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:08:29.693 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:08:29.730 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:29.726+0000 7f6b2ae19600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:29.727+0000 7f6b2ae19600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:29.750 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:08:29.750 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:08:29.751 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:08:29.751 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:08:29.751 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:08:29.752 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:08:29.752 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:29.753 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:29.753 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:08:29.754 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:29.754 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:08:29.754 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:29.754 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:29.755 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:08:29.755 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:29.755 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:08:29.756 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:29.756 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:08:29.756 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:29.757 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:29.757 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:29.757 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:29.758 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:08:29.758 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:29.759 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:29.759 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:08:29.759 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:29.759 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:29.760 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:29.760 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:08:29.760 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:29.761 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:29.761 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:29.761 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:29.761 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:08:29.762 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:29.762 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:29.763 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.187352_231.1", 2022-09-13T08:08:29.763 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:29.764 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:29.764 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:29.765 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T08:08:29.765 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:29.766 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:29.767 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:29.767 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:29.768 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:08:29.768 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:29.769 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:29.769 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:08:29.770 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:29.770 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:29.771 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:29.771 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:08:29.772 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:29.772 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:29.773 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:29.773 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:08:29.774 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:29.774 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:08:29.775 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:08:29.775 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:08:29.776 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:29.776 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:08:29Z" 2022-09-13T08:08:29.777 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:08:29.779 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.187352_231.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.187352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:08:29Z" } 2022-09-13T08:08:29.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.187352_231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.187352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:08:29.781 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:29.782 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.187352_231.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:29.783 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.187352_231.1 2022-09-13T08:08:34.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:34.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:34.787 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:08:34.870 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:08:34.871 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:08:34.944 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:34.940+0000 7fd31713f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:34.945 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:34.941+0000 7fd31713f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:34.967 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:08:34.968 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:08:34.968 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:08:34.968 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:08:34.969 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:08:34.969 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:08:34.969 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:34.970 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:34.970 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:08:34.971 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:34.971 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:08:34.972 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:34.972 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:34.972 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:08:34.972 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:34.973 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:08:34.973 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:34.973 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:08:34.974 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:34.974 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:34.974 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:34.975 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:34.975 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:08:34.975 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:34.975 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:34.976 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:08:34.976 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:34.976 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:34.976 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:34.977 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:08:34.977 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:34.977 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:34.978 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:34.978 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:34.978 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:08:34.978 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:34.979 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:34.979 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.186446_230.1", 2022-09-13T08:08:34.979 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:34.980 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:34.980 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:34.980 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.186446Z", 2022-09-13T08:08:34.980 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:34.981 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:34.981 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:34.981 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:34.981 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:08:34.982 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:34.982 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:34.982 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:08:34.983 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:34.983 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:34.983 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:34.984 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:08:34.984 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:34.984 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:34.984 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:34.985 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:08:34.985 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:34.985 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:08:34.985 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:08:34.986 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:08:34.986 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:34.986 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:08:34Z" 2022-09-13T08:08:34.987 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:08:34.988 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.186446_230.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.186446Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:08:34Z" } 2022-09-13T08:08:34.989 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.186446_230.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.186446Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:08:34.989 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:34.990 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.186446_230.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:34.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.186446_230.1 2022-09-13T08:08:39.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:39.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:39.994 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:08:40.044 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:08:40.045 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:08:40.091 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:40.088+0000 7f5c13bc4600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:40.092 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:40.089+0000 7f5c13bc4600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:40.108 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:08:40.108 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:08:40.108 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:08:40.109 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:08:40.109 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:08:40.109 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:08:40.109 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:40.110 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:40.110 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:08:40.110 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:40.110 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:08:40.111 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:40.111 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:40.111 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:08:40.111 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:40.112 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:08:40.112 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:40.112 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:08:40.113 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:40.113 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:40.114 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:40.114 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:40.115 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:08:40.115 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:40.115 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:40.116 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:08:40.116 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:40.116 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:40.117 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:40.117 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:08:40.118 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:40.118 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:40.119 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:40.119 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:40.119 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:08:40.120 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:40.120 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:40.121 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.142561_227.1", 2022-09-13T08:08:40.121 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:40.121 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:40.122 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:40.122 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T08:08:40.123 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:40.123 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:40.124 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:40.124 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:40.125 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:08:40.125 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:40.126 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:40.126 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:08:40.126 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:40.127 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:40.128 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:40.128 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:08:40.129 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:40.129 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:40.130 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:40.131 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:08:40.131 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:40.132 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:08:40.132 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:08:40.133 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:08:40.133 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:40.134 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:08:40Z" 2022-09-13T08:08:40.134 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:08:40.137 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.142561_227.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.142561Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:08:40Z" } 2022-09-13T08:08:40.138 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:08:40.139 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:40.140 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:40.140 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T08:08:45.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:45.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:45.145 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:08:45.199 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:08:45.200 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:08:45.242 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:45.239+0000 7fd0e6d1c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:45.243 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:45.240+0000 7fd0e6d1c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:45.261 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:08:45.261 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:08:45.262 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:08:45.262 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:08:45.263 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:08:45.263 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:08:45.263 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:45.264 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:45.264 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:08:45.265 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:45.266 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:08:45.266 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:45.267 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:45.267 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:08:45.268 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:45.268 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:08:45.268 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:45.270 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:08:45.270 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:45.271 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:45.271 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:45.271 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:45.272 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:08:45.272 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:45.272 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:45.272 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:08:45.273 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:45.273 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:45.273 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:45.273 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:08:45.274 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:45.274 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:45.274 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:45.275 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:45.275 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:08:45.275 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:45.276 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:45.276 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.432486_239.1", 2022-09-13T08:08:45.276 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:45.276 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:45.277 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:45.277 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.432486Z", 2022-09-13T08:08:45.277 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:45.278 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:45.278 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:45.278 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:45.278 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:08:45.279 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:45.279 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:45.279 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:08:45.280 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:45.280 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:45.280 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:45.281 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:08:45.281 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:45.281 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:45.281 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:45.282 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:08:45.282 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:45.282 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:08:45.283 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:08:45.283 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:08:45.283 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:45.284 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:08:45Z" 2022-09-13T08:08:45.284 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:08:45.286 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.432486_239.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.432486Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:08:45Z" } 2022-09-13T08:08:45.287 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.432486_239.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.432486Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:08:45.287 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:45.288 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.432486_239.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:45.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.432486_239.1 2022-09-13T08:08:50.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:50.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:50.293 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:08:50.350 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:08:50.351 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:08:50.385 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:50.381+0000 7f4ee2057600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:50.390 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:50.382+0000 7f4ee2057600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:50.402 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:08:50.402 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:08:50.403 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:08:50.403 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:08:50.403 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:08:50.403 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:08:50.404 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:50.404 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:50.404 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:08:50.404 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:50.405 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:08:50.405 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:50.405 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:50.405 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:08:50.406 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:50.406 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:08:50.406 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:50.406 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:08:50.407 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:50.407 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:50.407 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:50.408 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:50.408 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:08:50.408 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:50.408 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:50.409 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:08:50.409 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:50.409 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:50.409 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:50.410 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:08:50.410 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:50.411 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:50.411 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:50.411 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:50.411 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:08:50.412 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:50.412 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:50.412 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.142561_227.1", 2022-09-13T08:08:50.413 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:50.413 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:50.413 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:50.413 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T08:08:50.414 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:50.414 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:50.414 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:50.415 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:50.415 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:08:50.415 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:50.415 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:50.415 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:08:50.417 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:50.417 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:50.418 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:50.418 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:08:50.418 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:50.418 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:50.419 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:50.419 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:08:50.419 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:50.419 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:08:50.420 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:08:50.420 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:08:50.420 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:50.420 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:08:50Z" 2022-09-13T08:08:50.421 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:08:50.422 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.142561_227.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.142561Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:08:50Z" } 2022-09-13T08:08:50.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:08:50.423 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:50.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:50.424 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T08:08:55.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:55.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:08:55.428 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:08:55.500 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:08:55.500 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:08:55.548 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:55.545+0000 7f6888539600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:55.549 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:08:55.546+0000 7f6888539600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:08:55.570 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:08:55.570 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:08:55.571 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:08:55.571 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:08:55.571 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:08:55.572 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:08:55.572 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:55.573 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:55.573 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:08:55.574 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:55.574 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:08:55.575 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:55.575 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:55.576 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:08:55.576 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:55.577 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:08:55.577 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:55.577 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:08:55.578 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:55.584 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:55.584 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:55.584 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:55.584 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:08:55.585 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:55.585 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:55.585 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:08:55.585 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:55.586 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:55.586 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:55.586 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:08:55.587 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:55.587 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:55.587 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:55.587 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:55.588 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:08:55.588 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:55.588 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:55.588 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.142561_227.1", 2022-09-13T08:08:55.589 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:55.589 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:55.589 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:55.589 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T08:08:55.590 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:55.590 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:55.590 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:55.590 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:08:55.591 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:08:55.591 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:08:55.591 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:08:55.591 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:08:55.592 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:08:55.592 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:08:55.592 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:08:55.592 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:08:55.593 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:08:55.593 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:55.593 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:08:55.594 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:08:55.594 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:55.594 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:08:55.594 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:08:55.595 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:08:55.595 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:08:55.595 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:08:55Z" 2022-09-13T08:08:55.595 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:08:55.597 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.142561_227.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.142561Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:08:55Z" } 2022-09-13T08:08:55.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:08:55.598 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:55.598 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:08:55.599 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T08:09:00.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:00.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:00.601 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:09:00.650 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:09:00.651 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:09:00.694 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:00.691+0000 7fa6ba581600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:00.695 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:00.692+0000 7fa6ba581600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:00.714 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:09:00.714 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:09:00.715 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:09:00.715 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:09:00.716 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:09:00.716 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:09:00.717 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:00.717 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:00.718 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:09:00.718 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:00.719 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:09:00.719 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:00.720 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:00.720 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:09:00.721 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:00.721 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:09:00.722 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:00.722 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:09:00.723 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:00.723 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:00.723 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:00.724 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:00.724 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:09:00.725 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:00.725 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:00.725 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:09:00.725 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:00.726 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:00.726 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:00.726 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:09:00.727 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:00.727 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:00.728 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:00.729 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:00.729 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:09:00.729 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:00.730 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:00.730 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.142561_227.1", 2022-09-13T08:09:00.730 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:00.731 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:00.731 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:00.731 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T08:09:00.732 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:00.732 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:00.732 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:00.732 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:00.733 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:09:00.733 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:00.733 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:00.734 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:09:00.734 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:00.734 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:00.735 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:00.735 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:09:00.735 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:00.736 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:00.736 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:00.737 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:09:00.737 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:00.738 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:09:00.738 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:09:00.739 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:09:00.739 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:00.740 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:09:00Z" 2022-09-13T08:09:00.740 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:09:00.743 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.142561_227.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.142561Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:09:00Z" } 2022-09-13T08:09:00.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:09:00.745 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:00.746 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:00.747 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T08:09:05.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:05.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:05.752 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:09:05.803 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:09:05.804 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:09:05.849 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:05.845+0000 7fee78fae600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:05.851 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:05.846+0000 7fee78fae600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:05.866 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:09:05.867 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:09:05.868 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:09:05.868 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:09:05.869 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:09:05.869 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:09:05.870 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:05.870 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:05.871 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:09:05.871 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:05.871 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:09:05.872 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:05.872 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:05.873 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:09:05.873 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:05.873 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:09:05.874 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:05.874 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:09:05.875 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:05.875 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:05.876 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:05.876 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:05.876 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:09:05.877 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:05.878 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:05.878 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:09:05.879 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:05.879 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:05.879 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:05.880 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:09:05.880 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:05.881 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:05.881 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:05.882 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:05.882 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:09:05.882 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:05.883 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:05.883 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.432486_239.1", 2022-09-13T08:09:05.883 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:05.884 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:05.884 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:05.885 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.432486Z", 2022-09-13T08:09:05.885 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:05.888 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:05.888 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:05.888 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:05.888 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:09:05.889 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:05.889 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:05.889 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:09:05.889 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:05.890 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:05.890 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:05.890 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:09:05.890 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:05.891 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:05.891 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:05.891 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:09:05.892 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:05.892 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:09:05.892 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:09:05.893 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:09:05.893 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:05.893 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:09:05Z" 2022-09-13T08:09:05.894 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:09:05.896 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.432486_239.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.432486Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:09:05Z" } 2022-09-13T08:09:05.897 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.432486_239.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.432486Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:09:05.898 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:05.899 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.432486_239.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:05.900 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.432486_239.1 2022-09-13T08:09:10.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:10.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:10.903 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:09:10.957 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:09:10.958 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:09:10.996 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:10.992+0000 7fccd5dc8600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:10.997 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:10.993+0000 7fccd5dc8600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:11.015 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:09:11.016 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:09:11.016 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:09:11.016 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:09:11.017 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:09:11.017 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:09:11.017 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:11.017 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:11.018 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:09:11.018 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:11.018 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:09:11.019 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:11.019 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:11.019 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:09:11.019 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:11.020 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:09:11.020 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:11.020 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:09:11.020 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:11.021 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:11.021 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:11.021 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:11.021 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:09:11.022 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:11.022 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:11.022 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:09:11.023 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:11.023 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:11.023 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:11.024 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:09:11.024 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:11.024 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:11.025 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:11.025 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:11.025 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:09:11.026 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:11.026 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:11.026 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.142561_227.1", 2022-09-13T08:09:11.026 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:11.027 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:11.027 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:11.027 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T08:09:11.027 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:11.028 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:11.028 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:11.028 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:11.028 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:09:11.029 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:11.029 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:11.029 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:09:11.029 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:11.030 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:11.030 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:11.030 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:09:11.030 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:11.031 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:11.031 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:11.031 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:09:11.032 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:11.032 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:09:11.033 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:09:11.033 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:09:11.033 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:11.033 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:09:10Z" 2022-09-13T08:09:11.034 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:09:11.036 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.142561_227.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.142561Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:09:10Z" } 2022-09-13T08:09:11.036 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:09:11.037 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:11.037 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:11.038 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T08:09:16.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:16.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:16.041 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:09:16.116 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:09:16.117 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:09:16.236 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:16.227+0000 7fe7604ff600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:16.237 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:16.228+0000 7fe7604ff600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:16.247 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:09:16.247 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:09:16.248 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:09:16.248 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:09:16.249 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:09:16.249 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:09:16.250 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:16.250 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:16.253 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:09:16.253 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:16.253 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:09:16.253 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:16.254 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:16.254 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:09:16.254 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:16.255 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:09:16.255 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:16.255 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:09:16.256 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:16.256 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:16.257 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:16.257 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:16.258 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:09:16.259 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:16.259 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:16.260 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:09:16.260 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:16.261 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:16.261 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:16.262 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:09:16.263 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:16.263 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:16.263 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:16.264 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:16.264 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:09:16.264 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:16.264 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:16.265 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.223597_234.1", 2022-09-13T08:09:16.265 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:16.265 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:16.265 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:16.266 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.223597Z", 2022-09-13T08:09:16.266 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:16.267 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:16.267 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:16.268 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:16.268 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:09:16.269 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:16.269 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:16.270 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:09:16.270 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:16.271 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:16.271 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:16.272 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:09:16.272 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:16.273 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:16.273 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:16.274 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:09:16.274 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:16.275 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:09:16.275 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:09:16.276 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:09:16.276 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:16.277 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:09:16Z" 2022-09-13T08:09:16.277 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:09:16.279 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.223597_234.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.223597Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:09:16Z" } 2022-09-13T08:09:16.280 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.223597_234.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.223597Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:09:16.281 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:16.282 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.223597_234.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:16.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.223597_234.1 2022-09-13T08:09:21.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:21.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:21.287 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:09:21.338 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:09:21.339 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:09:21.369 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:21.366+0000 7f7371985600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:21.370 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:21.367+0000 7f7371985600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:21.384 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:09:21.385 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:09:21.385 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:09:21.386 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:09:21.387 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:09:21.387 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:09:21.387 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:21.387 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:21.388 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:09:21.388 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:21.388 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:09:21.388 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:21.389 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:21.389 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:09:21.389 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:21.389 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:09:21.390 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:21.390 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:09:21.390 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:21.391 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:21.391 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:21.391 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:21.391 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:09:21.392 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:21.392 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:21.392 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:09:21.393 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:21.393 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:21.393 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:21.394 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:09:21.394 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:21.394 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:21.394 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:21.395 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:21.395 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:09:21.395 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:21.396 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:21.396 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.142561_227.1", 2022-09-13T08:09:21.396 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:21.396 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:21.397 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:21.397 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T08:09:21.397 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:21.397 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:21.398 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:21.398 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:21.399 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:09:21.399 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:21.399 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:21.399 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:09:21.400 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:21.400 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:21.400 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:21.400 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:09:21.401 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:21.401 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:21.401 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:21.401 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:09:21.402 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:21.402 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:09:21.402 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:09:21.402 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:09:21.403 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:21.403 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:09:21Z" 2022-09-13T08:09:21.404 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:09:21.405 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.142561_227.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.142561Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:09:21Z" } 2022-09-13T08:09:21.406 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:09:21.407 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:21.407 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:21.408 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T08:09:26.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:26.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:26.412 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:09:26.467 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:09:26.468 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:09:26.507 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:26.504+0000 7f129af35600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:26.508 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:26.505+0000 7f129af35600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:26.523 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:09:26.524 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:09:26.524 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:09:26.525 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:09:26.525 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:09:26.526 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:09:26.526 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:26.527 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:26.527 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:09:26.528 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:26.528 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:09:26.529 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:26.529 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:26.530 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:09:26.530 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:26.531 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:09:26.532 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:26.532 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:09:26.533 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:26.533 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:26.534 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:26.534 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:26.535 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:09:26.535 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:26.536 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:26.536 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:09:26.537 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:26.537 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:26.538 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:26.538 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:09:26.539 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:26.539 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:26.540 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:26.540 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:26.541 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:09:26.541 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:26.542 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:26.543 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:09:26.543 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:26.544 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:26.544 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:26.545 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:09:26.545 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:26.546 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:26.547 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:26.547 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:26.548 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:09:26.548 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:26.549 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:26.549 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:09:26.550 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:26.550 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:26.551 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:26.551 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:09:26.552 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:26.552 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:26.553 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:26.553 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:09:26.554 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:26.554 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:09:26.555 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:09:26.555 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:09:26.556 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:26.556 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:09:26Z" 2022-09-13T08:09:26.557 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:09:26.559 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:09:26Z" } 2022-09-13T08:09:26.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:09:26.561 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:26.561 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:26.562 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.141531_226.1 2022-09-13T08:09:31.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:31.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:31.583 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:09:31.641 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:09:31.642 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:09:31.683 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:31.680+0000 7f5073e0c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:31.685 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:31.681+0000 7f5073e0c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:31.704 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:09:31.705 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:09:31.705 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:09:31.706 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:09:31.706 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:09:31.707 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:09:31.707 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:31.708 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:31.709 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:09:31.709 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:31.710 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:09:31.710 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:31.711 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:31.711 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:09:31.712 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:31.712 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:09:31.713 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:31.713 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:09:31.713 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:31.714 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:31.714 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:31.715 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:31.715 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:09:31.716 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:31.716 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:31.717 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:09:31.717 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:31.718 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:31.718 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:31.719 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:09:31.719 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:31.720 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:31.720 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:31.721 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:31.721 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:09:31.721 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:31.722 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:31.722 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.351164_238.1", 2022-09-13T08:09:31.723 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:31.723 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.351164Z", 2022-09-13T08:09:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:31.725 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:31.725 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:31.726 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:31.726 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:09:31.727 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:31.727 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:31.728 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:09:31.728 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:31.729 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:31.729 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:31.730 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:09:31.730 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:31.731 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:31.731 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:31.732 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:09:31.732 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:31.733 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:09:31.733 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:09:31.733 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:09:31.734 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:31.735 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:09:31Z" 2022-09-13T08:09:31.735 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:09:31.737 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.351164_238.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.351164Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:09:31Z" } 2022-09-13T08:09:31.738 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.351164_238.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.351164Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:09:31.739 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:31.740 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.351164_238.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:31.741 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.351164_238.1 2022-09-13T08:09:36.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:36.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:36.746 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:09:36.806 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:09:36.807 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:09:36.834 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:36.830+0000 7f5d39624600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:36.835 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:36.831+0000 7f5d39624600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:36.851 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:09:36.852 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:09:36.852 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:09:36.853 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:09:36.853 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:09:36.853 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:09:36.854 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:36.854 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:36.854 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:09:36.855 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:36.855 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:09:36.855 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:36.855 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:36.856 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:09:36.856 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:36.856 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:09:36.856 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:36.857 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:09:36.858 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:36.858 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:36.859 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:36.859 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:36.860 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:09:36.860 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:36.861 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:36.862 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:09:36.863 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:36.863 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:36.864 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:36.864 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:09:36.865 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:36.865 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:36.865 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:36.867 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:36.867 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:09:36.868 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:36.868 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:36.869 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.142561_227.1", 2022-09-13T08:09:36.869 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:36.869 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:36.870 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:36.870 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T08:09:36.870 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:36.870 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:36.871 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:36.871 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:36.871 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:09:36.871 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:36.872 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:36.872 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:09:36.872 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:36.872 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:36.873 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:36.873 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:09:36.873 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:36.873 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:36.874 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:36.874 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:09:36.874 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:36.875 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:09:36.875 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:09:36.875 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:09:36.875 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:36.876 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:09:36Z" 2022-09-13T08:09:36.876 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:09:36.876 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.142561_227.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.142561Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:09:36Z" } 2022-09-13T08:09:36.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:09:36.879 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:36.880 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:36.880 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T08:09:41.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:41.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:41.884 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:09:41.943 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:09:41.943 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:09:41.984 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:41.980+0000 7f48d2dd2600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:41.985 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:41.981+0000 7f48d2dd2600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:42.007 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:09:42.007 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:09:42.008 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:09:42.012 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:09:42.013 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:09:42.014 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:09:42.014 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:42.015 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:42.016 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:09:42.016 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:42.017 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:09:42.017 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:42.018 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:42.018 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:09:42.019 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:42.019 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:09:42.020 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:42.020 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:09:42.021 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:42.021 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:42.022 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:42.023 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:42.023 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:09:42.024 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:42.025 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:42.026 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:09:42.026 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:42.027 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:42.027 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:42.028 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:09:42.028 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:42.029 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:42.029 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:42.030 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:42.031 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:09:42.031 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:42.032 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:42.032 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.459269_243.1", 2022-09-13T08:09:42.033 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:42.033 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:42.034 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:42.036 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.459269Z", 2022-09-13T08:09:42.036 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:42.037 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:42.037 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:42.038 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:42.038 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:09:42.039 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:42.039 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:42.040 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:09:42.040 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:42.041 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:42.041 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:42.042 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:09:42.042 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:42.047 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:42.047 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:42.047 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:09:42.048 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:42.048 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:09:42.048 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:09:42.049 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:09:42.049 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:42.049 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:09:41Z" 2022-09-13T08:09:42.050 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:09:42.059 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.459269_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.459269Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:09:41Z" } 2022-09-13T08:09:42.060 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.459269_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.459269Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:09:42.061 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:42.062 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.459269_243.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:42.063 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.459269_243.1 2022-09-13T08:09:47.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:47.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:47.067 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:09:47.124 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:09:47.125 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:09:47.164 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:47.159+0000 7f95fe8b7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:47.165 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:47.160+0000 7f95fe8b7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:47.179 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:09:47.180 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:09:47.180 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:09:47.181 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:09:47.181 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:09:47.182 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:09:47.182 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:47.183 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:47.184 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:09:47.184 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:47.184 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:09:47.185 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:47.185 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:47.185 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:09:47.186 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:47.186 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:09:47.186 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:47.187 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:09:47.187 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:47.187 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:47.187 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:47.188 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:47.188 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:09:47.188 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:47.190 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:47.190 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:09:47.190 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:47.191 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:47.191 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:47.191 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:09:47.191 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:47.192 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:47.192 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:47.192 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:47.192 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:09:47.193 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:47.193 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:47.193 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.146590_228.1", 2022-09-13T08:09:47.193 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:47.197 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:47.197 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:47.198 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.146590Z", 2022-09-13T08:09:47.198 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:47.199 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:47.199 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:47.200 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:47.200 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:09:47.200 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:47.201 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:47.201 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:09:47.201 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:47.201 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:47.202 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:47.202 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:09:47.202 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:47.202 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:47.203 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:47.203 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:09:47.203 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:47.204 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:09:47.204 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:09:47.205 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:09:47.205 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:47.205 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:09:47Z" 2022-09-13T08:09:47.206 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:09:47.209 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.146590_228.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.146590Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:09:47Z" } 2022-09-13T08:09:47.210 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.146590_228.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.146590Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:09:47.210 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:47.211 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.146590_228.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:47.212 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.146590_228.1 2022-09-13T08:09:52.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:52.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:52.216 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:09:52.304 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:09:52.305 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:09:52.370 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:52.362+0000 7fb6ca414600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:52.371 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:52.364+0000 7fb6ca414600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:52.394 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:09:52.394 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:09:52.395 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:09:52.395 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:09:52.395 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:09:52.396 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:09:52.396 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:52.396 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:52.396 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:09:52.397 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:52.397 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:09:52.397 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:52.397 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:52.398 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:09:52.398 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:52.398 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:09:52.398 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:52.399 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:09:52.399 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:52.399 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:52.399 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:52.400 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:52.400 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:09:52.400 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:52.400 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:52.401 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:09:52.401 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:52.401 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:52.401 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:52.402 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:09:52.402 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:52.402 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:52.402 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:52.403 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:52.403 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:09:52.403 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:52.403 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:52.404 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055101.685819_259.1", 2022-09-13T08:09:52.404 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:52.404 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:52.404 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:52.405 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:45:01.685819Z", 2022-09-13T08:09:52.405 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:52.405 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:52.405 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:52.406 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:52.406 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:09:52.406 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:52.406 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:52.407 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:09:52.407 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:52.407 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:52.407 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:52.408 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:09:52.408 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:52.408 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:52.408 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:52.409 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:09:52.409 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:52.409 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:09:52.409 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:09:52.409 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:09:52.410 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:52.410 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:09:52Z" 2022-09-13T08:09:52.410 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:09:52.411 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055101.685819_259.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:45:01.685819Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:09:52Z" } 2022-09-13T08:09:52.412 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055101.685819_259.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:45:01.685819Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:09:52.413 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:52.413 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.685819_259.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:52.413 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055101.685819_259.1 2022-09-13T08:09:57.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:57.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:09:57.416 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:09:57.467 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:09:57.468 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:09:57.505 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:57.502+0000 7f33e14b9600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:09:57.503+0000 7f33e14b9600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:09:57.524 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:09:57.525 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:09:57.525 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:09:57.526 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:09:57.526 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:09:57.526 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:09:57.526 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:57.527 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:57.527 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:09:57.527 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:57.528 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:09:57.528 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:57.528 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:57.528 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:09:57.529 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:57.529 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:09:57.529 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:57.529 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:09:57.530 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:57.531 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:57.532 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:57.532 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:57.533 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:09:57.533 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:57.533 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:57.534 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:09:57.534 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:57.534 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:57.534 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:57.535 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:09:57.535 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:57.535 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:57.535 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:57.536 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:57.536 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:09:57.536 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:57.536 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:57.537 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.223597_234.1", 2022-09-13T08:09:57.538 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:57.538 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:57.538 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:57.539 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.223597Z", 2022-09-13T08:09:57.541 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:57.541 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:57.541 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:57.542 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:09:57.542 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:09:57.543 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:09:57.543 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:09:57.544 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:09:57.544 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:09:57.545 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:09:57.545 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:09:57.545 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:09:57.545 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:09:57.546 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:57.546 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:09:57.546 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:09:57.547 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:57.547 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:09:57.547 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:09:57.548 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:09:57.548 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:09:57.548 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:09:57Z" 2022-09-13T08:09:57.549 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:09:57.550 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.223597_234.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.223597Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:09:57Z" } 2022-09-13T08:09:57.551 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.223597_234.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.223597Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:09:57.552 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:57.553 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.223597_234.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:09:57.554 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.223597_234.1 2022-09-13T08:10:02.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:02.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:02.558 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:10:02.613 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:10:02.613 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:10:02.644 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:02.641+0000 7f560ca44600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:02.645 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:02.642+0000 7f560ca44600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:02.660 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:10:02.660 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:10:02.661 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:10:02.661 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:10:02.661 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:10:02.662 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:10:02.662 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:02.663 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:02.663 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:10:02.664 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:02.664 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:10:02.664 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:02.665 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:02.665 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:10:02.666 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:02.666 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:10:02.667 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:02.667 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:10:02.667 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:02.668 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:02.668 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:02.669 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:02.669 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:10:02.670 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:02.670 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:02.671 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:10:02.671 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:02.671 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:02.672 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:02.672 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:10:02.673 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:02.673 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:02.673 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:02.674 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:02.674 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:10:02.675 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:02.675 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:02.675 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.459269_243.1", 2022-09-13T08:10:02.676 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:02.676 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:02.677 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:02.677 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.459269Z", 2022-09-13T08:10:02.678 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:02.678 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:02.678 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:02.679 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:02.679 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:10:02.680 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:02.680 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:02.681 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:10:02.681 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:02.681 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:02.682 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:02.682 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:10:02.683 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:02.683 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:02.684 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:02.684 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:10:02.684 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:02.685 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:10:02.685 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:10:02.686 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:10:02.686 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:02.687 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:10:02Z" 2022-09-13T08:10:02.687 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:10:02.689 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.459269_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.459269Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:10:02Z" } 2022-09-13T08:10:02.690 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.459269_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.459269Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:10:02.691 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:02.692 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.459269_243.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:02.693 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.459269_243.1 2022-09-13T08:10:07.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:07.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:07.698 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:10:07.774 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:10:07.774 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:10:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:07.894+0000 7f3c85475600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:07.895+0000 7f3c85475600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:07.922 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:10:07.923 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:10:07.923 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:10:07.923 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:10:07.924 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:10:07.924 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:10:07.924 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:07.924 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:07.925 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:10:07.925 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:07.925 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:10:07.926 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:07.926 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:07.926 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:10:07.926 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:07.927 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:10:07.927 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:07.927 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:10:07.927 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:07.928 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:07.928 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:07.928 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:07.928 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:10:07.929 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:07.929 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:07.929 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:10:07.929 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:07.930 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:07.931 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:07.931 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:10:07.931 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:07.931 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:07.932 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:07.932 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:07.932 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:10:07.932 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:07.933 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:07.933 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.142561_227.1", 2022-09-13T08:10:07.933 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:07.933 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:07.934 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:07.934 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T08:10:07.934 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:07.935 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:07.935 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:07.935 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:07.935 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:10:07.936 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:07.936 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:07.937 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:10:07.937 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:07.937 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:07.937 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:07.938 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:10:07.938 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:07.938 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:07.939 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:07.939 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:10:07.939 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:07.939 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:10:07.940 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:10:07.940 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:10:07.940 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:07.940 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:10:07Z" 2022-09-13T08:10:07.941 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:10:07.941 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.142561_227.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.142561Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:10:07Z" } 2022-09-13T08:10:07.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:10:07.944 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:07.944 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:07.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T08:10:12.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:12.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:12.947 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:10:12.998 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:10:12.998 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:10:13.031 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:13.028+0000 7f9ec2d93600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:13.032 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:13.029+0000 7f9ec2d93600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:13.047 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:10:13.047 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:10:13.048 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:10:13.048 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:10:13.048 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:10:13.048 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:10:13.049 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:13.049 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:13.050 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:10:13.051 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:13.051 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:10:13.051 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:13.051 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:13.052 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:10:13.052 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:13.052 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:10:13.053 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:13.053 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:10:13.053 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:13.054 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:13.055 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:13.055 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:13.056 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:10:13.056 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:13.057 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:13.057 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:10:13.057 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:13.057 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:13.058 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:13.058 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:10:13.058 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:13.059 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:13.059 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:13.059 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:13.059 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:10:13.060 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:13.060 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:13.060 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.187352_231.1", 2022-09-13T08:10:13.060 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:13.061 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:13.061 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:13.061 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T08:10:13.061 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:13.062 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:13.062 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:13.062 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:13.063 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:10:13.063 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:13.063 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:13.063 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:10:13.064 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:13.064 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:13.064 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:13.064 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:10:13.065 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:13.065 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:13.065 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:13.065 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:10:13.066 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:13.066 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:10:13.066 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:10:13.066 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:10:13.067 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:13.067 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:10:13Z" 2022-09-13T08:10:13.067 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:10:13.069 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.187352_231.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.187352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:10:13Z" } 2022-09-13T08:10:13.069 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.187352_231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.187352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:10:13.070 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:13.070 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.187352_231.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:13.071 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.187352_231.1 2022-09-13T08:10:18.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:18.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:18.074 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:10:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:10:18.134 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:10:18.172 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:18.168+0000 7f663e527600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:18.173 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:18.169+0000 7f663e527600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:18.191 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:10:18.192 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:10:18.192 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:10:18.193 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:10:18.193 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:10:18.193 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:10:18.194 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:18.194 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:18.194 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:10:18.195 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:18.195 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:10:18.195 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:18.195 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:18.196 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:10:18.196 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:18.196 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:10:18.197 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:18.197 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:10:18.198 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:18.198 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:18.199 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:18.199 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:18.200 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:10:18.200 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:18.201 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:18.201 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:10:18.202 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:18.202 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:18.202 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:18.203 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:10:18.203 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:18.203 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:18.204 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:18.204 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:18.205 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:10:18.205 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:18.205 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:18.206 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:10:18.207 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:18.207 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:10:18.211 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:18.211 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:18.212 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:18.212 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:18.213 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:10:18.214 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:18.214 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:18.215 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:10:18.215 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:18.216 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:18.216 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:18.217 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:10:18.217 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:18.218 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:18.218 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:18.219 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:10:18.219 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:18.220 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:10:18.220 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:10:18.221 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:10:18.221 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:18.223 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:10:18Z" 2022-09-13T08:10:18.223 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:10:18.225 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:10:18Z" } 2022-09-13T08:10:18.226 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:10:18.227 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:18.228 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:18.229 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.141531_226.1 2022-09-13T08:10:23.231 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:23.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:23.233 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:10:23.290 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:10:23.291 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:10:23.326 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:23.322+0000 7fb392415600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:23.327 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:23.323+0000 7fb392415600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:23.343 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:10:23.343 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:10:23.344 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:10:23.344 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:10:23.345 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:10:23.345 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:10:23.345 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:23.346 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:23.347 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:10:23.347 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:23.348 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:10:23.348 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:23.349 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:23.349 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:10:23.350 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:23.350 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:10:23.351 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:23.351 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:10:23.351 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:23.352 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:23.352 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:23.353 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:23.353 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:10:23.354 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:23.354 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:23.354 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:10:23.355 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:23.355 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:23.355 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:23.356 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:10:23.356 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:23.357 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:23.357 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:23.358 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:23.358 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:10:23.359 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:23.359 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:23.359 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.340682_237.1", 2022-09-13T08:10:23.360 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:23.360 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:23.360 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:23.360 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T08:10:23.361 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:23.361 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:23.361 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:23.361 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:23.362 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:10:23.362 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:23.362 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:23.362 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:10:23.363 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:23.363 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:23.363 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:23.363 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:10:23.364 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:23.364 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:23.364 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:23.365 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:10:23.365 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:23.365 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:10:23.365 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:10:23.366 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:10:23.366 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:23.367 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:10:23Z" 2022-09-13T08:10:23.367 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:10:23.368 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.340682_237.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.340682Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:10:23Z" } 2022-09-13T08:10:23.369 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.340682_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.340682Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:10:23.370 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:23.370 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.340682_237.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:23.371 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.340682_237.1 2022-09-13T08:10:28.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:28.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:28.373 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:10:28.424 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:10:28.424 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:10:28.468 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:28.464+0000 7fbf0116e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:28.469 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:28.466+0000 7fbf0116e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:28.492 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:10:28.493 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:10:28.494 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:10:28.494 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:10:28.494 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:10:28.494 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:10:28.495 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:28.495 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:28.496 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:10:28.496 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:28.496 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:10:28.497 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:28.497 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:28.497 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:10:28.497 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:28.498 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:10:28.498 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:28.498 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:10:28.499 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:28.499 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:28.499 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:28.500 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:28.500 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:10:28.500 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:28.501 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:28.501 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:10:28.502 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:28.502 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:28.503 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:28.503 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:10:28.504 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:28.504 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:28.505 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:28.506 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:28.506 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:10:28.506 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:28.507 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:28.507 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.351164_238.1", 2022-09-13T08:10:28.508 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:28.508 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:28.508 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:28.509 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.351164Z", 2022-09-13T08:10:28.509 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:28.509 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:28.510 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:28.510 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:28.511 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:10:28.511 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:28.512 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:28.512 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:10:28.513 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:28.513 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:28.513 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:28.514 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:10:28.514 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:28.515 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:28.515 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:28.515 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:10:28.516 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:28.516 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:10:28.516 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:10:28.517 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:10:28.517 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:28.517 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:10:28Z" 2022-09-13T08:10:28.517 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:10:28.519 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.351164_238.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.351164Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:10:28Z" } 2022-09-13T08:10:28.520 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.351164_238.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.351164Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:10:28.520 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:28.521 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.351164_238.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:28.521 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.351164_238.1 2022-09-13T08:10:33.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:33.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:33.524 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:10:33.576 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:10:33.576 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:10:33.609 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:33.605+0000 7f1b4c8fc600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:33.610 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:33.606+0000 7f1b4c8fc600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:33.625 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:10:33.625 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:10:33.626 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:10:33.626 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:10:33.627 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:10:33.627 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:10:33.628 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:33.628 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:33.629 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:10:33.629 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:33.630 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:10:33.631 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:33.631 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:33.631 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:10:33.631 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:33.632 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:10:33.632 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:33.632 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:10:33.633 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:33.633 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:33.634 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:33.634 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:33.634 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:10:33.635 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:33.635 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:33.635 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:10:33.635 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:33.636 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:33.636 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:33.636 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:10:33.637 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:33.637 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:33.637 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:33.637 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:33.638 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:10:33.638 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:33.638 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:33.639 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.223597_234.1", 2022-09-13T08:10:33.639 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:33.639 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:33.639 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:33.640 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.223597Z", 2022-09-13T08:10:33.640 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:33.640 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:33.641 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:33.641 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:33.642 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:10:33.642 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:33.642 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:33.643 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:10:33.643 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:33.643 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:33.643 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:33.644 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:10:33.644 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:33.644 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:33.644 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:33.645 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:10:33.645 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:33.645 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:10:33.645 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:10:33.646 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:10:33.646 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:33.646 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:10:33Z" 2022-09-13T08:10:33.646 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:10:33.648 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.223597_234.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.223597Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:10:33Z" } 2022-09-13T08:10:33.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.223597_234.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.223597Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:10:33.649 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:33.650 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.223597_234.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:33.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.223597_234.1 2022-09-13T08:10:38.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:38.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:38.653 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:10:38.704 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:10:38.705 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:10:38.742 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:38.738+0000 7ff50c74f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:38.742 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:38.739+0000 7ff50c74f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:38.768 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:10:38.769 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:10:38.769 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:10:38.769 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:10:38.769 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:10:38.770 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:10:38.770 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:38.770 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:38.770 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:10:38.771 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:38.771 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:10:38.771 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:38.772 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:38.772 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:10:38.772 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:38.772 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:10:38.773 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:38.773 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:10:38.773 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:38.773 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:38.774 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:38.774 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:38.774 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:10:38.775 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:38.775 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:38.775 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:10:38.776 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:38.776 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:38.776 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:38.776 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:10:38.777 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:38.778 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:38.778 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:38.779 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:38.779 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:10:38.780 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:38.780 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:38.780 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.220845_233.1", 2022-09-13T08:10:38.780 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:38.781 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:38.781 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:38.782 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T08:10:38.792 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:38.793 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:38.793 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:38.794 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:38.794 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:10:38.794 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:38.795 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:38.795 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:10:38.795 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:38.796 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:38.796 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:38.796 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:10:38.796 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:38.797 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:38.797 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:38.797 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:10:38.797 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:38.798 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:10:38.798 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:10:38.798 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:10:38.798 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:38.799 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:10:38Z" 2022-09-13T08:10:38.799 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:10:38.800 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.220845_233.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.220845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:10:38Z" } 2022-09-13T08:10:38.801 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:10:38.802 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:38.803 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:38.803 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T08:10:43.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:43.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:43.807 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:10:43.864 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:10:43.865 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:10:43.900 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:43.896+0000 7f5681e3c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:43.901 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:43.898+0000 7f5681e3c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:43.918 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:10:43.919 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:10:43.919 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:10:43.919 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:10:43.920 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:10:43.920 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:10:43.920 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:43.920 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:43.921 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:10:43.921 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:43.921 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:10:43.921 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:43.922 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:43.922 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:10:43.922 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:43.922 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:10:43.923 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:43.923 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:10:43.923 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:43.923 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:43.924 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:43.924 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:43.924 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:10:43.924 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:43.925 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:43.925 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:10:43.925 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:43.925 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:43.926 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:43.926 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:10:43.926 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:43.926 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:43.927 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:43.927 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:43.927 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:10:43.928 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:43.928 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:43.928 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055101.622388_255.1", 2022-09-13T08:10:43.929 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:43.929 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:43.929 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:43.929 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:45:01.622388Z", 2022-09-13T08:10:43.929 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:43.930 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:43.930 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:43.931 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:43.931 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:10:43.931 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:43.931 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:43.932 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:10:43.932 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:43.932 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:43.932 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:43.933 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:10:43.933 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:43.933 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:43.933 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:43.934 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:10:43.934 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:43.934 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:10:43.934 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:10:43.935 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:10:43.935 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:43.935 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:10:43Z" 2022-09-13T08:10:43.936 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:10:43.937 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055101.622388_255.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:45:01.622388Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:10:43Z" } 2022-09-13T08:10:43.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055101.622388_255.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:45:01.622388Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:10:43.938 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:43.939 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.622388_255.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:43.939 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055101.622388_255.1 2022-09-13T08:10:48.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:48.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:48.942 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:10:48.993 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:10:48.994 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:10:49.043 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:49.040+0000 7fc3d0cdc600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:49.044 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:49.041+0000 7fc3d0cdc600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:49.067 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:10:49.067 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:10:49.068 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:10:49.069 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:10:49.069 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:10:49.070 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:10:49.070 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:49.071 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:49.071 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:10:49.072 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:49.072 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:10:49.073 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:49.073 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:49.074 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:10:49.074 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:49.075 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:10:49.075 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:49.075 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:10:49.078 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:49.079 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:49.079 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:49.079 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:49.080 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:10:49.080 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:49.081 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:49.081 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:10:49.081 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:49.082 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:49.082 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:49.083 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:10:49.083 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:49.083 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:49.083 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:49.084 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:49.084 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:10:49.084 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:49.084 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:49.085 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.223597_234.1", 2022-09-13T08:10:49.085 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:49.085 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:49.085 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:49.086 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.223597Z", 2022-09-13T08:10:49.086 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:49.086 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:49.086 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:49.087 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:49.087 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:10:49.087 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:49.087 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:49.088 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:10:49.088 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:49.088 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:49.088 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:49.089 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:10:49.089 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:49.089 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:49.089 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:49.090 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:10:49.090 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:49.090 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:10:49.090 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:10:49.091 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:10:49.091 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:49.091 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:10:49Z" 2022-09-13T08:10:49.091 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:10:49.092 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.223597_234.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.223597Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:10:49Z" } 2022-09-13T08:10:49.092 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.223597_234.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.223597Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:10:49.093 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:49.093 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.223597_234.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:49.094 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.223597_234.1 2022-09-13T08:10:54.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:54.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:54.097 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:10:54.153 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:10:54.153 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:10:54.204 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:54.201+0000 7f050b5b2600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:54.205 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:54.202+0000 7f050b5b2600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:54.219 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:10:54.220 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:10:54.220 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:10:54.221 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:10:54.221 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:10:54.222 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:10:54.222 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:54.222 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:54.223 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:10:54.223 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:54.224 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:10:54.224 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:54.224 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:54.225 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:10:54.225 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:54.226 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:10:54.226 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:54.226 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:10:54.227 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:54.227 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:54.227 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:54.228 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:54.228 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:10:54.229 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:54.229 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:54.230 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:10:54.231 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:54.231 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:54.232 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:54.232 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:10:54.233 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:54.233 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:54.234 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:54.234 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:54.235 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:10:54.235 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:54.236 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:54.236 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.220845_233.1", 2022-09-13T08:10:54.237 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:54.237 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:54.238 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:54.238 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T08:10:54.239 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:54.239 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:54.240 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:54.240 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:54.241 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:10:54.241 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:54.242 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:54.242 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:10:54.243 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:54.243 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:54.243 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:54.244 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:10:54.244 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:54.245 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:54.245 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:54.246 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:10:54.246 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:54.246 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:10:54.247 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:10:54.247 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:10:54.248 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:54.248 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:10:54Z" 2022-09-13T08:10:54.249 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:10:54.250 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.220845_233.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.220845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:10:54Z" } 2022-09-13T08:10:54.251 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:10:54.251 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:54.252 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:54.252 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T08:10:59.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:59.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:10:59.256 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:10:59.314 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:10:59.315 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:10:59.361 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:59.357+0000 7f9f94718600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:59.362 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:10:59.358+0000 7f9f94718600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:10:59.384 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:10:59.385 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:10:59.385 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:10:59.386 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:10:59.386 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:10:59.386 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:10:59.387 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:59.387 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:59.388 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:10:59.388 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:59.388 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:10:59.389 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:59.389 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:59.390 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:10:59.390 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:59.393 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:10:59.395 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:59.395 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:10:59.396 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:59.396 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:59.396 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:59.397 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:59.397 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:10:59.397 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:59.399 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:59.399 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:10:59.399 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:59.400 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:59.400 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:59.400 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:10:59.400 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:59.401 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:59.401 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:59.401 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:59.410 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:10:59.411 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:59.411 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:59.412 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.340682_237.1", 2022-09-13T08:10:59.412 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:59.413 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:59.413 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:59.414 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T08:10:59.415 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:59.415 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:59.415 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:59.416 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:10:59.416 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:10:59.417 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:10:59.418 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:10:59.418 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:10:59.419 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:10:59.419 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:10:59.420 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:10:59.420 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:10:59.421 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:10:59.421 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:59.422 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:10:59.422 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:10:59.423 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:59.424 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:10:59.424 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:10:59.425 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:10:59.425 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:10:59.426 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:10:59Z" 2022-09-13T08:10:59.426 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:10:59.429 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.340682_237.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.340682Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:10:59Z" } 2022-09-13T08:10:59.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.340682_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.340682Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:10:59.431 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:59.431 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.340682_237.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:10:59.432 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.340682_237.1 2022-09-13T08:11:04.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:04.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:04.434 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:11:04.538 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:04.539 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:04.574 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:04.571+0000 7f06496b5600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:04.575 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:04.572+0000 7f06496b5600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:04.591 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:11:04.591 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:11:04.592 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:11:04.592 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:11:04.592 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:11:04.592 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:11:04.593 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:04.593 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:04.593 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:11:04.594 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:04.594 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:11:04.594 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:04.595 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:04.595 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:11:04.595 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:04.596 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:11:04.596 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:04.596 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:11:04.596 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:04.597 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:04.597 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:04.597 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:04.598 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:11:04.598 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:04.599 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:04.599 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:11:04.599 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:04.599 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:04.600 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:04.601 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:11:04.601 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:04.602 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:04.602 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:04.603 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:04.603 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:11:04.603 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:04.604 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:04.604 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.220845_233.1", 2022-09-13T08:11:04.604 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:04.605 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:04.605 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:04.606 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T08:11:04.606 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:04.606 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:04.606 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:04.607 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:04.607 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:11:04.607 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:04.607 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:04.608 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:11:04.608 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:04.608 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:04.608 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:04.609 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:11:04.609 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:04.609 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:04.610 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:04.610 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:11:04.610 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:04.610 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:11:04.611 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:11:04.611 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:11:04.611 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:04.611 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:11:04Z" 2022-09-13T08:11:04.612 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:11:04.614 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.220845_233.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.220845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:11:04Z" } 2022-09-13T08:11:04.615 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:11:04.615 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:04.616 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:04.616 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T08:11:09.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:09.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:09.621 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:11:09.674 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:09.675 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:09.723 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:09.720+0000 7f3d2ce02600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:09.725 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:09.721+0000 7f3d2ce02600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:09.742 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:11:09.742 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:11:09.743 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:11:09.743 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:11:09.743 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:11:09.743 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:11:09.744 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:09.744 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:09.744 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:11:09.745 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:09.745 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:11:09.745 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:09.745 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:09.746 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:11:09.746 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:09.747 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:11:09.747 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:09.748 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:11:09.748 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:09.748 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:09.748 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:09.749 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:09.749 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:11:09.749 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:09.750 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:09.750 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:11:09.750 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:09.750 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:09.751 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:09.751 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:11:09.751 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:09.751 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:09.752 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:09.752 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:09.752 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:11:09.752 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:09.753 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:09.753 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055100.589473_253.1", 2022-09-13T08:11:09.753 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:09.754 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:09.754 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:09.755 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:45:00.589473Z", 2022-09-13T08:11:09.755 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:09.755 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:09.756 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:09.757 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:09.758 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:11:09.758 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:09.759 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:09.759 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:11:09.760 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:09.760 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:09.761 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:09.761 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:11:09.762 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:09.762 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:09.763 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:09.763 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:11:09.763 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:09.764 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:11:09.765 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:11:09.765 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:11:09.766 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:09.766 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:11:09Z" 2022-09-13T08:11:09.767 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:11:09.770 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055100.589473_253.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:45:00.589473Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:11:09Z" } 2022-09-13T08:11:09.771 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055100.589473_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:45:00.589473Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:11:09.772 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:09.773 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055100.589473_253.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:09.774 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055100.589473_253.1 2022-09-13T08:11:14.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:14.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:14.777 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:11:14.835 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:14.836 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:14.874 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:14.871+0000 7fc3ecf6a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:14.875 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:14.872+0000 7fc3ecf6a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:14.891 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:11:14.892 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:11:14.892 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:11:14.892 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:11:14.893 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:11:14.893 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:11:14.893 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:14.894 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:14.894 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:11:14.894 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:14.894 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:11:14.895 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:14.895 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:14.895 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:11:14.896 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:14.896 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:11:14.896 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:14.897 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:11:14.897 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:14.898 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:14.898 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:14.898 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:14.899 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:11:14.899 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:14.900 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:14.900 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:11:14.900 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:14.901 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:14.901 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:14.902 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:11:14.902 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:14.902 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:14.903 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:14.903 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:14.903 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:11:14.903 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:14.904 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:14.904 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.223597_234.1", 2022-09-13T08:11:14.904 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:14.904 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:14.905 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:14.905 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.223597Z", 2022-09-13T08:11:14.905 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:14.905 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:14.906 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:14.906 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:14.906 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:11:14.907 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:14.907 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:14.908 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:11:14.908 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:14.909 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:14.909 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:14.910 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:11:14.910 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:14.911 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:14.911 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:14.912 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:11:14.912 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:14.913 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:11:14.913 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:11:14.914 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:11:14.914 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:14.915 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:11:14Z" 2022-09-13T08:11:14.915 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:11:14.917 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.223597_234.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.223597Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:11:14Z" } 2022-09-13T08:11:14.918 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.223597_234.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.223597Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:11:14.919 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:14.920 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.223597_234.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:14.921 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.223597_234.1 2022-09-13T08:11:19.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:19.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:19.923 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:11:19.980 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:19.980 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:20.015 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:20.011+0000 7f2f552e7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:20.016 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:20.013+0000 7f2f552e7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:20.032 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:11:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:11:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:11:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:11:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:11:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:11:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:11:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:11:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:11:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:11:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:11:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:11:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:11:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:11:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:20.045 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:20.045 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:20.046 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:20.046 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:11:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.220845_233.1", 2022-09-13T08:11:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T08:11:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:11:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:11:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:11:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:11:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:11:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:11:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:11:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:11:20Z" 2022-09-13T08:11:20.056 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:11:20.059 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.220845_233.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.220845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:11:20Z" } 2022-09-13T08:11:20.060 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:11:20.061 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:20.061 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:20.062 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T08:11:25.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:25.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:25.065 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:11:25.122 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:25.123 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:25.242 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:25.238+0000 7f319537f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:25.243 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:25.239+0000 7f319537f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:25.287 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:11:25.287 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:11:25.288 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:11:25.288 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:11:25.288 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:11:25.288 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:11:25.289 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:25.289 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:25.289 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:11:25.289 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:25.290 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:11:25.290 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:25.290 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:25.290 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:11:25.291 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:25.291 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:11:25.291 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:25.291 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:11:25.292 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:25.292 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:25.292 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:25.292 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:25.293 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:11:25.293 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:25.293 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:25.293 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:11:25.294 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:25.294 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:25.294 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:25.294 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:11:25.295 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:25.295 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:25.295 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:25.295 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:25.296 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:11:25.296 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:25.296 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:25.296 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.182451_229.1", 2022-09-13T08:11:25.297 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:25.297 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:25.297 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:25.297 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T08:11:25.297 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:25.298 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:25.298 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:25.298 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:25.298 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:11:25.299 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:25.299 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:25.299 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:11:25.299 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:25.300 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:25.300 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:25.300 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:11:25.300 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:25.301 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:25.301 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:25.301 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:11:25.301 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:25.302 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:11:25.302 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:11:25.302 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:11:25.302 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:25.303 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:11:25Z" 2022-09-13T08:11:25.303 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:11:25.304 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.182451_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.182451Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:11:25Z" } 2022-09-13T08:11:25.305 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.182451_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.182451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:11:25.305 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:25.306 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.182451_229.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:25.306 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.182451_229.1 2022-09-13T08:11:30.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:30.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:30.309 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:11:30.358 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:30.359 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:30.389 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:30.386+0000 7f8767004600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:30.390 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:30.387+0000 7f8767004600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:30.409 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:11:30.409 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:11:30.409 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:11:30.410 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:11:30.410 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:11:30.410 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:11:30.411 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:30.411 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:30.411 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:11:30.411 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:30.412 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:11:30.412 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:30.412 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:30.412 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:11:30.413 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:30.413 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:11:30.413 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:30.414 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:11:30.414 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:30.414 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:30.414 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:30.415 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:30.415 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:11:30.415 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:30.415 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:30.416 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:11:30.416 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:30.416 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:30.416 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:30.417 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:11:30.417 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:30.417 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:30.418 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:30.418 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:30.418 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:11:30.418 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:30.419 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:30.419 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.220845_233.1", 2022-09-13T08:11:30.419 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:30.420 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:30.420 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:30.420 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T08:11:30.420 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:30.421 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:30.421 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:30.421 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:30.421 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:11:30.422 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:30.422 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:30.422 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:11:30.423 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:30.423 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:30.423 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:30.424 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:11:30.424 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:30.424 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:30.424 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:30.425 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:11:30.425 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:30.425 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:11:30.425 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:11:30.426 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:11:30.426 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:30.426 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:11:30Z" 2022-09-13T08:11:30.426 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:11:30.428 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.220845_233.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.220845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:11:30Z" } 2022-09-13T08:11:30.428 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:11:30.429 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:30.430 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:30.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T08:11:35.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:35.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:35.434 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:11:35.495 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:35.496 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:35.553 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:35.550+0000 7f0b49dd5600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:35.554 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:35.551+0000 7f0b49dd5600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:35.576 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:11:35.576 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:11:35.577 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:11:35.577 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:11:35.577 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:11:35.577 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:11:35.578 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:35.578 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:35.578 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:11:35.579 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:35.579 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:11:35.580 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:35.581 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:35.582 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:11:35.582 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:35.582 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:11:35.583 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:35.583 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:11:35.583 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:35.584 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:35.584 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:35.584 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:35.585 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:11:35.585 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:35.586 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:35.586 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:11:35.586 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:35.587 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:35.587 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:35.587 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:11:35.587 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:35.587 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:35.588 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:35.588 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:35.588 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:11:35.589 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:35.589 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:35.590 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.468110_246.1", 2022-09-13T08:11:35.590 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:35.590 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:35.591 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:35.591 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.468110Z", 2022-09-13T08:11:35.591 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:35.591 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:35.592 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:35.592 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:35.592 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:11:35.592 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:35.593 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:35.593 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:11:35.593 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:35.593 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:35.594 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:35.594 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:11:35.594 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:35.594 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:35.595 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:35.595 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:11:35.595 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:35.595 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:11:35.596 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:11:35.596 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:11:35.597 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:35.597 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:11:35Z" 2022-09-13T08:11:35.597 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:11:35.599 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.468110_246.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.468110Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:11:35Z" } 2022-09-13T08:11:35.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.468110_246.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.468110Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:11:35.600 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:35.600 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.468110_246.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:35.601 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.468110_246.1 2022-09-13T08:11:40.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:40.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:40.605 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:11:40.663 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:40.664 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:40.708 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:40.704+0000 7fdef4a10600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:40.709 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:40.705+0000 7fdef4a10600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:40.728 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:11:40.728 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:11:40.730 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:11:40.731 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:11:40.731 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:11:40.732 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:11:40.732 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:40.733 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:40.733 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:11:40.733 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:40.735 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:11:40.735 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:40.735 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:40.736 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:11:40.736 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:40.736 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:11:40.736 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:40.737 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:11:40.737 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:40.737 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:40.738 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:40.738 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:40.739 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:11:40.739 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:40.739 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:40.740 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:11:40.741 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:40.741 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:40.741 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:40.742 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:11:40.742 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:40.742 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:40.743 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:40.743 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:40.743 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:11:40.744 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:40.744 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:40.745 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.220845_233.1", 2022-09-13T08:11:40.745 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:40.745 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:40.745 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:40.746 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T08:11:40.746 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:40.746 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:40.747 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:40.747 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:40.747 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:11:40.747 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:40.748 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:40.748 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:11:40.748 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:40.749 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:40.749 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:40.749 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:11:40.750 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:40.751 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:40.752 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:40.752 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:11:40.753 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:40.753 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:11:40.753 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:11:40.754 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:11:40.754 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:40.755 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:11:40Z" 2022-09-13T08:11:40.755 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:11:40.757 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.220845_233.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.220845Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:11:40Z" } 2022-09-13T08:11:40.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:11:40.760 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:40.760 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:40.761 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T08:11:45.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:45.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:45.765 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:11:45.871 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:45.871 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:45.914 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:45.910+0000 7f481fdb7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:45.914 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:45.911+0000 7f481fdb7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:45.929 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:11:45.929 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:11:45.930 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:11:45.930 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:11:45.930 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:11:45.931 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:11:45.931 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:45.931 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:45.931 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:11:45.932 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:45.932 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:11:45.932 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:45.932 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:45.933 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:11:45.933 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:45.933 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:11:45.933 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:45.934 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:11:45.934 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:45.934 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:45.935 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:45.935 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:45.935 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:11:45.935 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:45.936 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:45.936 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:11:45.936 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:45.937 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:45.937 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:45.937 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:11:45.937 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:45.937 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:45.938 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:45.938 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:45.938 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:11:45.939 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:45.939 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:45.939 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.189736_232.1", 2022-09-13T08:11:45.940 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:45.940 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:45.940 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:45.940 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.189736Z", 2022-09-13T08:11:45.941 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:45.941 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:45.941 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:45.941 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:45.942 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:11:45.942 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:45.942 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:45.942 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:11:45.943 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:45.943 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:45.943 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:45.944 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:11:45.944 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:45.944 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:45.944 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:45.945 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:11:45.945 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:45.945 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:11:45.945 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:11:45.946 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:11:45.946 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:45.946 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:11:45Z" 2022-09-13T08:11:45.946 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:11:45.948 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.189736_232.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.189736Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:11:45Z" } 2022-09-13T08:11:45.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.189736_232.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.189736Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:11:45.949 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:45.949 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.189736_232.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:45.950 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.189736_232.1 2022-09-13T08:11:50.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... FAIL 2022-09-13T08:11:50.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:11:50.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:11:50.962 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T08:11:51.018 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:51.019 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:51.049 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T08:11:51.050 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "93648cda-eefa-4864-b5ed-3049bf1995e0", 2022-09-13T08:11:51.050 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "remove_zg", 2022-09-13T08:11:51.051 INFO:teuthology.orchestra.run.smithi036.stdout: "api_name": "remove_zg", 2022-09-13T08:11:51.051 INFO:teuthology.orchestra.run.smithi036.stdout: "is_master": false, 2022-09-13T08:11:51.052 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [], 2022-09-13T08:11:51.052 INFO:teuthology.orchestra.run.smithi036.stdout: "hostnames": [], 2022-09-13T08:11:51.053 INFO:teuthology.orchestra.run.smithi036.stdout: "hostnames_s3website": [], 2022-09-13T08:11:51.053 INFO:teuthology.orchestra.run.smithi036.stdout: "master_zone": "", 2022-09-13T08:11:51.054 INFO:teuthology.orchestra.run.smithi036.stdout: "zones": [], 2022-09-13T08:11:51.054 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_targets": [], 2022-09-13T08:11:51.055 INFO:teuthology.orchestra.run.smithi036.stdout: "default_placement": "", 2022-09-13T08:11:51.055 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T08:11:51.056 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_policy": { 2022-09-13T08:11:51.056 INFO:teuthology.orchestra.run.smithi036.stdout: "groups": [] 2022-09-13T08:11:51.057 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.057 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled_features": [ 2022-09-13T08:11:51.058 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T08:11:51.058 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.059 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T08:11:51.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:11:51.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:11:51.063 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T08:11:51.115 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:51.116 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:51.144 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T08:11:51.145 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "02818dd9-8459-4e80-9f15-163a5481b3d0", 2022-09-13T08:11:51.145 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "remove", 2022-09-13T08:11:51.146 INFO:teuthology.orchestra.run.smithi036.stdout: "domain_root": "remove.rgw.meta:root", 2022-09-13T08:11:51.146 INFO:teuthology.orchestra.run.smithi036.stdout: "control_pool": "remove.rgw.control", 2022-09-13T08:11:51.147 INFO:teuthology.orchestra.run.smithi036.stdout: "gc_pool": "remove.rgw.log:gc", 2022-09-13T08:11:51.147 INFO:teuthology.orchestra.run.smithi036.stdout: "lc_pool": "remove.rgw.log:lc", 2022-09-13T08:11:51.148 INFO:teuthology.orchestra.run.smithi036.stdout: "log_pool": "remove.rgw.log", 2022-09-13T08:11:51.148 INFO:teuthology.orchestra.run.smithi036.stdout: "intent_log_pool": "remove.rgw.log:intent", 2022-09-13T08:11:51.149 INFO:teuthology.orchestra.run.smithi036.stdout: "usage_log_pool": "remove.rgw.log:usage", 2022-09-13T08:11:51.149 INFO:teuthology.orchestra.run.smithi036.stdout: "roles_pool": "remove.rgw.meta:roles", 2022-09-13T08:11:51.150 INFO:teuthology.orchestra.run.smithi036.stdout: "reshard_pool": "remove.rgw.log:reshard", 2022-09-13T08:11:51.150 INFO:teuthology.orchestra.run.smithi036.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2022-09-13T08:11:51.150 INFO:teuthology.orchestra.run.smithi036.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2022-09-13T08:11:51.151 INFO:teuthology.orchestra.run.smithi036.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2022-09-13T08:11:51.151 INFO:teuthology.orchestra.run.smithi036.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2022-09-13T08:11:51.152 INFO:teuthology.orchestra.run.smithi036.stdout: "otp_pool": "remove.rgw.otp", 2022-09-13T08:11:51.152 INFO:teuthology.orchestra.run.smithi036.stdout: "system_key": { 2022-09-13T08:11:51.153 INFO:teuthology.orchestra.run.smithi036.stdout: "access_key": "", 2022-09-13T08:11:51.153 INFO:teuthology.orchestra.run.smithi036.stdout: "secret_key": "" 2022-09-13T08:11:51.154 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.154 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_pools": [ 2022-09-13T08:11:51.155 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.155 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "default-placement", 2022-09-13T08:11:51.156 INFO:teuthology.orchestra.run.smithi036.stdout: "val": { 2022-09-13T08:11:51.156 INFO:teuthology.orchestra.run.smithi036.stdout: "index_pool": "remove.rgw.buckets.index", 2022-09-13T08:11:51.157 INFO:teuthology.orchestra.run.smithi036.stdout: "storage_classes": { 2022-09-13T08:11:51.157 INFO:teuthology.orchestra.run.smithi036.stdout: "STANDARD": { 2022-09-13T08:11:51.157 INFO:teuthology.orchestra.run.smithi036.stdout: "data_pool": "remove.rgw.buckets.data" 2022-09-13T08:11:51.157 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:51.158 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.158 INFO:teuthology.orchestra.run.smithi036.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2022-09-13T08:11:51.158 INFO:teuthology.orchestra.run.smithi036.stdout: "index_type": 0 2022-09-13T08:11:51.159 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:51.159 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:51.160 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.160 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T08:11:51.161 INFO:teuthology.orchestra.run.smithi036.stdout: "notif_pool": "remove.rgw.log:notif" 2022-09-13T08:11:51.161 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T08:11:51.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:11:51.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:11:51.166 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T08:11:51.218 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:51.219 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:51.273 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T08:11:51.274 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:11:51.274 INFO:teuthology.orchestra.run.smithi036.stdout: "epoch": 5, 2022-09-13T08:11:51.275 INFO:teuthology.orchestra.run.smithi036.stdout: "predecessor_uuid": "0f1e9404-34a5-47ad-8196-42a4e6e2ff9c", 2022-09-13T08:11:51.275 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": [], 2022-09-13T08:11:51.276 INFO:teuthology.orchestra.run.smithi036.stdout: "period_map": { 2022-09-13T08:11:51.276 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:11:51.276 INFO:teuthology.orchestra.run.smithi036.stdout: "zonegroups": [ 2022-09-13T08:11:51.277 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.277 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "93648cda-eefa-4864-b5ed-3049bf1995e0", 2022-09-13T08:11:51.277 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "remove_zg", 2022-09-13T08:11:51.277 INFO:teuthology.orchestra.run.smithi036.stdout: "api_name": "remove_zg", 2022-09-13T08:11:51.278 INFO:teuthology.orchestra.run.smithi036.stdout: "is_master": false, 2022-09-13T08:11:51.278 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [], 2022-09-13T08:11:51.278 INFO:teuthology.orchestra.run.smithi036.stdout: "hostnames": [], 2022-09-13T08:11:51.279 INFO:teuthology.orchestra.run.smithi036.stdout: "hostnames_s3website": [], 2022-09-13T08:11:51.279 INFO:teuthology.orchestra.run.smithi036.stdout: "master_zone": "02818dd9-8459-4e80-9f15-163a5481b3d0", 2022-09-13T08:11:51.279 INFO:teuthology.orchestra.run.smithi036.stdout: "zones": [ 2022-09-13T08:11:51.279 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.280 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "02818dd9-8459-4e80-9f15-163a5481b3d0", 2022-09-13T08:11:51.280 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "remove", 2022-09-13T08:11:51.280 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [], 2022-09-13T08:11:51.281 INFO:teuthology.orchestra.run.smithi036.stdout: "log_meta": false, 2022-09-13T08:11:51.281 INFO:teuthology.orchestra.run.smithi036.stdout: "log_data": false, 2022-09-13T08:11:51.281 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_index_max_shards": 11, 2022-09-13T08:11:51.281 INFO:teuthology.orchestra.run.smithi036.stdout: "read_only": false, 2022-09-13T08:11:51.282 INFO:teuthology.orchestra.run.smithi036.stdout: "tier_type": "", 2022-09-13T08:11:51.282 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from_all": true, 2022-09-13T08:11:51.282 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from": [], 2022-09-13T08:11:51.283 INFO:teuthology.orchestra.run.smithi036.stdout: "redirect_zone": "", 2022-09-13T08:11:51.283 INFO:teuthology.orchestra.run.smithi036.stdout: "supported_features": [ 2022-09-13T08:11:51.283 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T08:11:51.283 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.284 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:51.284 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.284 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_targets": [ 2022-09-13T08:11:51.285 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.285 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "default-placement", 2022-09-13T08:11:51.285 INFO:teuthology.orchestra.run.smithi036.stdout: "tags": [], 2022-09-13T08:11:51.286 INFO:teuthology.orchestra.run.smithi036.stdout: "storage_classes": [ 2022-09-13T08:11:51.286 INFO:teuthology.orchestra.run.smithi036.stdout: "STANDARD" 2022-09-13T08:11:51.286 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.287 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:51.287 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.287 INFO:teuthology.orchestra.run.smithi036.stdout: "default_placement": "default-placement", 2022-09-13T08:11:51.287 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T08:11:51.288 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_policy": { 2022-09-13T08:11:51.288 INFO:teuthology.orchestra.run.smithi036.stdout: "groups": [] 2022-09-13T08:11:51.288 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.288 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled_features": [ 2022-09-13T08:11:51.289 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T08:11:51.289 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.289 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.290 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.290 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "test-zonegroup", 2022-09-13T08:11:51.290 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zonegroup", 2022-09-13T08:11:51.291 INFO:teuthology.orchestra.run.smithi036.stdout: "api_name": "", 2022-09-13T08:11:51.291 INFO:teuthology.orchestra.run.smithi036.stdout: "is_master": true, 2022-09-13T08:11:51.291 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [ 2022-09-13T08:11:51.292 INFO:teuthology.orchestra.run.smithi036.stdout: "http://smithi036.front.sepia.ceph.com:8000" 2022-09-13T08:11:51.292 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.292 INFO:teuthology.orchestra.run.smithi036.stdout: "hostnames": [], 2022-09-13T08:11:51.292 INFO:teuthology.orchestra.run.smithi036.stdout: "hostnames_s3website": [], 2022-09-13T08:11:51.293 INFO:teuthology.orchestra.run.smithi036.stdout: "master_zone": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T08:11:51.293 INFO:teuthology.orchestra.run.smithi036.stdout: "zones": [ 2022-09-13T08:11:51.293 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.294 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T08:11:51.294 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zone2", 2022-09-13T08:11:51.294 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [ 2022-09-13T08:11:51.295 INFO:teuthology.orchestra.run.smithi036.stdout: "http://smithi083.front.sepia.ceph.com:8000" 2022-09-13T08:11:51.295 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.295 INFO:teuthology.orchestra.run.smithi036.stdout: "log_meta": false, 2022-09-13T08:11:51.295 INFO:teuthology.orchestra.run.smithi036.stdout: "log_data": true, 2022-09-13T08:11:51.296 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_index_max_shards": 11, 2022-09-13T08:11:51.296 INFO:teuthology.orchestra.run.smithi036.stdout: "read_only": false, 2022-09-13T08:11:51.296 INFO:teuthology.orchestra.run.smithi036.stdout: "tier_type": "", 2022-09-13T08:11:51.297 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from_all": false, 2022-09-13T08:11:51.297 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from": [], 2022-09-13T08:11:51.297 INFO:teuthology.orchestra.run.smithi036.stdout: "redirect_zone": "", 2022-09-13T08:11:51.297 INFO:teuthology.orchestra.run.smithi036.stdout: "supported_features": [ 2022-09-13T08:11:51.298 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T08:11:51.298 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.298 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.299 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.299 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9", 2022-09-13T08:11:51.299 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zone3", 2022-09-13T08:11:51.300 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [ 2022-09-13T08:11:51.300 INFO:teuthology.orchestra.run.smithi036.stdout: "http://smithi036.front.sepia.ceph.com:8001" 2022-09-13T08:11:51.300 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.301 INFO:teuthology.orchestra.run.smithi036.stdout: "log_meta": false, 2022-09-13T08:11:51.301 INFO:teuthology.orchestra.run.smithi036.stdout: "log_data": true, 2022-09-13T08:11:51.301 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_index_max_shards": 11, 2022-09-13T08:11:51.301 INFO:teuthology.orchestra.run.smithi036.stdout: "read_only": false, 2022-09-13T08:11:51.302 INFO:teuthology.orchestra.run.smithi036.stdout: "tier_type": "", 2022-09-13T08:11:51.302 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from_all": true, 2022-09-13T08:11:51.302 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from": [], 2022-09-13T08:11:51.302 INFO:teuthology.orchestra.run.smithi036.stdout: "redirect_zone": "", 2022-09-13T08:11:51.303 INFO:teuthology.orchestra.run.smithi036.stdout: "supported_features": [ 2022-09-13T08:11:51.303 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T08:11:51.303 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.304 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.304 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.304 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T08:11:51.304 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zone1", 2022-09-13T08:11:51.306 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [ 2022-09-13T08:11:51.306 INFO:teuthology.orchestra.run.smithi036.stdout: "http://smithi036.front.sepia.ceph.com:8000" 2022-09-13T08:11:51.306 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.307 INFO:teuthology.orchestra.run.smithi036.stdout: "log_meta": false, 2022-09-13T08:11:51.307 INFO:teuthology.orchestra.run.smithi036.stdout: "log_data": true, 2022-09-13T08:11:51.307 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_index_max_shards": 11, 2022-09-13T08:11:51.307 INFO:teuthology.orchestra.run.smithi036.stdout: "read_only": false, 2022-09-13T08:11:51.308 INFO:teuthology.orchestra.run.smithi036.stdout: "tier_type": "", 2022-09-13T08:11:51.308 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from_all": true, 2022-09-13T08:11:51.308 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from": [], 2022-09-13T08:11:51.308 INFO:teuthology.orchestra.run.smithi036.stdout: "redirect_zone": "", 2022-09-13T08:11:51.309 INFO:teuthology.orchestra.run.smithi036.stdout: "supported_features": [ 2022-09-13T08:11:51.309 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T08:11:51.309 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.309 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:51.310 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.310 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_targets": [ 2022-09-13T08:11:51.310 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.310 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "default-placement", 2022-09-13T08:11:51.311 INFO:teuthology.orchestra.run.smithi036.stdout: "tags": [], 2022-09-13T08:11:51.311 INFO:teuthology.orchestra.run.smithi036.stdout: "storage_classes": [ 2022-09-13T08:11:51.311 INFO:teuthology.orchestra.run.smithi036.stdout: "STANDARD" 2022-09-13T08:11:51.312 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.312 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:51.312 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.313 INFO:teuthology.orchestra.run.smithi036.stdout: "default_placement": "default-placement", 2022-09-13T08:11:51.313 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T08:11:51.313 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_policy": { 2022-09-13T08:11:51.313 INFO:teuthology.orchestra.run.smithi036.stdout: "groups": [] 2022-09-13T08:11:51.314 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.314 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled_features": [ 2022-09-13T08:11:51.314 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T08:11:51.315 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.315 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:51.315 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.315 INFO:teuthology.orchestra.run.smithi036.stdout: "short_zone_ids": [ 2022-09-13T08:11:51.316 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.316 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "02818dd9-8459-4e80-9f15-163a5481b3d0", 2022-09-13T08:11:51.316 INFO:teuthology.orchestra.run.smithi036.stdout: "val": 4205716388 2022-09-13T08:11:51.317 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.317 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.317 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T08:11:51.318 INFO:teuthology.orchestra.run.smithi036.stdout: "val": 2827250327 2022-09-13T08:11:51.318 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.318 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.319 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9", 2022-09-13T08:11:51.319 INFO:teuthology.orchestra.run.smithi036.stdout: "val": 1723930414 2022-09-13T08:11:51.319 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.320 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.320 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T08:11:51.320 INFO:teuthology.orchestra.run.smithi036.stdout: "val": 2837352077 2022-09-13T08:11:51.321 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:51.321 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.322 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.322 INFO:teuthology.orchestra.run.smithi036.stdout: "master_zonegroup": "test-zonegroup", 2022-09-13T08:11:51.322 INFO:teuthology.orchestra.run.smithi036.stdout: "master_zone": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T08:11:51.322 INFO:teuthology.orchestra.run.smithi036.stdout: "period_config": { 2022-09-13T08:11:51.323 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_quota": { 2022-09-13T08:11:51.323 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false, 2022-09-13T08:11:51.323 INFO:teuthology.orchestra.run.smithi036.stdout: "check_on_raw": false, 2022-09-13T08:11:51.324 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size": -1, 2022-09-13T08:11:51.324 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size_kb": 0, 2022-09-13T08:11:51.324 INFO:teuthology.orchestra.run.smithi036.stdout: "max_objects": -1 2022-09-13T08:11:51.325 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.325 INFO:teuthology.orchestra.run.smithi036.stdout: "user_quota": { 2022-09-13T08:11:51.325 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false, 2022-09-13T08:11:51.326 INFO:teuthology.orchestra.run.smithi036.stdout: "check_on_raw": false, 2022-09-13T08:11:51.326 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size": -1, 2022-09-13T08:11:51.326 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size_kb": 0, 2022-09-13T08:11:51.326 INFO:teuthology.orchestra.run.smithi036.stdout: "max_objects": -1 2022-09-13T08:11:51.327 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.327 INFO:teuthology.orchestra.run.smithi036.stdout: "user_ratelimit": { 2022-09-13T08:11:51.327 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_ops": 0, 2022-09-13T08:11:51.328 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_ops": 0, 2022-09-13T08:11:51.328 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_bytes": 0, 2022-09-13T08:11:51.328 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_bytes": 0, 2022-09-13T08:11:51.328 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false 2022-09-13T08:11:51.329 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.329 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_ratelimit": { 2022-09-13T08:11:51.329 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_ops": 0, 2022-09-13T08:11:51.330 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_ops": 0, 2022-09-13T08:11:51.330 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_bytes": 0, 2022-09-13T08:11:51.330 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_bytes": 0, 2022-09-13T08:11:51.331 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false 2022-09-13T08:11:51.331 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.331 INFO:teuthology.orchestra.run.smithi036.stdout: "anonymous_ratelimit": { 2022-09-13T08:11:51.332 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_ops": 0, 2022-09-13T08:11:51.332 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_ops": 0, 2022-09-13T08:11:51.332 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_bytes": 0, 2022-09-13T08:11:51.332 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_bytes": 0, 2022-09-13T08:11:51.333 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false 2022-09-13T08:11:51.333 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:51.333 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.333 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T08:11:51.334 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_name": "test-realm", 2022-09-13T08:11:51.334 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T08:11:51.334 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T08:11:51.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:11:51.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:11:51.336 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T08:11:51.386 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:51.387 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:51.424 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T08:11:51.420+0000 7f6904386600 0 zone id 02818dd9-8459-4e80-9f15-163a5481b3d0 is not a part of zonegroup test-zonegroup 2022-09-13T08:11:51.424 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T08:11:51.420+0000 7f6904386600 0 zone id 02818dd9-8459-4e80-9f15-163a5481b3d0 is not a part of zonegroup default 2022-09-13T08:11:51.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:11:51.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:11:51.433 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T08:11:51.494 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:51.495 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:51.526 INFO:teuthology.orchestra.run.smithi036.stderr:2022-09-13T08:11:51.522+0000 7f151db42600 0 ERROR: zonegroup remove_zg has a non existent master zone 2022-09-13T08:11:51.536 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-09-13T08:11:51.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:11:51.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:11:51.540 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T08:11:51.599 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:51.600 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:51.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:11:51.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T08:11:51.674 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-13T08:11:51.743 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:51.744 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:51.858 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2022-09-13T08:11:51.859 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:11:51.860 INFO:teuthology.orchestra.run.smithi036.stdout: "epoch": 6, 2022-09-13T08:11:51.860 INFO:teuthology.orchestra.run.smithi036.stdout: "predecessor_uuid": "0f1e9404-34a5-47ad-8196-42a4e6e2ff9c", 2022-09-13T08:11:51.861 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_status": [], 2022-09-13T08:11:51.861 INFO:teuthology.orchestra.run.smithi036.stdout: "period_map": { 2022-09-13T08:11:51.862 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:11:51.862 INFO:teuthology.orchestra.run.smithi036.stdout: "zonegroups": [ 2022-09-13T08:11:51.863 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.863 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "test-zonegroup", 2022-09-13T08:11:51.864 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zonegroup", 2022-09-13T08:11:51.864 INFO:teuthology.orchestra.run.smithi036.stdout: "api_name": "", 2022-09-13T08:11:51.865 INFO:teuthology.orchestra.run.smithi036.stdout: "is_master": true, 2022-09-13T08:11:51.865 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [ 2022-09-13T08:11:51.866 INFO:teuthology.orchestra.run.smithi036.stdout: "http://smithi036.front.sepia.ceph.com:8000" 2022-09-13T08:11:51.866 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.867 INFO:teuthology.orchestra.run.smithi036.stdout: "hostnames": [], 2022-09-13T08:11:51.867 INFO:teuthology.orchestra.run.smithi036.stdout: "hostnames_s3website": [], 2022-09-13T08:11:51.868 INFO:teuthology.orchestra.run.smithi036.stdout: "master_zone": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T08:11:51.869 INFO:teuthology.orchestra.run.smithi036.stdout: "zones": [ 2022-09-13T08:11:51.869 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.870 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T08:11:51.870 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zone2", 2022-09-13T08:11:51.871 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [ 2022-09-13T08:11:51.871 INFO:teuthology.orchestra.run.smithi036.stdout: "http://smithi083.front.sepia.ceph.com:8000" 2022-09-13T08:11:51.872 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.872 INFO:teuthology.orchestra.run.smithi036.stdout: "log_meta": false, 2022-09-13T08:11:51.873 INFO:teuthology.orchestra.run.smithi036.stdout: "log_data": true, 2022-09-13T08:11:51.873 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_index_max_shards": 11, 2022-09-13T08:11:51.874 INFO:teuthology.orchestra.run.smithi036.stdout: "read_only": false, 2022-09-13T08:11:51.874 INFO:teuthology.orchestra.run.smithi036.stdout: "tier_type": "", 2022-09-13T08:11:51.875 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from_all": false, 2022-09-13T08:11:51.875 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from": [], 2022-09-13T08:11:51.876 INFO:teuthology.orchestra.run.smithi036.stdout: "redirect_zone": "", 2022-09-13T08:11:51.876 INFO:teuthology.orchestra.run.smithi036.stdout: "supported_features": [ 2022-09-13T08:11:51.877 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T08:11:51.878 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.878 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.879 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.879 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9", 2022-09-13T08:11:51.880 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zone3", 2022-09-13T08:11:51.880 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [ 2022-09-13T08:11:51.881 INFO:teuthology.orchestra.run.smithi036.stdout: "http://smithi036.front.sepia.ceph.com:8001" 2022-09-13T08:11:51.881 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.882 INFO:teuthology.orchestra.run.smithi036.stdout: "log_meta": false, 2022-09-13T08:11:51.882 INFO:teuthology.orchestra.run.smithi036.stdout: "log_data": true, 2022-09-13T08:11:51.883 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_index_max_shards": 11, 2022-09-13T08:11:51.883 INFO:teuthology.orchestra.run.smithi036.stdout: "read_only": false, 2022-09-13T08:11:51.884 INFO:teuthology.orchestra.run.smithi036.stdout: "tier_type": "", 2022-09-13T08:11:51.884 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from_all": true, 2022-09-13T08:11:51.885 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from": [], 2022-09-13T08:11:51.885 INFO:teuthology.orchestra.run.smithi036.stdout: "redirect_zone": "", 2022-09-13T08:11:51.886 INFO:teuthology.orchestra.run.smithi036.stdout: "supported_features": [ 2022-09-13T08:11:51.886 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T08:11:51.887 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.887 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.888 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.888 INFO:teuthology.orchestra.run.smithi036.stdout: "id": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T08:11:51.889 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "test-zone1", 2022-09-13T08:11:51.889 INFO:teuthology.orchestra.run.smithi036.stdout: "endpoints": [ 2022-09-13T08:11:51.890 INFO:teuthology.orchestra.run.smithi036.stdout: "http://smithi036.front.sepia.ceph.com:8000" 2022-09-13T08:11:51.890 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.891 INFO:teuthology.orchestra.run.smithi036.stdout: "log_meta": false, 2022-09-13T08:11:51.891 INFO:teuthology.orchestra.run.smithi036.stdout: "log_data": true, 2022-09-13T08:11:51.892 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_index_max_shards": 11, 2022-09-13T08:11:51.892 INFO:teuthology.orchestra.run.smithi036.stdout: "read_only": false, 2022-09-13T08:11:51.893 INFO:teuthology.orchestra.run.smithi036.stdout: "tier_type": "", 2022-09-13T08:11:51.893 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from_all": true, 2022-09-13T08:11:51.894 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_from": [], 2022-09-13T08:11:51.894 INFO:teuthology.orchestra.run.smithi036.stdout: "redirect_zone": "", 2022-09-13T08:11:51.895 INFO:teuthology.orchestra.run.smithi036.stdout: "supported_features": [ 2022-09-13T08:11:51.895 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T08:11:51.896 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.896 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:51.897 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.897 INFO:teuthology.orchestra.run.smithi036.stdout: "placement_targets": [ 2022-09-13T08:11:51.898 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.898 INFO:teuthology.orchestra.run.smithi036.stdout: "name": "default-placement", 2022-09-13T08:11:51.899 INFO:teuthology.orchestra.run.smithi036.stdout: "tags": [], 2022-09-13T08:11:51.899 INFO:teuthology.orchestra.run.smithi036.stdout: "storage_classes": [ 2022-09-13T08:11:51.900 INFO:teuthology.orchestra.run.smithi036.stdout: "STANDARD" 2022-09-13T08:11:51.900 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.901 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:51.901 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.902 INFO:teuthology.orchestra.run.smithi036.stdout: "default_placement": "default-placement", 2022-09-13T08:11:51.902 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T08:11:51.903 INFO:teuthology.orchestra.run.smithi036.stdout: "sync_policy": { 2022-09-13T08:11:51.903 INFO:teuthology.orchestra.run.smithi036.stdout: "groups": [] 2022-09-13T08:11:51.903 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.903 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled_features": [ 2022-09-13T08:11:51.904 INFO:teuthology.orchestra.run.smithi036.stdout: "resharding" 2022-09-13T08:11:51.904 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.904 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:51.905 INFO:teuthology.orchestra.run.smithi036.stdout: ], 2022-09-13T08:11:51.905 INFO:teuthology.orchestra.run.smithi036.stdout: "short_zone_ids": [ 2022-09-13T08:11:51.905 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.905 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T08:11:51.906 INFO:teuthology.orchestra.run.smithi036.stdout: "val": 2827250327 2022-09-13T08:11:51.906 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.906 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.907 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9", 2022-09-13T08:11:51.907 INFO:teuthology.orchestra.run.smithi036.stdout: "val": 1723930414 2022-09-13T08:11:51.907 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.907 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:51.908 INFO:teuthology.orchestra.run.smithi036.stdout: "key": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T08:11:51.908 INFO:teuthology.orchestra.run.smithi036.stdout: "val": 2837352077 2022-09-13T08:11:51.908 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:51.909 INFO:teuthology.orchestra.run.smithi036.stdout: ] 2022-09-13T08:11:51.909 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.910 INFO:teuthology.orchestra.run.smithi036.stdout: "master_zonegroup": "test-zonegroup", 2022-09-13T08:11:51.911 INFO:teuthology.orchestra.run.smithi036.stdout: "master_zone": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T08:11:51.911 INFO:teuthology.orchestra.run.smithi036.stdout: "period_config": { 2022-09-13T08:11:51.911 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_quota": { 2022-09-13T08:11:51.911 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false, 2022-09-13T08:11:51.912 INFO:teuthology.orchestra.run.smithi036.stdout: "check_on_raw": false, 2022-09-13T08:11:51.912 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size": -1, 2022-09-13T08:11:51.912 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size_kb": 0, 2022-09-13T08:11:51.913 INFO:teuthology.orchestra.run.smithi036.stdout: "max_objects": -1 2022-09-13T08:11:51.913 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.913 INFO:teuthology.orchestra.run.smithi036.stdout: "user_quota": { 2022-09-13T08:11:51.913 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false, 2022-09-13T08:11:51.914 INFO:teuthology.orchestra.run.smithi036.stdout: "check_on_raw": false, 2022-09-13T08:11:51.914 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size": -1, 2022-09-13T08:11:51.914 INFO:teuthology.orchestra.run.smithi036.stdout: "max_size_kb": 0, 2022-09-13T08:11:51.915 INFO:teuthology.orchestra.run.smithi036.stdout: "max_objects": -1 2022-09-13T08:11:51.915 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.915 INFO:teuthology.orchestra.run.smithi036.stdout: "user_ratelimit": { 2022-09-13T08:11:51.916 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_ops": 0, 2022-09-13T08:11:51.916 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_ops": 0, 2022-09-13T08:11:51.917 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_bytes": 0, 2022-09-13T08:11:51.917 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_bytes": 0, 2022-09-13T08:11:51.918 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false 2022-09-13T08:11:51.918 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.918 INFO:teuthology.orchestra.run.smithi036.stdout: "bucket_ratelimit": { 2022-09-13T08:11:51.919 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_ops": 0, 2022-09-13T08:11:51.919 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_ops": 0, 2022-09-13T08:11:51.919 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_bytes": 0, 2022-09-13T08:11:51.920 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_bytes": 0, 2022-09-13T08:11:51.920 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false 2022-09-13T08:11:51.920 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.921 INFO:teuthology.orchestra.run.smithi036.stdout: "anonymous_ratelimit": { 2022-09-13T08:11:51.921 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_ops": 0, 2022-09-13T08:11:51.921 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_ops": 0, 2022-09-13T08:11:51.921 INFO:teuthology.orchestra.run.smithi036.stdout: "max_read_bytes": 0, 2022-09-13T08:11:51.922 INFO:teuthology.orchestra.run.smithi036.stdout: "max_write_bytes": 0, 2022-09-13T08:11:51.922 INFO:teuthology.orchestra.run.smithi036.stdout: "enabled": false 2022-09-13T08:11:51.922 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:51.922 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:51.923 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T08:11:51.923 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_name": "test-realm", 2022-09-13T08:11:51.924 INFO:teuthology.orchestra.run.smithi036.stdout: "realm_epoch": 2 2022-09-13T08:11:51.924 INFO:teuthology.orchestra.run.smithi036.stdout:} 2022-09-13T08:11:51.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2022-09-13T08:11:51.926 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-46 2022-09-13T08:11:51.927 DEBUG:boto:path=/yxvvuh-46/ 2022-09-13T08:11:51.930 DEBUG:boto:auth_path=/yxvvuh-46/ 2022-09-13T08:11:51.932 DEBUG:boto:Method: PUT 2022-09-13T08:11:51.933 DEBUG:boto:Path: /yxvvuh-46/ 2022-09-13T08:11:51.934 DEBUG:boto:Data: 2022-09-13T08:11:51.935 DEBUG:boto:Headers: {} 2022-09-13T08:11:51.935 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T08:11:51.936 DEBUG:boto:Port: 8000 2022-09-13T08:11:51.937 DEBUG:boto:Params: {} 2022-09-13T08:11:51.938 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:11:51.940 DEBUG:boto:Token: None 2022-09-13T08:11:51.942 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:11:51 GMT /yxvvuh-46/ 2022-09-13T08:11:51.942 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7Zvxe6Lfu6bWUjLKds3i3UIGsH8= 2022-09-13T08:11:51.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:51 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7Zvxe6Lfu6bWUjLKds3i3UIGsH8=', 'Content-Length': '0'} 2022-09-13T08:11:52.655 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000054b7eba01abe59be-0063203b48-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:11:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:52.660 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-47 2022-09-13T08:11:52.661 DEBUG:boto:path=/yxvvuh-47/ 2022-09-13T08:11:52.661 DEBUG:boto:auth_path=/yxvvuh-47/ 2022-09-13T08:11:52.662 DEBUG:boto:Method: PUT 2022-09-13T08:11:52.663 DEBUG:boto:Path: /yxvvuh-47/ 2022-09-13T08:11:52.663 DEBUG:boto:Data: 2022-09-13T08:11:52.664 DEBUG:boto:Headers: {} 2022-09-13T08:11:52.664 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T08:11:52.665 DEBUG:boto:Port: 8000 2022-09-13T08:11:52.665 DEBUG:boto:Params: {} 2022-09-13T08:11:52.666 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:11:52.667 DEBUG:boto:Token: None 2022-09-13T08:11:52.667 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:11:52 GMT /yxvvuh-47/ 2022-09-13T08:11:52.668 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:zmva6yIg72OBdfmpgl9YCkuo4F0= 2022-09-13T08:11:52.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:zmva6yIg72OBdfmpgl9YCkuo4F0=', 'Content-Length': '0'} 2022-09-13T08:11:52.737 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bf5bc068c3fe2ad3-0063203b48-1a1a-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:11:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:52.738 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-48 2022-09-13T08:11:52.738 DEBUG:boto:path=/yxvvuh-48/ 2022-09-13T08:11:52.739 DEBUG:boto:auth_path=/yxvvuh-48/ 2022-09-13T08:11:52.739 DEBUG:boto:Method: PUT 2022-09-13T08:11:52.740 DEBUG:boto:Path: /yxvvuh-48/ 2022-09-13T08:11:52.740 DEBUG:boto:Data: 2022-09-13T08:11:52.741 DEBUG:boto:Headers: {} 2022-09-13T08:11:52.742 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T08:11:52.743 DEBUG:boto:Port: 8001 2022-09-13T08:11:52.744 DEBUG:boto:Params: {} 2022-09-13T08:11:52.745 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-13T08:11:52.747 DEBUG:boto:Token: None 2022-09-13T08:11:52.749 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:11:52 GMT /yxvvuh-48/ 2022-09-13T08:11:52.750 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:YORMeqTX8t+dl666/ik1olyw4Zc= 2022-09-13T08:11:52.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YORMeqTX8t+dl666/ik1olyw4Zc=', 'Content-Length': '0'} 2022-09-13T08:11:52.837 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000087c80e9e9f51ac22-0063203b48-1450-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:11:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:52.838 DEBUG:boto:path=/yxvvuh-46/ 2022-09-13T08:11:52.840 DEBUG:boto:auth_path=/yxvvuh-46/ 2022-09-13T08:11:52.840 DEBUG:boto:path=/yxvvuh-46/?website 2022-09-13T08:11:52.841 DEBUG:boto:auth_path=/yxvvuh-46/?website 2022-09-13T08:11:52.841 DEBUG:boto:Method: PUT 2022-09-13T08:11:52.842 DEBUG:boto:Path: /yxvvuh-46/?website 2022-09-13T08:11:52.842 DEBUG:boto:Data: index.htmlerror.html 2022-09-13T08:11:52.843 DEBUG:boto:Headers: {} 2022-09-13T08:11:52.843 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T08:11:52.844 DEBUG:boto:Port: 8000 2022-09-13T08:11:52.844 DEBUG:boto:Params: {} 2022-09-13T08:11:52.845 DEBUG:boto:Token: None 2022-09-13T08:11:52.845 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:11:52 GMT /yxvvuh-46/?website 2022-09-13T08:11:52.846 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:gBXK0OjXVOvydu4iygkPH7FaqfA= 2022-09-13T08:11:52.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:gBXK0OjXVOvydu4iygkPH7FaqfA=', 'Content-Length': '241'} 2022-09-13T08:11:52.849 DEBUG:boto:Response headers: [('Content-Length', '202'), ('x-amz-request-id', 'tx00000e272d33cb74cbf66-0063203b48-1452-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 08:11:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:52.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2022-09-13T08:11:52.862 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-49 2022-09-13T08:11:52.862 DEBUG:boto:path=/yxvvuh-49/ 2022-09-13T08:11:52.863 DEBUG:boto:auth_path=/yxvvuh-49/ 2022-09-13T08:11:52.863 DEBUG:boto:Method: PUT 2022-09-13T08:11:52.864 DEBUG:boto:Path: /yxvvuh-49/ 2022-09-13T08:11:52.864 DEBUG:boto:Data: 2022-09-13T08:11:52.865 DEBUG:boto:Headers: {} 2022-09-13T08:11:52.866 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T08:11:52.866 DEBUG:boto:Port: 8000 2022-09-13T08:11:52.867 DEBUG:boto:Params: {} 2022-09-13T08:11:52.868 DEBUG:boto:Token: None 2022-09-13T08:11:52.868 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:11:52 GMT /yxvvuh-49/ 2022-09-13T08:11:52.869 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:D/LYkr1xJA5cYsiuBW/e2a85t8c= 2022-09-13T08:11:52.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:D/LYkr1xJA5cYsiuBW/e2a85t8c=', 'Content-Length': '0'} 2022-09-13T08:11:52.895 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a69da97cd7506b94-0063203b48-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:11:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:52.896 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-50 2022-09-13T08:11:52.897 DEBUG:boto:path=/yxvvuh-50/ 2022-09-13T08:11:52.897 DEBUG:boto:auth_path=/yxvvuh-50/ 2022-09-13T08:11:52.898 DEBUG:boto:Method: PUT 2022-09-13T08:11:52.898 DEBUG:boto:Path: /yxvvuh-50/ 2022-09-13T08:11:52.899 DEBUG:boto:Data: 2022-09-13T08:11:52.900 DEBUG:boto:Headers: {} 2022-09-13T08:11:52.901 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T08:11:52.901 DEBUG:boto:Port: 8000 2022-09-13T08:11:52.902 DEBUG:boto:Params: {} 2022-09-13T08:11:52.903 DEBUG:boto:Token: None 2022-09-13T08:11:52.904 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:11:52 GMT /yxvvuh-50/ 2022-09-13T08:11:52.904 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1U1+d7FhKxyvru3s0v//Nwmg2TI= 2022-09-13T08:11:52.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1U1+d7FhKxyvru3s0v//Nwmg2TI=', 'Content-Length': '0'} 2022-09-13T08:11:52.938 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ad43c74f4497c8bf-0063203b48-1a1a-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:11:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:52.939 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-51 2022-09-13T08:11:52.940 DEBUG:boto:path=/yxvvuh-51/ 2022-09-13T08:11:52.941 DEBUG:boto:auth_path=/yxvvuh-51/ 2022-09-13T08:11:52.942 DEBUG:boto:Method: PUT 2022-09-13T08:11:52.942 DEBUG:boto:Path: /yxvvuh-51/ 2022-09-13T08:11:52.943 DEBUG:boto:Data: 2022-09-13T08:11:52.944 DEBUG:boto:Headers: {} 2022-09-13T08:11:52.945 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T08:11:52.946 DEBUG:boto:Port: 8001 2022-09-13T08:11:52.947 DEBUG:boto:Params: {} 2022-09-13T08:11:52.948 DEBUG:boto:Token: None 2022-09-13T08:11:52.949 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:11:52 GMT /yxvvuh-51/ 2022-09-13T08:11:52.950 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:d3JrmDWzvtR2B/570TjP1ZBmVZw= 2022-09-13T08:11:52.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:52 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:d3JrmDWzvtR2B/570TjP1ZBmVZw=', 'Content-Length': '0'} 2022-09-13T08:11:53.023 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000008a7b4f852158236-0063203b48-1450-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:11:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:53.024 DEBUG:boto:path=/yxvvuh-49/ 2022-09-13T08:11:53.025 DEBUG:boto:auth_path=/yxvvuh-49/ 2022-09-13T08:11:53.025 DEBUG:boto:path=/yxvvuh-49/?policy 2022-09-13T08:11:53.026 DEBUG:boto:auth_path=/yxvvuh-49/?policy 2022-09-13T08:11:53.027 DEBUG:boto:Method: PUT 2022-09-13T08:11:53.027 DEBUG:boto:Path: /yxvvuh-49/?policy 2022-09-13T08:11:53.028 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2022-09-13T08:11:53.028 DEBUG:boto:Headers: {} 2022-09-13T08:11:53.028 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T08:11:53.029 DEBUG:boto:Port: 8000 2022-09-13T08:11:53.029 DEBUG:boto:Params: {} 2022-09-13T08:11:53.030 DEBUG:boto:Token: None 2022-09-13T08:11:53.031 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:11:53 GMT /yxvvuh-49/?policy 2022-09-13T08:11:53.032 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:0599r0ISDUNtUNrXWcgO3KhrAmE= 2022-09-13T08:11:53.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0599r0ISDUNtUNrXWcgO3KhrAmE=', 'Content-Length': '97'} 2022-09-13T08:11:53.041 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006cadbdd0d25c8922-0063203b49-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 08:11:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:53.042 DEBUG:boto:path=/yxvvuh-49/ 2022-09-13T08:11:53.042 DEBUG:boto:auth_path=/yxvvuh-49/ 2022-09-13T08:11:53.043 DEBUG:boto:path=/yxvvuh-49/?policy 2022-09-13T08:11:53.043 DEBUG:boto:auth_path=/yxvvuh-49/?policy 2022-09-13T08:11:53.044 DEBUG:boto:Method: GET 2022-09-13T08:11:53.045 DEBUG:boto:Path: /yxvvuh-49/?policy 2022-09-13T08:11:53.045 DEBUG:boto:Data: 2022-09-13T08:11:53.046 DEBUG:boto:Headers: {} 2022-09-13T08:11:53.046 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T08:11:53.047 DEBUG:boto:Port: 8000 2022-09-13T08:11:53.047 DEBUG:boto:Params: {} 2022-09-13T08:11:53.048 DEBUG:boto:Token: None 2022-09-13T08:11:53.048 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 08:11:53 GMT /yxvvuh-49/?policy 2022-09-13T08:11:53.049 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:OGx0FsxwwA8wQ1s2Qvp9Dhzy8MU= 2022-09-13T08:11:53.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:OGx0FsxwwA8wQ1s2Qvp9Dhzy8MU=', 'Content-Length': '0'} 2022-09-13T08:11:53.054 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000095a46cea6545e052-0063203b49-1452-test-zone1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Tue, 13 Sep 2022 08:11:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:53.056 DEBUG:boto:path=/yxvvuh-50/ 2022-09-13T08:11:53.057 DEBUG:boto:auth_path=/yxvvuh-50/ 2022-09-13T08:11:53.058 DEBUG:boto:path=/yxvvuh-50/?policy 2022-09-13T08:11:53.059 DEBUG:boto:auth_path=/yxvvuh-50/?policy 2022-09-13T08:11:53.060 DEBUG:boto:Method: PUT 2022-09-13T08:11:53.060 DEBUG:boto:Path: /yxvvuh-50/?policy 2022-09-13T08:11:53.061 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2022-09-13T08:11:53.062 DEBUG:boto:Headers: {} 2022-09-13T08:11:53.063 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T08:11:53.064 DEBUG:boto:Port: 8000 2022-09-13T08:11:53.065 DEBUG:boto:Params: {} 2022-09-13T08:11:53.066 DEBUG:boto:Token: None 2022-09-13T08:11:53.067 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:11:53 GMT /yxvvuh-50/?policy 2022-09-13T08:11:53.068 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:iwuxlCEhqpiCZr0TqTEdVBWE+Rg= 2022-09-13T08:11:53.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:53 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:iwuxlCEhqpiCZr0TqTEdVBWE+Rg=', 'Content-Length': '97'} 2022-09-13T08:11:54.081 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000058d4c3a58d9fa00d-0063203b49-1a1a-test-zone2'), ('Date', 'Tue, 13 Sep 2022 08:11:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:54.083 DEBUG:boto:path=/yxvvuh-50/ 2022-09-13T08:11:54.083 DEBUG:boto:auth_path=/yxvvuh-50/ 2022-09-13T08:11:54.084 DEBUG:boto:path=/yxvvuh-50/?policy 2022-09-13T08:11:54.085 DEBUG:boto:auth_path=/yxvvuh-50/?policy 2022-09-13T08:11:54.085 DEBUG:boto:Method: GET 2022-09-13T08:11:54.086 DEBUG:boto:Path: /yxvvuh-50/?policy 2022-09-13T08:11:54.087 DEBUG:boto:Data: 2022-09-13T08:11:54.088 DEBUG:boto:Headers: {} 2022-09-13T08:11:54.088 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T08:11:54.089 DEBUG:boto:Port: 8000 2022-09-13T08:11:54.089 DEBUG:boto:Params: {} 2022-09-13T08:11:54.090 DEBUG:boto:Token: None 2022-09-13T08:11:54.090 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 08:11:54 GMT /yxvvuh-50/?policy 2022-09-13T08:11:54.091 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:D9DjdCMxC6L5w8LaJj4PcAeDjh4= 2022-09-13T08:11:54.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:D9DjdCMxC6L5w8LaJj4PcAeDjh4=', 'Content-Length': '0'} 2022-09-13T08:11:54.093 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dafdf6a942a57a9d-0063203b4a-1a1a-test-zone2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Tue, 13 Sep 2022 08:11:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:54.094 DEBUG:boto:path=/yxvvuh-51/ 2022-09-13T08:11:54.095 DEBUG:boto:auth_path=/yxvvuh-51/ 2022-09-13T08:11:54.096 DEBUG:boto:path=/yxvvuh-51/?policy 2022-09-13T08:11:54.096 DEBUG:boto:auth_path=/yxvvuh-51/?policy 2022-09-13T08:11:54.097 DEBUG:boto:Method: PUT 2022-09-13T08:11:54.098 DEBUG:boto:Path: /yxvvuh-51/?policy 2022-09-13T08:11:54.098 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2022-09-13T08:11:54.099 DEBUG:boto:Headers: {} 2022-09-13T08:11:54.099 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T08:11:54.100 DEBUG:boto:Port: 8001 2022-09-13T08:11:54.100 DEBUG:boto:Params: {} 2022-09-13T08:11:54.101 DEBUG:boto:Token: None 2022-09-13T08:11:54.102 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:11:54 GMT /yxvvuh-51/?policy 2022-09-13T08:11:54.102 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Ww/OTGQ4DtT6Elf3zrSdOeW45I0= 2022-09-13T08:11:54.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Ww/OTGQ4DtT6Elf3zrSdOeW45I0=', 'Content-Length': '97'} 2022-09-13T08:11:55.114 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000138a12bd4d81bf9-0063203b4a-1450-test-zone3'), ('Date', 'Tue, 13 Sep 2022 08:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:55.115 DEBUG:boto:path=/yxvvuh-51/ 2022-09-13T08:11:55.115 DEBUG:boto:auth_path=/yxvvuh-51/ 2022-09-13T08:11:55.116 DEBUG:boto:path=/yxvvuh-51/?policy 2022-09-13T08:11:55.117 DEBUG:boto:auth_path=/yxvvuh-51/?policy 2022-09-13T08:11:55.117 DEBUG:boto:Method: GET 2022-09-13T08:11:55.118 DEBUG:boto:Path: /yxvvuh-51/?policy 2022-09-13T08:11:55.118 DEBUG:boto:Data: 2022-09-13T08:11:55.119 DEBUG:boto:Headers: {} 2022-09-13T08:11:55.120 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T08:11:55.120 DEBUG:boto:Port: 8001 2022-09-13T08:11:55.121 DEBUG:boto:Params: {} 2022-09-13T08:11:55.121 DEBUG:boto:Token: None 2022-09-13T08:11:55.122 DEBUG:boto:StringToSign: GET Tue, 13 Sep 2022 08:11:55 GMT /yxvvuh-51/?policy 2022-09-13T08:11:55.122 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RQpFxKfY6Atliz1RE+HFqMyqvII= 2022-09-13T08:11:55.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:55 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RQpFxKfY6Atliz1RE+HFqMyqvII=', 'Content-Length': '0'} 2022-09-13T08:11:55.126 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008610d9bae04bb477-0063203b4b-1450-test-zone3'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Tue, 13 Sep 2022 08:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:55.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2022-09-13T08:11:55.127 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-52 2022-09-13T08:11:55.128 DEBUG:boto:path=/yxvvuh-52/ 2022-09-13T08:11:55.128 DEBUG:boto:auth_path=/yxvvuh-52/ 2022-09-13T08:11:55.129 DEBUG:boto:Method: PUT 2022-09-13T08:11:55.129 DEBUG:boto:Path: /yxvvuh-52/ 2022-09-13T08:11:55.130 DEBUG:boto:Data: 2022-09-13T08:11:55.130 DEBUG:boto:Headers: {} 2022-09-13T08:11:55.131 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T08:11:55.131 DEBUG:boto:Port: 8000 2022-09-13T08:11:55.132 DEBUG:boto:Params: {} 2022-09-13T08:11:55.132 DEBUG:boto:Token: None 2022-09-13T08:11:55.133 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:11:55 GMT /yxvvuh-52/ 2022-09-13T08:11:55.134 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:LXBbGOU86tjSaNYkw7yk0O8luI8= 2022-09-13T08:11:55.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:55 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LXBbGOU86tjSaNYkw7yk0O8luI8=', 'Content-Length': '0'} 2022-09-13T08:11:55.157 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a120a50547e3f4e8-0063203b4b-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:55.158 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-53 2022-09-13T08:11:55.159 DEBUG:boto:path=/yxvvuh-53/ 2022-09-13T08:11:55.160 DEBUG:boto:auth_path=/yxvvuh-53/ 2022-09-13T08:11:55.161 DEBUG:boto:Method: PUT 2022-09-13T08:11:55.161 DEBUG:boto:Path: /yxvvuh-53/ 2022-09-13T08:11:55.162 DEBUG:boto:Data: 2022-09-13T08:11:55.163 DEBUG:boto:Headers: {} 2022-09-13T08:11:55.164 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T08:11:55.165 DEBUG:boto:Port: 8000 2022-09-13T08:11:55.166 DEBUG:boto:Params: {} 2022-09-13T08:11:55.167 DEBUG:boto:Token: None 2022-09-13T08:11:55.168 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:11:55 GMT /yxvvuh-53/ 2022-09-13T08:11:55.168 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:jQ7zlkmtF1ZpzlO6qgtLc7XpZLU= 2022-09-13T08:11:55.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:55 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jQ7zlkmtF1ZpzlO6qgtLc7XpZLU=', 'Content-Length': '0'} 2022-09-13T08:11:55.211 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a5c68dba85ee1ea4-0063203b4b-1a1a-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:55.213 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=yxvvuh-54 2022-09-13T08:11:55.213 DEBUG:boto:path=/yxvvuh-54/ 2022-09-13T08:11:55.214 DEBUG:boto:auth_path=/yxvvuh-54/ 2022-09-13T08:11:55.216 DEBUG:boto:Method: PUT 2022-09-13T08:11:55.217 DEBUG:boto:Path: /yxvvuh-54/ 2022-09-13T08:11:55.217 DEBUG:boto:Data: 2022-09-13T08:11:55.219 DEBUG:boto:Headers: {} 2022-09-13T08:11:55.220 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T08:11:55.220 DEBUG:boto:Port: 8001 2022-09-13T08:11:55.221 DEBUG:boto:Params: {} 2022-09-13T08:11:55.223 DEBUG:boto:Token: None 2022-09-13T08:11:55.224 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:11:55 GMT /yxvvuh-54/ 2022-09-13T08:11:55.225 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:UJ5nlOU47ikEkds9CAs0arW2AZE= 2022-09-13T08:11:55.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:55 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UJ5nlOU47ikEkds9CAs0arW2AZE=', 'Content-Length': '0'} 2022-09-13T08:11:55.266 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002a86517a0685aa2e-0063203b4b-1450-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:11:55.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:55.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:55.268 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:11:55.326 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:55.326 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:55.357 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T08:11:55.374 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T08:11:55.374 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:55.375 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:11:55.375 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T08:11:55.375 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:55.375 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:55.376 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T08:11:55.376 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T08:11:55.376 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:55.377 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:55.377 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663056715.246326_454.1", 2022-09-13T08:11:55.378 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T08:11:55.246326Z" 2022-09-13T08:11:55.378 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:11:55.378 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:11:55.379 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:11:55.379 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T08:11:55.379 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:11:55.379 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T08:11:55.383 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:55.383 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T08:11:55.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:55.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:11:55.385 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:11:55.442 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:11:55.442 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:11:55.490 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:55.486+0000 7f8355435600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:55.491 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:11:55.488+0000 7f8355435600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:11:55.515 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:11:55.516 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:11:55.517 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:11:55.517 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:11:55.517 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:11:55.518 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:11:55.518 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:55.519 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:55.519 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:11:55.520 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:55.520 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:11:55.521 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:55.521 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:55.522 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:11:55.522 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:11:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:11:55.524 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:55.524 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:55.524 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:55.524 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:55.525 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:11:55.525 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:55.526 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:55.526 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:11:55.526 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:55.527 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:55.527 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:55.527 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:11:55.528 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:55.528 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:55.528 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:55.529 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:55.529 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:11:55.530 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:55.530 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:55.530 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:11:55.531 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:55.531 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:55.532 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:55.532 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:11:55.532 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:55.533 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:55.533 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:55.534 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:11:55.534 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:11:55.535 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:11:55.535 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:11:55.536 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:11:55.536 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:11:55.537 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:11:55.537 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:11:55.537 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:11:55.537 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:11:55.538 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:55.539 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:11:55.539 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:11:55.540 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:55.540 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:11:55.540 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:11:55.541 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:11:55.541 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:11:55.541 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:11:55Z" 2022-09-13T08:11:55.542 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:11:55.543 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:11:55Z" } 2022-09-13T08:11:55.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:11:55.545 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:55.545 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:11:55.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:11:56.007 INFO:tasks.rgw.c2.client.0.smithi083.stdout:*** Caught signal (Segmentation fault) ** 2022-09-13T08:11:56.008 INFO:tasks.rgw.c2.client.0.smithi083.stdout: in thread 7fd5fbfb7700 thread_name:lua_background 2022-09-13T08:11:56.009 INFO:tasks.rgw.c2.client.0.smithi083.stdout: ceph version 17.0.0-14918-g840d0541 (840d05418c459144c92fb2a90d74a2ded29610da) quincy (dev) 2022-09-13T08:11:56.010 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fd735f91cf0] 2022-09-13T08:11:56.010 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 2: (RGWSI_SysObj::Obj::ROp::read(DoutPrefixProvider const*, long, long, ceph::buffer::v15_2_0::list*, optional_yield)+0x43) [0x55828ed55a33] 2022-09-13T08:11:56.011 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 3: (rgw_get_system_obj(RGWSI_SysObj*, rgw_pool const&, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list&, RGWObjVersionTracker*, std::chrono::time_point > >*, optional_yield, DoutPrefixProvider const*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, rgw_cache_entry_info*, boost::optional, bool)+0x1f7) [0x55828f23b807] 2022-09-13T08:11:56.011 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 4: (rgw::sal::RadosLuaManager::get_script(DoutPrefixProvider const*, optional_yield, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator >&)+0x1a5) [0x55828f225665] 2022-09-13T08:11:56.012 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 5: (rgw::lua::read_script(DoutPrefixProvider const*, rgw::sal::LuaManager*, std::__cxx11::basic_string, std::allocator > const&, optional_yield, rgw::lua::context, std::__cxx11::basic_string, std::allocator >&)+0x75) [0x55828f345d65] 2022-09-13T08:11:56.013 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 6: (rgw::lua::Background::read_script()+0x9f) [0x55828f35132f] 2022-09-13T08:11:56.013 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 7: (rgw::lua::Background::run()+0x6f6) [0x55828f352386] 2022-09-13T08:11:56.014 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 8: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fd73570aba3] 2022-09-13T08:11:56.014 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fd735f871ca] 2022-09-13T08:11:56.015 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 10: clone() 2022-09-13T08:11:56.015 INFO:tasks.rgw.c2.client.0.smithi083.stdout:2022-09-13T08:11:56.006+0000 7fd5fbfb7700 -1 *** Caught signal (Segmentation fault) ** 2022-09-13T08:11:56.016 INFO:tasks.rgw.c2.client.0.smithi083.stdout: in thread 7fd5fbfb7700 thread_name:lua_background 2022-09-13T08:11:56.016 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 2022-09-13T08:11:56.017 INFO:tasks.rgw.c2.client.0.smithi083.stdout: ceph version 17.0.0-14918-g840d0541 (840d05418c459144c92fb2a90d74a2ded29610da) quincy (dev) 2022-09-13T08:11:56.017 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fd735f91cf0] 2022-09-13T08:11:56.018 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 2: (RGWSI_SysObj::Obj::ROp::read(DoutPrefixProvider const*, long, long, ceph::buffer::v15_2_0::list*, optional_yield)+0x43) [0x55828ed55a33] 2022-09-13T08:11:56.018 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 3: (rgw_get_system_obj(RGWSI_SysObj*, rgw_pool const&, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list&, RGWObjVersionTracker*, std::chrono::time_point > >*, optional_yield, DoutPrefixProvider const*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, rgw_cache_entry_info*, boost::optional, bool)+0x1f7) [0x55828f23b807] 2022-09-13T08:11:56.019 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 4: (rgw::sal::RadosLuaManager::get_script(DoutPrefixProvider const*, optional_yield, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator >&)+0x1a5) [0x55828f225665] 2022-09-13T08:11:56.019 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 5: (rgw::lua::read_script(DoutPrefixProvider const*, rgw::sal::LuaManager*, std::__cxx11::basic_string, std::allocator > const&, optional_yield, rgw::lua::context, std::__cxx11::basic_string, std::allocator >&)+0x75) [0x55828f345d65] 2022-09-13T08:11:56.020 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 6: (rgw::lua::Background::read_script()+0x9f) [0x55828f35132f] 2022-09-13T08:11:56.021 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 7: (rgw::lua::Background::run()+0x6f6) [0x55828f352386] 2022-09-13T08:11:56.021 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 8: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fd73570aba3] 2022-09-13T08:11:56.022 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fd735f871ca] 2022-09-13T08:11:56.022 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 10: clone() 2022-09-13T08:11:56.023 INFO:tasks.rgw.c2.client.0.smithi083.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-13T08:11:56.023 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 2022-09-13T08:11:56.024 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 0> 2022-09-13T08:11:56.006+0000 7fd5fbfb7700 -1 *** Caught signal (Segmentation fault) ** 2022-09-13T08:11:56.025 INFO:tasks.rgw.c2.client.0.smithi083.stdout: in thread 7fd5fbfb7700 thread_name:lua_background 2022-09-13T08:11:56.025 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 2022-09-13T08:11:56.026 INFO:tasks.rgw.c2.client.0.smithi083.stdout: ceph version 17.0.0-14918-g840d0541 (840d05418c459144c92fb2a90d74a2ded29610da) quincy (dev) 2022-09-13T08:11:56.026 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fd735f91cf0] 2022-09-13T08:11:56.027 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 2: (RGWSI_SysObj::Obj::ROp::read(DoutPrefixProvider const*, long, long, ceph::buffer::v15_2_0::list*, optional_yield)+0x43) [0x55828ed55a33] 2022-09-13T08:11:56.027 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 3: (rgw_get_system_obj(RGWSI_SysObj*, rgw_pool const&, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list&, RGWObjVersionTracker*, std::chrono::time_point > >*, optional_yield, DoutPrefixProvider const*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, rgw_cache_entry_info*, boost::optional, bool)+0x1f7) [0x55828f23b807] 2022-09-13T08:11:56.028 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 4: (rgw::sal::RadosLuaManager::get_script(DoutPrefixProvider const*, optional_yield, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator >&)+0x1a5) [0x55828f225665] 2022-09-13T08:11:56.028 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 5: (rgw::lua::read_script(DoutPrefixProvider const*, rgw::sal::LuaManager*, std::__cxx11::basic_string, std::allocator > const&, optional_yield, rgw::lua::context, std::__cxx11::basic_string, std::allocator >&)+0x75) [0x55828f345d65] 2022-09-13T08:11:56.029 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 6: (rgw::lua::Background::read_script()+0x9f) [0x55828f35132f] 2022-09-13T08:11:56.029 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 7: (rgw::lua::Background::run()+0x6f6) [0x55828f352386] 2022-09-13T08:11:56.030 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 8: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fd73570aba3] 2022-09-13T08:11:56.030 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fd735f871ca] 2022-09-13T08:11:56.031 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 10: clone() 2022-09-13T08:11:56.031 INFO:tasks.rgw.c2.client.0.smithi083.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-13T08:11:56.032 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 2022-09-13T08:11:56.035 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 0> 2022-09-13T08:11:56.006+0000 7fd5fbfb7700 -1 *** Caught signal (Segmentation fault) ** 2022-09-13T08:11:56.035 INFO:tasks.rgw.c2.client.0.smithi083.stdout: in thread 7fd5fbfb7700 thread_name:lua_background 2022-09-13T08:11:56.036 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 2022-09-13T08:11:56.036 INFO:tasks.rgw.c2.client.0.smithi083.stdout: ceph version 17.0.0-14918-g840d0541 (840d05418c459144c92fb2a90d74a2ded29610da) quincy (dev) 2022-09-13T08:11:56.037 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fd735f91cf0] 2022-09-13T08:11:56.037 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 2: (RGWSI_SysObj::Obj::ROp::read(DoutPrefixProvider const*, long, long, ceph::buffer::v15_2_0::list*, optional_yield)+0x43) [0x55828ed55a33] 2022-09-13T08:11:56.038 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 3: (rgw_get_system_obj(RGWSI_SysObj*, rgw_pool const&, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list&, RGWObjVersionTracker*, std::chrono::time_point > >*, optional_yield, DoutPrefixProvider const*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, rgw_cache_entry_info*, boost::optional, bool)+0x1f7) [0x55828f23b807] 2022-09-13T08:11:56.038 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 4: (rgw::sal::RadosLuaManager::get_script(DoutPrefixProvider const*, optional_yield, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator >&)+0x1a5) [0x55828f225665] 2022-09-13T08:11:56.039 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 5: (rgw::lua::read_script(DoutPrefixProvider const*, rgw::sal::LuaManager*, std::__cxx11::basic_string, std::allocator > const&, optional_yield, rgw::lua::context, std::__cxx11::basic_string, std::allocator >&)+0x75) [0x55828f345d65] 2022-09-13T08:11:56.039 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 6: (rgw::lua::Background::read_script()+0x9f) [0x55828f35132f] 2022-09-13T08:11:56.040 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 7: (rgw::lua::Background::run()+0x6f6) [0x55828f352386] 2022-09-13T08:11:56.040 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 8: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fd73570aba3] 2022-09-13T08:11:56.040 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fd735f871ca] 2022-09-13T08:11:56.041 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 10: clone() 2022-09-13T08:11:56.041 INFO:tasks.rgw.c2.client.0.smithi083.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-13T08:11:56.042 INFO:tasks.rgw.c2.client.0.smithi083.stdout: 2022-09-13T08:11:56.182 INFO:tasks.rgw.c2.client.0.smithi083.stderr:daemon-helper: command crashed with signal 11 2022-09-13T08:11:59.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-09-13T08:12:00.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:00.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:00.548 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:12:00.606 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:12:00.606 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:12:00.640 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:00.636+0000 7fecc5f7e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:00.641 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:00.637+0000 7fecc5f7e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:00.656 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:12:00.657 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:12:00.657 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:12:00.658 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:12:00.658 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:12:00.658 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:12:00.658 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:00.659 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:00.659 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:12:00.659 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:00.660 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:12:00.660 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:00.660 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:00.660 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:12:00.661 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:00.661 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:12:00.661 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:00.661 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:12:00.662 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:00.662 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:00.662 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:00.662 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:00.663 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:12:00.663 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:00.663 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:00.664 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:12:00.664 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:00.664 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:00.665 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:00.665 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:12:00.665 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:00.666 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:00.666 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:00.671 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:00.671 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:12:00.672 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:00.672 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:00.673 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:12:00.673 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:00.673 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:00.674 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:00.674 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:12:00.675 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:00.675 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:00.676 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:00.676 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:00.677 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:12:00.677 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:00.678 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:00.678 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:12:00.678 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:00.679 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:00.679 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:00.680 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:12:00.680 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:00.680 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:00.680 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:00.681 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:12:00.681 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:00.681 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:12:00.681 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:12:00.682 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:12:00.682 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:00.683 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:12:00Z" 2022-09-13T08:12:00.683 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:12:00.685 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:12:00Z" } 2022-09-13T08:12:00.686 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:12:00.687 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:00.687 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:00.687 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:12:04.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~6s 2022-09-13T08:12:05.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:05.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:05.690 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:12:05.747 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:12:05.748 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:12:05.780 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:05.776+0000 7f165ea73600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:05.781 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:05.778+0000 7f165ea73600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:05.798 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:12:05.798 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:12:05.799 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:12:05.799 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:12:05.799 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:12:05.800 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:12:05.800 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:05.800 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:05.800 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:12:05.801 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:05.801 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:12:05.801 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:05.801 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:05.802 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:12:05.802 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:05.803 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:12:05.803 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:05.803 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:12:05.803 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:05.804 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:05.804 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:05.804 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:05.804 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:12:05.805 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:05.805 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:05.805 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:12:05.806 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:05.806 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:05.806 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:05.807 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:12:05.807 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:05.807 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:05.808 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:05.808 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:05.808 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:12:05.808 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:05.809 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:05.809 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:12:05.809 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:05.809 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:05.810 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:05.810 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:12:05.810 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:05.811 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:05.811 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:05.811 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:05.811 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:12:05.812 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:05.812 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:05.812 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:12:05.812 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:05.813 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:05.813 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:05.813 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:12:05.814 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:05.814 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:05.814 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:05.815 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:12:05.815 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:05.815 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:12:05.815 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:12:05.816 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:12:05.816 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:05.816 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:12:05Z" 2022-09-13T08:12:05.816 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:12:05.818 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:12:05Z" } 2022-09-13T08:12:05.818 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:12:05.819 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:05.820 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:05.820 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:12:10.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~11s 2022-09-13T08:12:10.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:10.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:10.824 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:12:10.880 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:12:10.881 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:12:10.933 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:10.930+0000 7f20c3eaf600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:10.934 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:10.931+0000 7f20c3eaf600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:10.956 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:12:10.957 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:12:10.957 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:12:10.958 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:12:10.958 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:12:10.958 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:12:10.958 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:10.959 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:10.959 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:12:10.960 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:10.960 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:12:10.960 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:10.960 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:10.961 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:12:10.961 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:10.961 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:12:10.961 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:10.962 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:12:10.962 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:10.962 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:10.963 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:10.963 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:10.963 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:12:10.964 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:10.964 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:10.964 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:12:10.965 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:10.965 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:10.965 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:10.966 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:12:10.966 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:10.966 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:10.966 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:10.967 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:10.967 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:12:10.967 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:10.968 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:10.968 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:12:10.968 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:10.969 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:10.969 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:10.969 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:12:10.969 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:10.970 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:10.970 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:10.970 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:10.971 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:12:10.971 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:10.971 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:10.971 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:12:10.972 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:10.972 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:10.972 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:10.973 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:12:10.973 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:10.973 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:10.974 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:10.974 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:12:10.974 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:10.975 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:12:10.975 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:12:10.975 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:12:10.975 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:10.976 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:12:10Z" 2022-09-13T08:12:10.976 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:12:10.977 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:12:10Z" } 2022-09-13T08:12:10.978 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:12:10.979 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:10.979 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:10.980 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:12:15.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:15.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:15.983 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:12:15.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~17s 2022-09-13T08:12:16.040 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:12:16.040 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:12:16.073 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:16.070+0000 7f5d1600e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:16.071+0000 7f5d1600e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:16.090 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:12:16.090 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:12:16.091 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:12:16.091 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:12:16.091 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:12:16.092 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:12:16.092 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:16.093 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:16.093 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:12:16.093 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:16.093 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:12:16.094 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:16.094 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:16.094 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:12:16.095 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:16.095 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:12:16.095 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:16.095 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:12:16.096 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:16.096 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:16.096 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:16.096 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:16.097 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:12:16.097 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:16.097 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:16.097 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:12:16.098 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:16.098 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:16.098 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:16.098 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:12:16.099 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:16.099 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:16.099 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:16.100 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:16.100 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:12:16.100 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:16.100 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:16.101 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:12:16.101 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:16.101 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:16.101 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:16.102 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:12:16.102 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:16.102 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:16.103 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:16.103 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:16.103 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:12:16.103 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:16.104 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:16.104 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:12:16.104 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:16.105 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:16.105 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:16.105 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:12:16.105 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:16.106 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:16.106 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:16.106 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:12:16.107 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:16.107 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:12:16.107 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:12:16.107 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:12:16.108 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:16.108 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:12:16Z" 2022-09-13T08:12:16.108 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:12:16.109 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:12:16Z" } 2022-09-13T08:12:16.110 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:12:16.111 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:16.111 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:16.112 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:12:21.113 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:21.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:21.115 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:12:21.172 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:12:21.173 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:12:21.214 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:21.211+0000 7f787ea8f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:21.214 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:21.212+0000 7f787ea8f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:21.230 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:12:21.231 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:12:21.231 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:12:21.234 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:12:21.234 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:12:21.234 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:12:21.235 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:21.235 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:21.235 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:12:21.236 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:21.236 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:12:21.236 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:21.236 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:21.237 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:12:21.237 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:21.237 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:12:21.237 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:21.238 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:12:21.238 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:21.238 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:21.238 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:21.239 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:21.239 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:12:21.239 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:21.239 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:21.240 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:12:21.240 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:21.240 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:21.240 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:21.241 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:12:21.241 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:21.241 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:21.241 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:21.242 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:21.242 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:12:21.242 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:21.242 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:21.243 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:12:21.243 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:21.243 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:21.243 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:21.244 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:12:21.244 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:21.244 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:21.244 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:21.245 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:21.245 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:12:21.245 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:21.245 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:21.246 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:12:21.246 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:21.246 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:21.246 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:21.247 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:12:21.247 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:21.247 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:21.247 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:21.247 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:12:21.248 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:21.248 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:12:21.248 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:12:21.248 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:12:21.249 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:21.249 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:12:21Z" 2022-09-13T08:12:21.249 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:12:21.250 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:12:21Z" } 2022-09-13T08:12:21.251 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:12:21.252 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:21.252 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:21.253 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:12:21.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~22s 2022-09-13T08:12:26.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:26.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:26.256 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:12:26.312 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:12:26.313 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:12:26.347 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:26.343+0000 7f0780951600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:26.348 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:26.344+0000 7f0780951600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:26.364 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:12:26.364 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:12:26.365 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:12:26.365 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:12:26.365 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:12:26.366 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:12:26.366 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:26.367 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:26.367 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:12:26.368 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:26.368 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:12:26.368 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:26.369 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:26.369 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:12:26.370 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:26.370 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:12:26.371 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:26.371 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:12:26.371 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:26.372 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:26.372 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:26.373 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:26.373 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:12:26.373 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:26.373 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:26.374 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:12:26.374 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:26.375 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:26.375 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:26.376 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:12:26.376 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:26.376 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:26.377 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:26.377 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:26.378 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:12:26.378 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:26.379 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:26.379 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:12:26.380 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:26.380 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:26.380 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:26.381 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:12:26.381 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:26.382 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:26.382 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:26.383 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:26.383 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:12:26.383 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:26.384 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:26.384 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:12:26.385 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:26.385 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:26.386 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:26.386 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:12:26.387 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:26.387 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:26.387 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:26.388 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:12:26.388 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:26.389 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:12:26.389 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:12:26.390 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:12:26.390 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:26.391 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:12:26Z" 2022-09-13T08:12:26.391 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:12:26.393 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:12:26Z" } 2022-09-13T08:12:26.394 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:12:26.395 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:26.395 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:26.396 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:12:27.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~28s 2022-09-13T08:12:31.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:31.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:31.399 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:12:31.456 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:12:31.456 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:12:31.507 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:31.503+0000 7f187ad4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:31.508 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:31.504+0000 7f187ad4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:31.524 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:12:31.524 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:12:31.524 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:12:31.525 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:12:31.525 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:12:31.525 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:12:31.525 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:31.526 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:31.526 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:12:31.526 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:31.527 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:12:31.527 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:31.527 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:31.528 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:12:31.528 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:31.528 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:12:31.529 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:31.529 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:12:31.529 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:31.530 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:31.530 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:31.530 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:31.530 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:12:31.531 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:31.531 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:31.531 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:12:31.531 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:31.532 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:31.532 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:31.532 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:12:31.532 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:31.533 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:31.533 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:31.533 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:31.534 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:12:31.534 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:31.534 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:31.535 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:12:31.535 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:31.535 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:31.536 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:31.536 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:12:31.536 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:31.537 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:31.537 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:31.537 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:31.537 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:12:31.538 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:31.538 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:31.538 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:12:31.538 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:31.539 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:31.539 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:31.539 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:12:31.539 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:31.540 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:31.540 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:31.541 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:12:31.541 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:31.541 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:12:31.541 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:12:31.542 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:12:31.542 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:31.542 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:12:31Z" 2022-09-13T08:12:31.542 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:12:31.544 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:12:31Z" } 2022-09-13T08:12:31.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:12:31.545 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:31.545 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:31.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:12:32.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~34s 2022-09-13T08:12:36.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:36.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:36.548 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:12:36.606 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:12:36.607 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:12:36.641 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:36.637+0000 7f65c9a14600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:36.641 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:36.638+0000 7f65c9a14600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:36.659 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:12:36.659 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:12:36.659 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:12:36.660 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:12:36.660 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:12:36.660 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:12:36.660 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:36.661 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:36.661 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:12:36.661 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:36.662 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:12:36.662 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:36.662 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:36.662 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:12:36.663 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:36.663 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:12:36.663 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:36.663 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:12:36.664 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:36.664 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:36.664 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:36.665 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:36.665 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:12:36.665 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:36.665 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:36.666 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:12:36.666 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:36.666 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:36.666 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:36.667 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:12:36.667 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:36.667 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:36.667 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:36.668 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:36.668 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:12:36.668 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:36.669 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:36.669 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:12:36.670 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:36.670 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:36.670 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:36.671 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:12:36.671 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:36.671 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:36.671 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:36.672 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:36.672 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:12:36.672 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:36.672 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:36.673 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:12:36.673 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:36.673 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:36.674 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:36.674 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:12:36.674 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:36.674 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:36.675 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:36.675 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:12:36.675 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:36.675 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:12:36.676 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:12:36.676 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:12:36.676 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:36.677 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:12:36Z" 2022-09-13T08:12:36.677 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:12:36.678 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:12:36Z" } 2022-09-13T08:12:36.679 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:12:36.680 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:36.680 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:36.680 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:12:38.415 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~39s 2022-09-13T08:12:41.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:41.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:41.683 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:12:41.740 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:12:41.741 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:12:41.789 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:41.785+0000 7f9e35ced600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:41.790 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:41.786+0000 7f9e35ced600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:41.813 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:12:41.813 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:12:41.814 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:12:41.814 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:12:41.814 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:12:41.815 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:12:41.815 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:41.815 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:41.815 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:12:41.815 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:41.816 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:12:41.816 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:41.816 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:41.816 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:12:41.817 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:41.817 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:12:41.817 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:41.817 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:12:41.818 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:41.818 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:41.819 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:41.819 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:41.819 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:12:41.820 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:41.820 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:41.821 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:12:41.821 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:41.821 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:41.821 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:41.822 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:12:41.822 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:41.822 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:41.823 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:41.823 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:41.823 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:12:41.823 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:41.824 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:41.824 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:12:41.824 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:41.824 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:41.825 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:41.825 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:12:41.825 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:41.825 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:41.826 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:41.826 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:41.826 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:12:41.826 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:41.827 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:41.827 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:12:41.827 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:41.831 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:41.831 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:41.831 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:12:41.831 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:41.832 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:41.832 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:41.832 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:12:41.832 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:41.833 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:12:41.833 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:12:41.833 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:12:41.834 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:41.834 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:12:41Z" 2022-09-13T08:12:41.835 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:12:41.837 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:12:41Z" } 2022-09-13T08:12:41.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:12:41.839 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:41.840 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:41.841 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:12:44.022 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~45s 2022-09-13T08:12:46.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:46.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:46.845 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:12:46.902 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:12:46.903 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:12:46.944 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:46.940+0000 7fbb82081600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:46.944 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:46.941+0000 7fbb82081600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:46.961 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:12:46.961 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:12:46.961 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:12:46.962 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:12:46.962 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:12:46.962 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:12:46.963 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:46.963 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:46.963 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:12:46.964 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:46.964 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:12:46.965 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:46.965 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:46.970 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:12:46.971 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:46.971 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:12:46.972 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:46.972 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:12:46.973 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:46.973 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:46.974 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:46.974 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:46.975 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:12:46.976 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:46.976 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:46.976 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:12:46.976 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:46.977 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:46.977 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:46.977 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:12:46.977 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:46.978 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:46.978 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:46.978 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:46.979 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:12:46.979 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:46.979 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:46.979 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:12:46.980 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:46.980 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:46.980 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:46.980 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:12:46.981 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:46.981 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:46.981 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:46.981 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:46.982 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:12:46.982 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:46.982 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:46.982 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:12:46.983 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:46.983 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:46.983 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:46.983 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:12:46.984 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:46.984 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:46.984 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:46.984 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:12:46.985 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:46.985 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:12:46.985 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:12:46.985 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:12:46.986 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:46.986 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:12:46Z" 2022-09-13T08:12:46.986 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:12:46.988 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:12:46Z" } 2022-09-13T08:12:46.989 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:12:46.989 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:46.990 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:46.991 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:12:49.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~50s 2022-09-13T08:12:51.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:51.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:51.994 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:12:52.052 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:12:52.053 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:12:52.097 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:52.094+0000 7f45eca84600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:52.098 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:52.095+0000 7f45eca84600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:52.122 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:12:52.122 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:12:52.123 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:12:52.123 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:12:52.124 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:12:52.124 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:12:52.125 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:52.125 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:52.125 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:12:52.126 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:52.126 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:12:52.126 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:52.127 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:52.127 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:12:52.127 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:52.127 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:12:52.128 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:52.128 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:12:52.128 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:52.129 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:52.129 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:52.129 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:52.129 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:12:52.130 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:52.130 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:52.130 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:12:52.130 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:52.131 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:52.131 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:52.131 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:12:52.131 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:52.132 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:52.132 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:52.132 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:52.132 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:12:52.133 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:52.133 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:52.133 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:12:52.133 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:52.134 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:52.134 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:52.134 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:12:52.135 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:52.135 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:52.135 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:52.135 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:52.136 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:12:52.136 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:52.136 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:52.137 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:12:52.137 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:52.137 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:52.138 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:52.138 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:12:52.138 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:52.138 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:52.139 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:52.139 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:12:52.139 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:52.139 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:12:52.140 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:12:52.140 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:12:52.140 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:52.141 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:12:52Z" 2022-09-13T08:12:52.141 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:12:52.142 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:12:52Z" } 2022-09-13T08:12:52.143 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:12:52.143 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:52.144 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:52.144 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:12:55.233 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~56s 2022-09-13T08:12:57.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:57.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:12:57.148 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:12:57.208 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:12:57.209 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:12:57.254 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:57.251+0000 7f9c1bcb3600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:57.254 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:12:57.252+0000 7f9c1bcb3600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:12:57.270 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:12:57.271 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:12:57.271 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:12:57.271 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:12:57.272 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:12:57.272 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:12:57.272 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:57.272 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:57.273 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:12:57.273 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:57.273 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:12:57.273 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:57.274 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:57.274 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:12:57.274 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:57.274 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:12:57.275 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:57.275 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:12:57.275 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:57.276 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:57.276 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:57.276 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:57.276 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:12:57.277 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:57.277 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:57.277 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:12:57.277 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:57.277 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:57.278 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:57.278 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:12:57.278 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:57.278 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:57.279 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:57.279 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:57.279 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:12:57.280 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:57.280 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:57.280 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:12:57.281 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:57.281 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:57.281 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:57.281 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:12:57.282 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:57.282 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:57.282 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:57.282 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:12:57.283 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:12:57.283 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:12:57.283 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:12:57.283 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:12:57.284 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:12:57.284 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:12:57.284 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:12:57.284 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:12:57.285 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:12:57.285 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:57.285 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:12:57.285 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:12:57.285 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:57.286 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:12:57.286 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:12:57.286 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:12:57.286 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:12:57.287 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:12:57Z" 2022-09-13T08:12:57.287 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:12:57.290 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:12:57Z" } 2022-09-13T08:12:57.291 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:12:57.291 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:57.292 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:12:57.292 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:13:00.840 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~62s 2022-09-13T08:13:02.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:02.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:02.298 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:13:02.356 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:13:02.357 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:13:02.396 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:02.394+0000 7f06dd355600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:02.397 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:02.395+0000 7f06dd355600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:02.423 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:13:02.424 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:13:02.424 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:13:02.425 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:13:02.425 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:13:02.425 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:13:02.426 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:02.426 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:02.427 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:13:02.427 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:02.427 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:13:02.428 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:02.432 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:02.433 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:13:02.433 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:02.434 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:13:02.435 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:02.435 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:13:02.436 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:02.436 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:02.436 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:02.437 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:02.437 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:13:02.438 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:02.438 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:02.438 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:13:02.439 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:02.439 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:02.439 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:02.440 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:13:02.440 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:02.440 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:02.441 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:02.441 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:02.441 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:13:02.441 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:02.442 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:02.442 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:13:02.443 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:02.443 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:02.443 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:02.444 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:13:02.444 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:02.444 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:02.445 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:02.445 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:02.445 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:13:02.446 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:02.446 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:02.446 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:13:02.447 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:02.447 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:02.447 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:02.448 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:13:02.448 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:02.448 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:02.449 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:02.449 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:13:02.450 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:02.450 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:13:02.450 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:13:02.451 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:13:02.451 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:02.451 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:13:02Z" 2022-09-13T08:13:02.452 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:13:02.452 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:13:02Z" } 2022-09-13T08:13:02.453 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:13:02.454 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:02.454 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:02.455 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:13:06.446 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~67s 2022-09-13T08:13:07.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:07.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:07.457 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:13:07.514 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:13:07.514 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:13:07.560 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:07.557+0000 7f3380a45600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:07.561 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:07.558+0000 7f3380a45600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:07.580 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:13:07.580 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:13:07.581 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:13:07.581 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:13:07.581 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:13:07.582 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:13:07.582 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:07.582 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:07.583 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:13:07.583 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:07.583 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:13:07.583 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:07.584 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:07.584 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:13:07.584 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:07.585 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:13:07.585 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:07.585 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:13:07.586 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:07.586 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:07.586 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:07.586 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:07.587 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:13:07.587 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:07.587 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:07.588 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:13:07.588 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:07.588 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:07.589 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:07.589 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:13:07.589 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:07.589 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:07.590 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:07.590 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:07.591 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:13:07.591 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:07.591 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:07.592 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:13:07.592 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:07.592 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:07.593 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:07.593 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:13:07.593 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:07.594 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:07.594 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:07.594 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:07.595 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:13:07.595 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:07.596 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:07.596 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:13:07.596 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:07.597 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:07.597 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:07.597 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:13:07.597 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:07.598 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:07.598 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:07.599 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:13:07.599 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:07.599 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:13:07.600 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:13:07.600 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:13:07.600 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:07.601 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:13:07Z" 2022-09-13T08:13:07.601 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:13:07.602 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:13:07Z" } 2022-09-13T08:13:07.603 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:13:07.604 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:07.604 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:07.605 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:13:12.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~73s 2022-09-13T08:13:12.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:12.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:12.608 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:13:12.665 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:13:12.666 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:13:12.699 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:12.696+0000 7f3a60a31600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:12.700 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:12.697+0000 7f3a60a31600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:12.718 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:13:12.718 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:13:12.719 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:13:12.719 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:13:12.719 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:13:12.720 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:13:12.720 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:12.720 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:12.720 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:13:12.721 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:12.721 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:13:12.721 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:12.722 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:12.722 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:13:12.722 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:12.722 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:13:12.723 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:12.723 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:13:12.723 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:12.723 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:12.724 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:12.724 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:12.724 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:13:12.724 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:12.725 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:12.725 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:13:12.725 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:12.725 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:12.726 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:12.726 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:13:12.726 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:12.726 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:12.727 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:12.727 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:12.727 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:13:12.728 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:12.728 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:12.728 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:13:12.728 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:12.729 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:12.729 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:12.730 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:13:12.730 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:12.730 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:12.731 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:12.731 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:12.731 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:13:12.731 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:12.732 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:12.732 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:13:12.732 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:12.732 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:12.733 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:12.733 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:13:12.733 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:12.733 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:12.734 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:12.734 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:13:12.734 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:12.735 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:13:12.735 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:13:12.736 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:13:12.736 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:12.736 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:13:12Z" 2022-09-13T08:13:12.736 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:13:12.738 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:13:12Z" } 2022-09-13T08:13:12.739 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:13:12.739 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:12.740 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:12.740 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:13:17.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~78s 2022-09-13T08:13:17.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:17.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:17.743 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:13:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:13:17.802 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:13:17.850 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:17.846+0000 7f765dbf6600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:17.851 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:17.847+0000 7f765dbf6600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:17.869 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:13:17.869 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:13:17.870 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:13:17.870 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:13:17.870 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:13:17.871 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:13:17.871 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:17.871 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:17.872 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:13:17.872 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:17.872 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:13:17.873 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:17.873 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:17.873 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:13:17.873 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:17.874 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:13:17.874 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:17.874 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:13:17.874 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:17.875 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:17.875 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:17.875 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:17.876 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:13:17.876 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:17.876 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:17.877 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:13:17.877 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:17.877 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:17.877 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:17.878 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:13:17.878 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:17.878 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:17.879 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:17.879 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:17.879 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:13:17.879 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:17.880 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:17.880 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:13:17.880 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:17.880 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:17.881 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:17.881 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:13:17.881 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:17.882 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:17.882 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:17.882 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:17.883 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:13:17.883 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:17.883 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:17.883 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:13:17.884 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:17.884 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:17.884 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:17.884 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:13:17.885 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:17.885 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:17.885 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:17.885 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:13:17.886 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:17.886 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:13:17.886 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:13:17.887 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:13:17.887 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:17.887 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:13:17Z" 2022-09-13T08:13:17.887 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:13:17.889 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:13:17Z" } 2022-09-13T08:13:17.890 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:13:17.890 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:17.891 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:17.891 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:13:22.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:22.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:22.895 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:13:22.952 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:13:22.952 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:13:22.991 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:22.988+0000 7fe0defb3600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:22.992 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:22.989+0000 7fe0defb3600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:23.010 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:13:23.010 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:13:23.011 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:13:23.011 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:13:23.011 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:13:23.012 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:13:23.012 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:23.012 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:23.012 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:13:23.013 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:23.013 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:13:23.013 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:23.013 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:23.014 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:13:23.014 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:23.014 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:13:23.015 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:23.015 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:13:23.016 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:23.016 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:23.016 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:23.016 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:23.017 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:13:23.017 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:23.017 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:23.017 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:13:23.018 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:23.018 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:23.019 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:23.019 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:13:23.020 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:23.020 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:23.020 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:23.020 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:23.021 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:13:23.021 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:23.021 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:23.021 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:13:23.022 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:23.022 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:23.022 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:23.023 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:13:23.023 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:23.023 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:23.023 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:23.024 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:23.024 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:13:23.024 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:23.024 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:23.025 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:13:23.025 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:23.025 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:23.026 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:23.026 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:13:23.026 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:23.027 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:23.027 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:23.027 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:13:23.028 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:23.028 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:13:23.028 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:13:23.028 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:13:23.029 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:23.029 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:13:22Z" 2022-09-13T08:13:23.030 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:13:23.031 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:13:22Z" } 2022-09-13T08:13:23.032 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:13:23.033 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:23.033 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:23.033 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:13:23.264 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~84s 2022-09-13T08:13:28.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:28.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:28.037 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:13:28.095 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:13:28.095 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:13:28.150 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:28.147+0000 7f3608167600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:28.151 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:28.148+0000 7f3608167600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:28.171 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:13:28.172 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:13:28.172 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:13:28.172 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:13:28.173 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:13:28.173 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:13:28.173 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:28.173 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:28.174 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:13:28.174 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:28.174 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:13:28.174 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:28.175 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:28.175 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:13:28.175 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:28.175 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:13:28.176 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:28.176 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:13:28.176 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:28.176 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:28.177 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:28.177 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:28.177 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:13:28.177 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:28.178 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:28.178 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:13:28.178 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:28.178 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:28.179 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:28.179 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:13:28.179 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:28.179 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:28.180 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:28.180 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:28.180 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:13:28.180 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:28.181 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:28.181 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:13:28.181 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:28.181 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:28.182 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:28.182 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:13:28.182 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:28.183 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:28.183 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:28.183 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:28.183 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:13:28.184 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:28.184 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:28.184 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:13:28.184 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:28.185 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:28.185 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:28.185 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:13:28.185 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:28.185 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:28.186 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:28.186 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:13:28.186 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:28.187 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:13:28.187 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:13:28.187 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:13:28.187 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:28.188 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:13:28Z" 2022-09-13T08:13:28.188 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:13:28.189 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:13:28Z" } 2022-09-13T08:13:28.190 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:13:28.190 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:28.191 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:28.191 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:13:28.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~90s 2022-09-13T08:13:33.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:33.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:33.194 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:13:33.250 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:13:33.250 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:13:33.286 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:33.283+0000 7fd3b91df600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:33.287 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:33.284+0000 7fd3b91df600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:33.304 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:13:33.305 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:13:33.305 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:13:33.305 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:13:33.306 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:13:33.306 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:13:33.306 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:33.306 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:33.307 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:13:33.307 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:33.307 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:13:33.307 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:33.308 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:33.308 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:13:33.308 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:33.308 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:13:33.309 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:33.309 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:13:33.309 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:33.309 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:33.310 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:33.310 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:33.310 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:13:33.311 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:33.311 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:33.311 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:13:33.311 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:33.312 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:33.312 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:33.312 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:13:33.312 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:33.313 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:33.313 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:33.313 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:33.313 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:13:33.314 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:33.314 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:33.314 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:13:33.314 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:33.315 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:33.315 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:33.315 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:13:33.316 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:33.316 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:33.316 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:33.316 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:33.317 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:13:33.317 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:33.317 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:33.317 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:13:33.318 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:33.318 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:33.318 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:33.319 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:13:33.319 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:33.319 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:33.319 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:33.320 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:13:33.320 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:33.320 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:13:33.320 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:13:33.321 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:13:33.321 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:33.321 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:13:33Z" 2022-09-13T08:13:33.321 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:13:33.323 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:13:33Z" } 2022-09-13T08:13:33.324 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:13:33.324 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:33.325 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:33.325 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:13:34.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~95s 2022-09-13T08:13:38.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:38.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:38.329 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:13:38.386 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:13:38.386 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:13:38.420 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:38.418+0000 7fa7b167b600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:38.421 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:38.419+0000 7fa7b167b600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:38.437 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:13:38.438 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:13:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:13:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:13:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:13:38.440 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:13:38.440 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:13:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:13:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:13:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:13:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:13:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:38.446 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:38.446 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:13:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:38.448 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:38.448 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:13:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:38.450 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:13:38.450 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:38.451 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:38.451 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:38.452 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:38.452 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:13:38.452 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:38.453 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:38.453 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:13:38.454 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:38.454 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:38.454 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:38.455 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:13:38.455 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:38.456 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:38.456 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:38.456 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:38.457 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:13:38.457 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:38.458 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:38.458 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:13:38.458 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:38.459 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:38.459 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:38.460 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:13:38.460 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:38.460 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:38.461 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:38.461 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:13:38.462 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:38.462 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:13:38.462 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:13:38.463 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:13:38.463 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:38.464 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:13:38Z" 2022-09-13T08:13:38.464 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:13:38.466 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:13:38Z" } 2022-09-13T08:13:38.467 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:13:38.468 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:38.468 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:38.469 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:13:40.083 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~101s 2022-09-13T08:13:43.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:43.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:43.472 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:13:43.530 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:13:43.531 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:13:43.576 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:43.574+0000 7f3b27702600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:43.577 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:43.575+0000 7f3b27702600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:43.602 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:13:43.602 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:13:43.603 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:13:43.603 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:13:43.603 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:13:43.604 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:13:43.604 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:43.604 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:43.604 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:13:43.605 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:43.605 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:13:43.605 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:43.605 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:43.606 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:13:43.606 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:43.606 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:13:43.607 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:43.607 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:13:43.607 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:43.607 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:43.608 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:43.608 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:43.608 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:13:43.608 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:43.609 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:43.609 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:13:43.609 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:43.609 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:43.610 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:43.610 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:13:43.610 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:43.611 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:43.611 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:43.611 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:43.611 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:13:43.612 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:43.612 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:43.612 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:13:43.612 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:43.613 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:43.613 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:43.613 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:13:43.613 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:43.614 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:43.614 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:43.614 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:43.614 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:13:43.615 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:43.615 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:43.615 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:13:43.615 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:43.616 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:43.616 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:43.616 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:13:43.616 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:43.617 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:43.617 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:43.617 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:13:43.617 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:43.618 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:13:43.618 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:13:43.618 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:13:43.618 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:43.619 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:13:43Z" 2022-09-13T08:13:43.619 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:13:43.620 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:13:43Z" } 2022-09-13T08:13:43.621 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:13:43.621 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:43.622 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:43.622 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:13:45.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~107s 2022-09-13T08:13:48.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:48.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:48.625 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:13:48.683 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:13:48.683 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:13:48.730 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:48.726+0000 7f4f24e2f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:48.730 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:48.727+0000 7f4f24e2f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:48.746 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:13:48.747 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:13:48.747 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:13:48.747 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:13:48.747 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:13:48.748 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:13:48.748 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:48.748 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:48.748 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:13:48.749 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:48.749 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:13:48.749 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:48.749 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:48.750 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:13:48.751 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:48.751 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:13:48.752 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:48.752 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:13:48.753 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:48.753 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:48.753 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:48.754 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:48.754 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:13:48.754 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:48.755 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:48.755 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:13:48.755 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:48.755 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:48.755 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:48.756 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:13:48.756 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:48.756 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:48.756 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:48.757 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:48.757 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:13:48.757 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:48.757 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:48.758 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:13:48.758 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:48.758 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:48.758 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:48.759 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:13:48.759 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:48.760 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:48.760 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:48.760 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:48.760 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:13:48.761 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:48.761 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:48.761 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:13:48.761 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:48.762 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:48.762 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:48.762 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:13:48.763 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:48.771 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:48.772 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:48.772 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:13:48.773 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:48.773 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:13:48.774 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:13:48.774 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:13:48.775 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:48.775 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:13:48Z" 2022-09-13T08:13:48.776 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:13:48.778 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:13:48Z" } 2022-09-13T08:13:48.779 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:13:48.780 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:48.780 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:48.781 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:13:51.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~112s 2022-09-13T08:13:53.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:53.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:53.783 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:13:53.841 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:13:53.841 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:13:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:53.872+0000 7fe690cb3600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:53.873+0000 7fe690cb3600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:53.896 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:13:53.896 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:13:53.896 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:13:53.896 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:13:53.897 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:13:53.897 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:13:53.897 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:53.898 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:53.898 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:13:53.898 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:53.898 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:13:53.899 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:53.899 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:53.899 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:13:53.899 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:53.900 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:13:53.900 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:53.900 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:13:53.900 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:53.901 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:53.901 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:53.901 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:53.901 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:13:53.902 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:53.902 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:53.902 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:13:53.902 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:53.903 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:53.903 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:53.903 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:13:53.903 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:53.904 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:53.904 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:53.904 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:53.904 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:13:53.905 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:53.905 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:53.905 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:13:53.905 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:53.906 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:53.906 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:53.906 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:13:53.906 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:53.907 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:53.907 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:53.907 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:53.907 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:13:53.908 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:53.908 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:53.908 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:13:53.908 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:53.909 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:53.909 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:53.909 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:13:53.909 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:53.910 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:53.910 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:53.910 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:13:53.910 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:53.911 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:13:53.911 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:13:53.911 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:13:53.911 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:53.912 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:13:53Z" 2022-09-13T08:13:53.912 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:13:53.913 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:13:53Z" } 2022-09-13T08:13:53.914 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:13:53.914 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:53.915 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:53.915 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:13:56.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~118s 2022-09-13T08:13:58.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:58.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:13:58.918 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:13:58.976 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:13:58.977 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:13:59.021 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:59.018+0000 7f8da7f43600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:59.022 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:13:59.019+0000 7f8da7f43600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:13:59.038 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:13:59.039 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:13:59.039 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:13:59.039 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:13:59.040 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:13:59.040 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:13:59.040 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:59.041 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:59.041 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:13:59.041 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:59.041 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:13:59.042 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:59.042 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:59.042 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:13:59.042 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:59.043 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:13:59.043 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:59.043 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:13:59.044 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:59.044 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:59.044 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:59.045 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:59.045 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:13:59.045 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:59.045 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:59.046 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:13:59.046 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:59.046 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:59.046 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:59.047 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:13:59.047 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:59.047 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:59.047 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:59.048 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:59.048 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:13:59.048 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:59.048 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:59.049 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:13:59.049 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:59.050 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:59.050 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:59.050 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:13:59.050 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:59.051 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:59.051 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:59.051 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:13:59.052 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:13:59.052 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:13:59.052 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:13:59.053 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:13:59.053 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:13:59.053 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:13:59.053 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:13:59.054 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:13:59.054 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:13:59.054 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:59.054 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:13:59.055 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:13:59.055 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:59.055 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:13:59.056 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:13:59.056 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:13:59.056 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:13:59.056 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:13:59Z" 2022-09-13T08:13:59.057 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:13:59.058 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:13:59Z" } 2022-09-13T08:13:59.059 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:13:59.060 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:59.061 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:13:59.061 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:14:02.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~123s 2022-09-13T08:14:04.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:04.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:04.063 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:14:04.120 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:14:04.121 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:14:04.156 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:04.152+0000 7f5e2a85c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:04.157 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:04.153+0000 7f5e2a85c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:04.173 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:14:04.173 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:14:04.174 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:14:04.174 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:14:04.175 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:14:04.175 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:14:04.175 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:04.175 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:04.176 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:14:04.176 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:04.176 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:14:04.177 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:04.177 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:04.178 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:14:04.178 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:04.179 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:14:04.179 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:04.180 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:14:04.180 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:04.181 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:04.181 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:04.182 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:04.182 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:14:04.183 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:04.183 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:04.184 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:14:04.184 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:04.185 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:04.185 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:04.186 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:14:04.186 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:04.187 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:04.187 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:04.188 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:04.188 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:14:04.189 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:04.189 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:04.190 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:14:04.190 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:04.191 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:04.191 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:04.192 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:14:04.192 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:04.193 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:04.193 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:04.194 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:04.194 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:14:04.195 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:04.195 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:04.196 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:14:04.196 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:04.197 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:04.197 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:04.198 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:14:04.198 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:04.199 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:04.199 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:04.200 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:14:04.200 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:04.201 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:14:04.201 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:14:04.202 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:14:04.202 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:04.203 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:14:04Z" 2022-09-13T08:14:04.203 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:14:04.205 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:14:04Z" } 2022-09-13T08:14:04.206 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:14:04.207 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:04.208 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:04.209 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:14:08.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~129s 2022-09-13T08:14:09.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:09.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:09.212 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:14:09.269 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:14:09.270 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:14:09.322 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:09.318+0000 7ff8f6e46600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:09.322 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:09.319+0000 7ff8f6e46600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:09.342 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:14:09.342 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:14:09.343 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:14:09.343 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:14:09.343 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:14:09.344 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:14:09.344 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:09.344 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:09.345 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:14:09.345 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:09.345 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:14:09.346 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:09.346 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:09.346 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:14:09.347 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:09.347 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:14:09.347 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:09.347 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:14:09.348 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:09.348 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:09.348 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:09.348 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:09.349 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:14:09.349 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:09.349 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:09.349 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:14:09.350 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:09.350 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:09.350 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:09.350 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:14:09.351 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:09.351 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:09.351 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:09.352 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:09.352 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:14:09.352 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:09.352 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:09.353 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:14:09.353 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:09.353 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:09.353 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:09.354 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:14:09.354 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:09.354 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:09.354 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:09.355 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:09.355 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:14:09.355 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:09.355 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:09.356 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:14:09.356 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:09.356 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:09.356 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:09.357 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:14:09.357 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:09.357 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:09.357 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:09.358 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:14:09.358 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:09.358 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:14:09.358 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:14:09.359 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:14:09.359 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:09.359 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:14:09Z" 2022-09-13T08:14:09.359 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:14:09.361 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:14:09Z" } 2022-09-13T08:14:09.362 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:14:09.362 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:09.363 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:09.363 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:14:13.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~135s 2022-09-13T08:14:14.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:14.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:14.366 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:14:14.424 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:14:14.424 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:14:14.459 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:14.455+0000 7f5263f92600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:14.461 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:14.456+0000 7f5263f92600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:14.475 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:14:14.476 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:14:14.476 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:14:14.476 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:14:14.476 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:14:14.477 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:14:14.477 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:14.477 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:14.478 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:14:14.478 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:14.478 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:14:14.479 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:14.479 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:14.479 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:14:14.480 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:14.480 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:14:14.480 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:14.480 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:14:14.481 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:14.481 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:14.481 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:14.481 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:14.482 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:14:14.482 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:14.482 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:14.482 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:14:14.483 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:14.483 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:14.483 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:14.483 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:14:14.484 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:14.484 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:14.484 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:14.484 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:14.485 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:14:14.485 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:14.485 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:14.485 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:14:14.486 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:14.486 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:14.486 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:14.487 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:14:14.487 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:14.487 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:14.487 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:14.488 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:14.488 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:14:14.488 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:14.488 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:14.489 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:14:14.489 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:14.489 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:14.489 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:14.490 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:14:14.490 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:14.490 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:14.491 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:14.491 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:14:14.491 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:14.492 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:14:14.492 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:14:14.492 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:14:14.493 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:14.493 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:14:14Z" 2022-09-13T08:14:14.493 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:14:14.495 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:14:14Z" } 2022-09-13T08:14:14.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:14:14.496 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:14.497 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:14.497 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:14:19.322 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~140s 2022-09-13T08:14:19.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:19.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:19.501 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:14:19.559 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:14:19.560 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:14:19.607 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:19.604+0000 7fd6d1c99600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:19.608 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:19.606+0000 7fd6d1c99600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:19.628 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:14:19.628 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:14:19.628 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:14:19.629 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:14:19.629 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:14:19.629 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:14:19.630 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:19.630 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:19.631 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:14:19.631 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:19.632 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:14:19.632 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:19.632 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:19.633 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:14:19.634 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:19.634 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:14:19.634 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:19.635 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:14:19.635 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:19.635 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:19.636 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:19.636 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:19.636 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:14:19.636 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:19.636 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:19.637 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:14:19.637 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:19.637 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:19.637 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:19.638 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:14:19.638 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:19.638 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:19.639 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:19.639 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:19.639 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:14:19.640 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:19.640 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:19.640 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:14:19.640 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:19.641 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:19.641 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:19.641 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:14:19.641 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:19.642 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:19.642 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:19.642 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:19.643 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:14:19.643 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:19.643 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:19.643 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:14:19.643 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:19.644 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:19.644 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:19.644 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:14:19.645 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:19.645 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:19.645 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:19.646 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:14:19.646 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:19.646 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:14:19.647 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:14:19.647 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:14:19.647 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:19.647 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:14:19Z" 2022-09-13T08:14:19.648 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:14:19.649 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:14:19Z" } 2022-09-13T08:14:19.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:14:19.651 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:19.651 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:19.652 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:14:24.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:24.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:24.655 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:14:24.711 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:14:24.711 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:14:24.759 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:24.756+0000 7f710b79f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:24.760 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:24.757+0000 7f710b79f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:24.783 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:14:24.783 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:14:24.783 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:14:24.784 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:14:24.784 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:14:24.784 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:14:24.784 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:24.785 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:24.785 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:14:24.785 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:24.785 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:14:24.786 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:24.786 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:24.786 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:14:24.786 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:24.787 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:14:24.787 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:24.787 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:14:24.787 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:24.788 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:24.788 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:24.788 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:24.789 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:14:24.789 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:24.789 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:24.789 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:14:24.790 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:24.790 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:24.791 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:24.791 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:14:24.791 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:24.792 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:24.792 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:24.792 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:24.793 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:14:24.793 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:24.793 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:24.793 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:14:24.794 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:24.794 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:24.794 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:24.794 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:14:24.795 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:24.795 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:24.795 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:24.796 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:24.796 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:14:24.796 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:24.796 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:24.797 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:14:24.797 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:24.797 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:24.799 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:24.800 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:14:24.801 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:24.801 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:24.801 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:24.802 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:14:24.803 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:24.804 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:14:24.804 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:14:24.804 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:14:24.805 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:24.805 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:14:24Z" 2022-09-13T08:14:24.805 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:14:24.806 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:14:24Z" } 2022-09-13T08:14:24.807 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:14:24.808 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:24.809 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:24.810 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:14:24.929 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~146s 2022-09-13T08:14:29.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:29.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:29.813 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:14:29.870 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:14:29.870 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:14:29.905 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:29.901+0000 7ff968bee600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:29.906 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:29.902+0000 7ff968bee600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:29.925 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:14:29.926 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:14:29.926 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:14:29.926 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:14:29.927 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:14:29.927 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:14:29.927 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:29.927 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:29.928 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:14:29.928 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:29.928 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:14:29.928 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:29.929 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:29.929 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:14:29.929 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:29.929 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:14:29.930 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:29.930 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:14:29.930 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:29.930 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:29.931 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:29.931 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:29.931 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:14:29.931 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:29.932 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:29.932 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:14:29.932 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:29.933 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:29.933 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:29.933 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:14:29.933 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:29.934 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:29.934 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:29.934 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:29.934 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:14:29.935 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:29.935 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:29.935 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:14:29.936 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:29.936 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:29.936 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:29.937 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:14:29.937 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:29.937 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:29.937 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:29.938 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:29.938 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:14:29.938 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:29.938 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:29.939 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:14:29.939 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:29.939 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:29.939 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:29.940 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:14:29.940 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:29.940 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:29.940 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:29.941 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:14:29.941 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:29.941 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:14:29.941 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:14:29.942 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:14:29.942 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:29.942 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:14:29Z" 2022-09-13T08:14:29.943 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:14:29.944 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:14:29Z" } 2022-09-13T08:14:29.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:14:29.945 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:29.946 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:29.946 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:14:30.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~151s 2022-09-13T08:14:34.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:34.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:34.949 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:14:35.005 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:14:35.006 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:14:35.055 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:35.051+0000 7f53bdd4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:35.055 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:35.052+0000 7f53bdd4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:35.071 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:14:35.071 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:14:35.071 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:14:35.072 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:14:35.072 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:14:35.072 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:14:35.072 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:35.073 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:35.073 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:14:35.073 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:35.074 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:14:35.074 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:35.074 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:35.074 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:14:35.075 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:35.075 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:14:35.075 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:35.075 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:14:35.076 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:35.076 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:35.076 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:35.076 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:35.077 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:14:35.077 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:35.077 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:35.077 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:14:35.078 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:35.078 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:35.078 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:35.078 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:14:35.079 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:35.079 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:35.079 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:35.079 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:35.080 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:14:35.080 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:35.080 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:35.080 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:14:35.081 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:35.081 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:35.081 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:35.081 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:14:35.082 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:35.082 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:35.082 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:35.082 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:35.083 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:14:35.083 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:35.083 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:35.083 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:14:35.084 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:35.084 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:35.084 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:35.084 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:14:35.085 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:35.085 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:35.085 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:35.085 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:14:35.086 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:35.086 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:14:35.086 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:14:35.086 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:14:35.087 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:35.087 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:14:35Z" 2022-09-13T08:14:35.087 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:14:35.088 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:14:35Z" } 2022-09-13T08:14:35.089 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:14:35.090 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:35.090 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:35.090 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:14:36.154 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~157s 2022-09-13T08:14:40.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:40.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:40.094 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:14:40.151 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:14:40.152 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:14:40.205 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:40.201+0000 7f0701110600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:40.205 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:40.202+0000 7f0701110600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:40.221 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:14:40.221 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:14:40.221 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:14:40.222 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:14:40.222 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:14:40.222 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:14:40.223 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:40.223 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:40.223 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:14:40.224 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:40.224 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:14:40.224 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:40.224 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:40.225 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:14:40.225 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:40.225 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:14:40.225 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:40.226 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:14:40.226 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:40.226 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:40.226 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:40.227 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:40.227 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:14:40.227 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:40.228 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:40.228 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:14:40.228 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:40.228 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:40.229 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:40.229 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:14:40.229 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:40.229 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:40.230 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:40.230 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:40.230 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:14:40.231 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:40.231 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:40.231 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:14:40.231 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:40.232 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:40.232 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:40.232 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:14:40.232 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:40.233 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:40.233 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:40.233 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:40.233 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:14:40.234 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:40.234 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:40.234 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:14:40.234 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:40.235 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:40.235 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:40.235 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:14:40.235 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:40.236 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:40.236 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:40.236 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:14:40.236 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:40.237 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:14:40.237 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:14:40.237 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:14:40.237 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:40.238 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:14:40Z" 2022-09-13T08:14:40.238 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:14:40.239 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:14:40Z" } 2022-09-13T08:14:40.240 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:14:40.240 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:40.241 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:40.241 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:14:41.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~163s 2022-09-13T08:14:45.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:45.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:45.244 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:14:45.301 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:14:45.302 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:14:45.336 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:45.333+0000 7fe73bed4600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:45.337 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:45.334+0000 7fe73bed4600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:45.357 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:14:45.358 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:14:45.358 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:14:45.359 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:14:45.359 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:14:45.359 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:14:45.359 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:45.360 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:45.360 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:14:45.360 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:45.360 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:14:45.361 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:45.361 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:45.361 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:14:45.361 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:45.361 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:14:45.362 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:45.362 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:14:45.362 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:45.363 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:45.363 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:45.363 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:45.363 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:14:45.364 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:45.364 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:45.364 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:14:45.364 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:45.365 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:45.365 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:45.365 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:14:45.366 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:45.367 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:45.367 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:45.367 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:45.367 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:14:45.368 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:45.368 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:45.369 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:14:45.369 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:45.370 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:45.370 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:45.371 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:14:45.372 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:45.372 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:45.373 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:45.373 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:45.374 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:14:45.374 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:45.375 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:45.375 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:14:45.376 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:45.376 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:45.377 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:45.377 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:14:45.378 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:45.378 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:45.379 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:45.379 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:14:45.380 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:45.380 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:14:45.381 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:14:45.381 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:14:45.382 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:45.382 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:14:45Z" 2022-09-13T08:14:45.383 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:14:45.385 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:14:45Z" } 2022-09-13T08:14:45.386 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:14:45.387 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:45.388 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:45.389 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:14:47.366 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~168s 2022-09-13T08:14:50.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:50.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:50.392 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:14:50.449 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:14:50.449 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:14:50.496 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:50.492+0000 7f1be223f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:50.497 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:50.493+0000 7f1be223f600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:50.518 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:14:50.518 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:14:50.519 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:14:50.519 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:14:50.519 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:14:50.519 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:14:50.520 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:50.520 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:50.520 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:14:50.520 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:50.521 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:14:50.521 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:50.521 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:50.521 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:14:50.522 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:50.522 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:14:50.522 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:50.522 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:14:50.523 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:50.523 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:50.523 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:50.523 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:50.524 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:14:50.524 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:50.524 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:50.524 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:14:50.525 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:50.525 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:50.525 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:50.525 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:14:50.525 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:50.526 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:50.526 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:50.526 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:50.526 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:14:50.527 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:50.527 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:50.527 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:14:50.527 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:50.528 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:50.528 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:50.528 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:14:50.528 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:50.529 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:50.529 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:50.529 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:50.529 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:14:50.530 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:50.530 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:50.530 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:14:50.530 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:50.531 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:50.531 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:50.531 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:14:50.531 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:50.531 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:50.532 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:50.532 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:14:50.532 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:50.532 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:14:50.533 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:14:50.533 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:14:50.533 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:50.533 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:14:50Z" 2022-09-13T08:14:50.534 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:14:50.535 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:14:50Z" } 2022-09-13T08:14:50.536 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:14:50.536 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:50.537 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:50.537 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:14:52.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~174s 2022-09-13T08:14:55.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:55.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:14:55.540 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:14:55.595 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:14:55.596 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:14:55.649 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:55.646+0000 7efe49817600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:55.651 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:14:55.647+0000 7efe49817600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:14:55.668 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:14:55.668 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:14:55.669 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:14:55.669 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:14:55.669 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:14:55.669 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:14:55.670 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:55.670 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:55.670 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:14:55.671 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:55.671 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:14:55.671 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:55.672 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:55.672 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:14:55.672 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:55.673 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:14:55.673 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:55.673 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:14:55.674 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:55.674 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:55.674 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:55.674 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:55.675 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:14:55.675 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:55.675 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:55.676 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:14:55.676 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:55.676 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:55.676 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:55.677 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:14:55.677 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:55.677 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:55.678 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:55.678 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:55.678 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:14:55.679 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:55.679 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:55.679 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:14:55.680 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:55.680 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:55.680 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:55.680 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:14:55.681 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:55.681 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:55.681 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:55.682 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:14:55.682 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:14:55.682 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:14:55.683 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:14:55.683 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:14:55.683 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:14:55.684 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:14:55.684 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:14:55.684 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:14:55.684 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:14:55.685 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:55.685 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:14:55.685 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:14:55.685 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:55.686 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:14:55.686 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:14:55.686 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:14:55.686 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:14:55.687 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:14:55Z" 2022-09-13T08:14:55.687 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:14:55.688 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:14:55Z" } 2022-09-13T08:14:55.689 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:14:55.690 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:55.690 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:14:55.691 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:14:58.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~179s 2022-09-13T08:15:00.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:00.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:00.694 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:15:00.751 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:15:00.751 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:15:00.784 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:00.781+0000 7f8474590600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:00.785 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:00.782+0000 7f8474590600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:00.805 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:15:00.805 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:15:00.805 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:15:00.806 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:15:00.806 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:15:00.806 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:15:00.807 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:00.807 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:00.807 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:15:00.807 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:00.808 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:15:00.808 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:00.808 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:00.808 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:15:00.809 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:00.809 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:15:00.809 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:00.810 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:15:00.810 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:00.810 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:00.810 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:00.811 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:00.811 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:15:00.811 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:00.811 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:00.812 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:15:00.812 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:00.812 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:00.812 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:00.813 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:15:00.813 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:00.813 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:00.813 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:00.814 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:00.814 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:15:00.814 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:00.814 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:00.815 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:15:00.815 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:00.815 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:00.815 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:00.816 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:15:00.816 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:00.816 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:00.816 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:00.817 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:00.817 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:15:00.817 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:00.817 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:00.818 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:15:00.818 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:00.818 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:00.818 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:00.819 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:15:00.819 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:00.819 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:00.819 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:00.820 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:15:00.820 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:00.820 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:15:00.820 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:15:00.821 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:15:00.821 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:00.821 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:15:00Z" 2022-09-13T08:15:00.821 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:15:00.823 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:15:00Z" } 2022-09-13T08:15:00.824 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:15:00.824 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:00.825 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:00.825 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:15:04.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~185s 2022-09-13T08:15:05.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:05.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:05.828 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:15:05.884 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:15:05.885 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:15:05.929 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:05.926+0000 7f7f7526a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:05.930 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:05.927+0000 7f7f7526a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:05.954 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:15:05.954 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:15:05.955 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:15:05.955 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:15:05.956 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:15:05.956 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:15:05.956 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:05.957 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:05.957 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:15:05.958 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:05.958 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:15:05.959 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:05.959 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:05.960 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:15:05.960 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:05.960 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:15:05.961 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:05.961 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:15:05.962 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:05.962 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:05.963 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:05.963 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:05.964 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:15:05.964 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:05.964 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:05.965 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:15:05.965 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:05.966 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:05.966 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:05.967 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:15:05.967 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:05.968 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:05.968 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:05.968 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:05.969 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:15:05.969 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:05.970 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:05.970 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:15:05.971 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:05.971 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:05.972 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:05.972 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:15:05.972 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:05.973 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:05.973 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:05.974 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:05.974 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:15:05.975 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:05.975 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:05.976 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:15:05.976 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:05.976 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:05.977 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:05.978 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:15:05.978 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:05.978 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:05.979 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:05.979 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:15:05.980 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:05.980 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:15:05.981 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:15:05.981 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:15:05.981 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:05.982 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:15:05Z" 2022-09-13T08:15:05.982 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:15:05.984 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:15:05Z" } 2022-09-13T08:15:05.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:15:05.986 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:05.987 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:05.988 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:15:09.789 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~191s 2022-09-13T08:15:10.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:10.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:10.991 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:15:11.049 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:15:11.049 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:15:11.100 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:11.098+0000 7f82fea20600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:11.101 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:11.099+0000 7f82fea20600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:11.118 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:15:11.118 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:15:11.119 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:15:11.120 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:15:11.120 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:15:11.120 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:15:11.120 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:11.121 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:11.121 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:15:11.121 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:11.121 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:15:11.122 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:11.122 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:11.122 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:15:11.122 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:11.123 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:15:11.123 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:11.123 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:15:11.123 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:11.124 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:11.124 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:11.124 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:11.124 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:15:11.125 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:11.125 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:11.125 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:15:11.125 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:11.126 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:11.126 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:11.126 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:15:11.126 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:11.127 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:11.127 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:11.127 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:11.128 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:15:11.128 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:11.128 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:11.128 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:15:11.128 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:11.129 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:11.129 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:11.129 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:15:11.130 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:11.130 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:11.130 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:11.130 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:11.131 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:15:11.131 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:11.131 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:11.131 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:15:11.132 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:11.132 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:11.132 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:11.132 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:15:11.133 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:11.133 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:11.133 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:11.133 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:15:11.134 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:11.134 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:15:11.134 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:15:11.134 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:15:11.135 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:11.135 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:15:11Z" 2022-09-13T08:15:11.135 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:15:11.137 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:15:11Z" } 2022-09-13T08:15:11.137 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:15:11.138 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:11.138 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:11.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:15:15.395 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~196s 2022-09-13T08:15:16.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:16.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:16.142 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:15:16.200 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:15:16.200 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:15:16.235 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:16.231+0000 7f4ceb7d7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:16.236 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:16.232+0000 7f4ceb7d7600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:16.253 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:15:16.253 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:15:16.254 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:15:16.254 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:15:16.254 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:15:16.254 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:15:16.255 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:16.255 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:16.255 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:15:16.256 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:16.256 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:15:16.256 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:16.256 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:16.257 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:15:16.257 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:16.257 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:15:16.257 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:16.258 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:15:16.258 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:16.258 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:16.258 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:16.259 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:16.259 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:15:16.259 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:16.259 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:16.260 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:15:16.260 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:16.260 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:16.261 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:16.261 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:15:16.261 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:16.261 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:16.262 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:16.262 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:16.262 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:15:16.262 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:16.263 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:16.263 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:15:16.263 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:16.264 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:16.264 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:16.264 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:15:16.264 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:16.265 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:16.265 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:16.265 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:16.265 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:15:16.266 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:16.266 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:16.266 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:15:16.266 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:16.267 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:16.267 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:16.267 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:15:16.267 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:16.268 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:16.268 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:16.268 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:15:16.268 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:16.269 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:15:16.269 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:15:16.269 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:15:16.269 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:16.270 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:15:16Z" 2022-09-13T08:15:16.270 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:15:16.271 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:15:16Z" } 2022-09-13T08:15:16.272 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:15:16.272 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:16.273 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:16.273 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:15:21.002 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~202s 2022-09-13T08:15:21.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:21.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:21.277 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:15:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:15:21.336 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:15:21.384 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:21.380+0000 7fc9d6cac600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:21.385 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:21.381+0000 7fc9d6cac600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:21.402 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:15:21.402 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:15:21.403 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:15:21.403 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:15:21.403 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:15:21.403 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:15:21.404 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:21.404 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:21.404 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:15:21.404 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:21.405 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:15:21.405 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:21.405 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:21.405 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:15:21.406 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:21.406 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:15:21.406 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:21.406 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:15:21.407 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:21.407 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:21.407 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:21.408 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:21.408 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:15:21.408 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:21.408 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:21.409 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:15:21.409 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:21.409 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:21.409 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:21.410 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:15:21.410 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:21.410 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:21.411 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:21.411 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:21.411 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:15:21.411 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:21.412 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:21.412 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:15:21.412 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:21.412 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:21.412 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:21.413 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:15:21.413 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:21.413 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:21.413 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:21.414 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:21.414 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:15:21.414 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:21.414 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:21.415 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:15:21.415 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:21.415 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:21.415 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:21.416 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:15:21.416 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:21.416 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:21.416 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:21.417 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:15:21.417 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:21.417 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:15:21.417 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:15:21.417 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:15:21.418 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:21.418 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:15:21Z" 2022-09-13T08:15:21.419 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:15:21.420 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:15:21Z" } 2022-09-13T08:15:21.420 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:15:21.421 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:21.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:21.422 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:15:26.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:26.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:26.425 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:15:26.482 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:15:26.483 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:15:26.517 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:26.513+0000 7f39ac91d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:26.518 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:26.515+0000 7f39ac91d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:26.533 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:15:26.534 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:15:26.534 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:15:26.534 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:15:26.535 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:15:26.535 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:15:26.535 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:26.535 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:26.536 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:15:26.536 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:26.536 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:15:26.536 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:26.537 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:26.537 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:15:26.537 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:26.537 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:15:26.538 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:26.538 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:15:26.539 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:26.539 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:26.539 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:26.539 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:26.540 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:15:26.540 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:26.540 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:26.541 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:15:26.541 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:26.542 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:26.543 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:26.543 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:15:26.543 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:26.543 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:26.544 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:26.544 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:26.544 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:15:26.544 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:26.545 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:26.545 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:15:26.545 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:26.545 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:26.546 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:26.546 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:15:26.546 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:26.556 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:26.557 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:26.557 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:26.558 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:15:26.558 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:26.559 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:26.559 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:15:26.560 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:26.560 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:26.561 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:26.561 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:15:26.561 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:26.562 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:26.562 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:26.562 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:15:26.562 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:26.563 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:15:26.563 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:15:26.563 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:15:26.564 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:26.564 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:15:26Z" 2022-09-13T08:15:26.564 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:15:26.566 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:15:26Z" } 2022-09-13T08:15:26.566 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:15:26.567 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:26.568 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:26.568 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:15:26.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~207s 2022-09-13T08:15:31.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:31.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:31.571 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:15:31.628 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:15:31.628 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:15:31.683 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:31.680+0000 7fd585f4e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:31.685 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:31.681+0000 7fd585f4e600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:31.704 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:15:31.705 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:15:31.706 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:15:31.706 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:15:31.707 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:15:31.707 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:15:31.708 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:31.708 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:31.709 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:15:31.709 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:31.710 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:15:31.710 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:31.711 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:31.711 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:15:31.712 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:31.712 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:15:31.712 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:31.712 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:15:31.713 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:31.713 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:31.713 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:31.713 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:31.714 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:15:31.714 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:31.714 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:31.715 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:15:31.715 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:31.715 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:31.716 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:31.716 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:15:31.716 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:31.716 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:31.717 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:31.717 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:31.717 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:15:31.717 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:31.718 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:31.718 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:15:31.719 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:31.719 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:31.719 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:31.720 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:15:31.720 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:31.720 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:31.721 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:31.721 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:31.721 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:15:31.721 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:31.722 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:31.722 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:15:31.722 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:31.722 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:31.723 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:31.723 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:15:31.723 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:31.723 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:15:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:31.724 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:15:31.725 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:15:31.725 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:15:31.725 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:31.726 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:15:31Z" 2022-09-13T08:15:31.726 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:15:31.727 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:15:31Z" } 2022-09-13T08:15:31.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:15:31.728 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:31.729 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:31.729 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:15:32.230 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~213s 2022-09-13T08:15:36.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:36.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:36.733 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:15:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:15:36.790 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:15:36.828 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:36.824+0000 7f81b7eef600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:36.826+0000 7f81b7eef600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:36.844 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:15:36.845 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:15:36.845 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:15:36.845 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:15:36.846 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:15:36.846 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:15:36.846 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:36.847 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:36.847 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:15:36.847 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:36.847 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:15:36.848 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:36.848 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:36.848 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:15:36.848 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:36.849 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:15:36.849 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:36.849 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:15:36.849 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:36.850 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:36.850 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:36.850 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:36.850 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:15:36.851 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:36.851 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:36.851 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:15:36.852 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:36.853 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:36.853 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:36.854 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:15:36.854 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:36.855 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:36.855 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:36.856 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:36.856 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:15:36.856 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:36.857 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:36.857 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:15:36.858 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:36.858 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:36.858 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:36.859 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:15:36.859 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:36.860 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:36.860 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:36.861 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:36.861 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:15:36.862 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:36.862 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:36.862 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:15:36.862 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:36.863 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:36.863 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:36.863 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:15:36.863 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:36.864 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:36.864 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:36.864 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:15:36.865 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:36.865 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:15:36.866 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:15:36.866 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:15:36.867 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:36.867 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:15:36Z" 2022-09-13T08:15:36.868 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:15:36.869 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:15:36Z" } 2022-09-13T08:15:36.870 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:15:36.871 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:36.872 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:36.873 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:15:37.839 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~219s 2022-09-13T08:15:41.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:41.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:41.876 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:15:41.934 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:15:41.935 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:15:41.968 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:41.964+0000 7fad411e4600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:41.969 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:41.965+0000 7fad411e4600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:41.987 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:15:41.988 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:15:41.988 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:15:41.989 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:15:41.989 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:15:41.990 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:15:41.991 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:41.991 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:41.991 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:15:41.992 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:41.992 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:15:41.993 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:41.993 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:41.994 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:15:41.994 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:41.995 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:15:41.995 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:41.996 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:15:41.996 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:41.997 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:41.997 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:41.998 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:41.998 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:15:41.999 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:42.000 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:42.000 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:15:42.001 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:42.001 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:42.002 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:42.002 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:15:42.003 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:42.003 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:42.004 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:42.004 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:42.005 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:15:42.005 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:42.006 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:42.006 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:15:42.007 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:42.007 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:42.008 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:42.009 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:15:42.009 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:42.010 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:42.010 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:42.011 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:42.011 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:15:42.012 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:42.012 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:42.013 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:15:42.013 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:42.014 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:42.014 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:42.015 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:15:42.015 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:42.016 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:42.016 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:42.017 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:15:42.018 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:42.018 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:15:42.019 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:15:42.019 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:15:42.020 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:42.020 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:15:41Z" 2022-09-13T08:15:42.021 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:15:42.023 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:15:41Z" } 2022-09-13T08:15:42.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:15:42.025 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:42.026 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:42.027 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:15:43.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~224s 2022-09-13T08:15:47.029 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:47.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:47.030 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:15:47.088 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:15:47.088 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:15:47.129 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:47.126+0000 7ff60ad5a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:47.130 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:47.127+0000 7ff60ad5a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:47.147 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:15:47.147 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:15:47.147 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:15:47.148 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:15:47.148 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:15:47.148 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:15:47.149 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:47.149 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:47.149 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:15:47.149 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:47.150 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:15:47.150 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:47.150 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:47.151 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:15:47.151 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:47.151 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:15:47.151 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:47.152 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:15:47.152 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:47.152 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:47.152 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:47.153 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:47.153 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:15:47.153 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:47.154 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:47.154 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:15:47.154 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:47.154 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:47.155 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:47.155 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:15:47.155 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:47.155 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:47.156 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:47.156 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:47.156 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:15:47.157 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:47.157 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:47.157 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:15:47.157 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:47.158 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:47.158 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:47.158 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:15:47.158 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:47.159 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:47.159 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:47.159 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:47.159 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:15:47.160 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:47.160 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:47.160 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:15:47.160 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:47.161 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:47.161 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:47.161 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:15:47.161 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:47.162 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:47.171 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:47.172 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:15:47.172 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:47.172 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:15:47.173 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:15:47.173 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:15:47.173 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:47.173 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:15:47Z" 2022-09-13T08:15:47.174 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:15:47.175 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:15:47Z" } 2022-09-13T08:15:47.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:15:47.176 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:47.177 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:47.177 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:15:49.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~230s 2022-09-13T08:15:52.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:52.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:52.183 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:15:52.240 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:15:52.240 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:15:52.282 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:52.279+0000 7fc142ae2600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:52.283 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:52.280+0000 7fc142ae2600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:52.301 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:15:52.302 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:15:52.303 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:15:52.303 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:15:52.303 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:15:52.304 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:15:52.304 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:52.304 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:52.305 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:15:52.305 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:52.305 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:15:52.305 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:52.305 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:52.306 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:15:52.306 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:52.307 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:15:52.307 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:52.307 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:15:52.307 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:52.308 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:52.308 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:52.308 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:52.308 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:15:52.309 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:52.309 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:52.309 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:15:52.309 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:52.310 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:52.310 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:52.310 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:15:52.310 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:52.311 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:52.311 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:52.311 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:52.311 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:15:52.312 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:52.312 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:52.312 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:15:52.313 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:52.313 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:52.313 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:52.313 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:15:52.314 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:52.314 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:52.314 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:52.314 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:52.315 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:15:52.315 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:52.315 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:52.315 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:15:52.316 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:52.316 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:52.316 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:52.316 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:15:52.317 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:52.317 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:52.317 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:52.317 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:15:52.318 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:52.318 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:15:52.318 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:15:52.318 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:15:52.319 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:52.319 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:15:52Z" 2022-09-13T08:15:52.319 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:15:52.320 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:15:52Z" } 2022-09-13T08:15:52.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:15:52.321 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:52.322 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:52.322 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:15:54.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~235s 2022-09-13T08:15:57.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:57.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:15:57.325 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:15:57.385 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:15:57.386 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:15:57.429 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:57.426+0000 7f74d7ad1600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:57.431 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:15:57.427+0000 7f74d7ad1600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:15:57.448 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:15:57.449 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:15:57.449 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:15:57.449 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:15:57.450 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:15:57.450 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:15:57.450 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:57.451 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:57.451 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:15:57.451 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:57.451 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:15:57.452 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:57.452 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:57.452 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:15:57.453 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:57.453 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:15:57.453 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:57.454 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:15:57.454 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:57.454 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:57.455 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:57.455 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:57.455 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:15:57.456 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:57.456 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:57.456 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:15:57.456 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:57.457 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:57.457 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:57.457 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:15:57.458 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:57.458 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:57.458 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:57.459 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:57.459 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:15:57.459 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:57.460 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:57.460 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:15:57.460 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:57.460 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:57.461 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:57.461 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:15:57.461 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:57.462 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:57.462 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:57.462 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:15:57.463 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:15:57.463 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:15:57.463 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:15:57.464 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:15:57.464 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:15:57.464 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:15:57.464 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:15:57.465 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:15:57.465 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:15:57.466 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:57.466 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:15:57.466 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:15:57.466 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:57.467 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:15:57.467 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:15:57.468 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:15:57.468 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:15:57.468 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:15:57Z" 2022-09-13T08:15:57.469 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:15:57.471 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:15:57Z" } 2022-09-13T08:15:57.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:15:57.473 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:57.473 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:15:57.474 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:16:00.264 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~241s 2022-09-13T08:16:02.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:02.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:02.477 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:16:02.534 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:16:02.535 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:16:02.587 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:02.583+0000 7f70e27a3600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:02.587 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:02.584+0000 7f70e27a3600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:02.612 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:16:02.613 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:16:02.613 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:16:02.613 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:16:02.613 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:16:02.614 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:16:02.614 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:02.614 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:02.615 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:16:02.615 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:02.615 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:16:02.615 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:02.615 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:02.616 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:16:02.616 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:02.616 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:16:02.616 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:02.617 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:16:02.617 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:02.617 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:02.617 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:02.618 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:02.618 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:16:02.618 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:02.619 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:02.619 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:16:02.619 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:02.619 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:02.619 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:02.620 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:16:02.620 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:02.620 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:02.620 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:02.621 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:02.621 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:16:02.621 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:02.621 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:02.622 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:16:02.622 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:02.622 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:02.623 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:02.623 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:16:02.623 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:02.623 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:02.624 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:02.624 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:02.624 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:16:02.624 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:02.625 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:02.625 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:16:02.625 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:02.625 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:02.626 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:02.626 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:16:02.626 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:02.627 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:02.627 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:02.627 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:16:02.627 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:02.628 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:16:02.628 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:16:02.628 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:16:02.628 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:02.629 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:16:02Z" 2022-09-13T08:16:02.629 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:16:02.630 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:16:02Z" } 2022-09-13T08:16:02.631 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:16:02.631 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:02.632 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:02.632 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:16:05.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~247s 2022-09-13T08:16:07.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:07.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:07.635 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:16:07.692 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:16:07.693 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:16:07.726 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:07.723+0000 7fd306271600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:07.727 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:07.724+0000 7fd306271600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:07.743 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:16:07.744 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:16:07.745 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:16:07.745 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:16:07.746 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:16:07.746 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:16:07.747 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:07.747 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:07.748 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:16:07.748 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:07.748 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:16:07.749 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:07.749 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:07.750 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:16:07.750 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:07.750 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:16:07.751 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:07.751 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:16:07.752 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:07.752 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:07.752 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:07.753 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:07.753 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:16:07.754 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:07.754 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:07.755 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:16:07.756 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:07.756 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:07.757 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:07.757 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:16:07.758 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:07.759 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:07.759 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:07.759 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:07.760 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:16:07.760 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:07.760 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:07.760 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:16:07.761 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:07.761 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:07.761 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:07.761 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:16:07.762 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:07.762 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:07.763 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:07.763 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:07.763 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:16:07.763 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:07.764 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:07.764 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:16:07.764 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:07.764 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:07.765 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:07.765 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:16:07.765 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:07.765 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:07.766 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:07.766 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:16:07.766 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:07.766 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:16:07.767 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:16:07.767 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:16:07.767 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:07.768 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:16:07Z" 2022-09-13T08:16:07.768 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:16:07.769 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:16:07Z" } 2022-09-13T08:16:07.770 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:16:07.770 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:07.771 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:07.771 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:16:11.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~252s 2022-09-13T08:16:12.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:12.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:12.775 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:16:12.832 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:16:12.833 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:16:12.889 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:12.885+0000 7f17e226c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:12.890 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:12.886+0000 7f17e226c600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:12.913 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:16:12.913 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:16:12.914 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:16:12.914 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:16:12.914 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:16:12.914 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:16:12.915 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:12.915 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:12.915 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:16:12.915 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:12.916 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:16:12.916 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:12.916 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:12.916 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:16:12.917 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:12.917 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:16:12.917 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:12.917 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:16:12.918 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:12.918 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:12.918 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:12.918 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:12.919 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:16:12.919 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:12.919 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:12.919 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:16:12.920 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:12.920 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:12.920 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:12.920 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:16:12.921 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:12.921 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:12.921 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:12.921 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:12.921 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:16:12.922 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:12.922 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:12.922 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:16:12.922 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:12.923 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:12.923 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:12.923 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:16:12.923 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:12.924 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:12.924 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:12.924 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:12.924 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:16:12.925 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:12.925 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:12.925 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:16:12.925 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:12.926 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:12.926 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:12.926 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:16:12.926 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:12.927 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:12.927 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:12.927 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:16:12.927 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:12.927 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:16:12.928 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:16:12.928 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:16:12.928 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:12.928 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:16:12Z" 2022-09-13T08:16:12.929 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:16:12.930 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:16:12Z" } 2022-09-13T08:16:12.931 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:16:12.931 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:12.932 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:12.932 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:16:17.083 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~258s 2022-09-13T08:16:17.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:17.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:17.934 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:16:17.990 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:16:17.990 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:16:18.026 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:18.023+0000 7fc231b4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:18.028 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:18.024+0000 7fc231b4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:18.042 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:16:18.043 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:16:18.043 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:16:18.044 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:16:18.044 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:16:18.044 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:16:18.044 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:18.045 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:18.045 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:16:18.045 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:18.046 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:16:18.046 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:18.046 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:18.047 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:16:18.047 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:18.047 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:16:18.047 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:18.048 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:16:18.048 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:18.048 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:18.048 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:18.049 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:18.049 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:16:18.049 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:18.050 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:18.050 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:16:18.051 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:18.051 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:18.051 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:18.052 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:16:18.052 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:18.053 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:18.053 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:18.053 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:18.054 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:16:18.054 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:18.054 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:18.055 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:16:18.055 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:18.055 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:18.056 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:18.056 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:16:18.056 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:18.056 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:18.057 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:18.057 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:18.057 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:16:18.058 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:18.058 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:18.058 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:16:18.058 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:18.059 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:18.059 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:18.059 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:16:18.060 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:18.060 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:18.060 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:18.060 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:16:18.061 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:18.061 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:16:18.062 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:16:18.062 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:16:18.062 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:18.062 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:16:18Z" 2022-09-13T08:16:18.063 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:16:18.064 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:16:18Z" } 2022-09-13T08:16:18.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:16:18.066 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:18.066 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:18.067 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:16:22.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~264s 2022-09-13T08:16:23.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:23.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:23.071 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:16:23.125 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:16:23.125 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:16:23.173 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:23.169+0000 7ff389905600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:23.173 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:23.170+0000 7ff389905600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:23.197 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:16:23.197 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:16:23.197 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:16:23.198 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:16:23.198 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:16:23.198 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:16:23.199 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:23.199 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:23.199 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:16:23.199 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:23.200 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:16:23.200 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:23.200 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:23.201 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:16:23.201 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:23.201 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:16:23.201 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:23.202 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:16:23.202 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:23.202 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:23.203 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:23.203 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:23.203 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:16:23.203 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:23.204 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:23.204 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:16:23.204 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:23.205 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:23.205 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:23.205 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:16:23.205 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:23.206 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:23.206 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:23.206 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:23.206 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:16:23.207 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:23.207 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:23.207 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:16:23.208 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:23.208 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:23.208 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:23.208 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:16:23.209 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:23.209 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:23.209 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:23.209 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:23.210 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:16:23.210 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:23.210 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:23.211 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:16:23.211 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:23.211 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:23.211 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:23.212 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:16:23.212 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:23.212 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:23.212 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:23.213 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:16:23.213 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:23.213 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:16:23.214 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:16:23.214 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:16:23.214 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:23.214 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:16:23Z" 2022-09-13T08:16:23.215 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:16:23.216 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:16:23Z" } 2022-09-13T08:16:23.217 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:16:23.217 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:23.218 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:23.218 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:16:28.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:28.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:28.223 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:16:28.283 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:16:28.283 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:16:28.296 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~269s 2022-09-13T08:16:28.333 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:28.330+0000 7f9c97f3a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:28.334 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:28.331+0000 7f9c97f3a600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:28.349 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:16:28.350 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:16:28.350 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:16:28.351 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:16:28.351 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:16:28.352 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:16:28.352 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:28.353 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:28.353 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:16:28.354 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:28.354 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:16:28.355 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:28.355 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:28.356 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:16:28.356 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:28.357 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:16:28.357 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:28.358 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:16:28.358 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:28.358 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:28.358 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:28.359 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:28.359 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:16:28.359 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:28.359 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:28.360 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:16:28.360 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:28.361 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:28.361 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:28.361 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:16:28.362 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:28.362 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:28.363 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:28.363 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:28.364 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:16:28.364 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:28.365 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:28.365 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:16:28.365 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:28.366 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:28.366 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:28.367 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:16:28.367 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:28.368 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:28.368 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:28.368 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:28.369 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:16:28.369 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:28.370 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:28.370 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:16:28.370 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:28.370 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:28.371 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:28.371 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:16:28.371 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:28.371 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:28.372 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:28.372 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:16:28.372 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:28.372 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:16:28.373 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:16:28.373 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:16:28.373 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:28.373 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:16:28Z" 2022-09-13T08:16:28.374 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:16:28.375 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:16:28Z" } 2022-09-13T08:16:28.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:16:28.376 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:28.377 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:28.377 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:16:33.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:33.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:33.380 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:16:33.438 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:16:33.439 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:16:33.471 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:33.468+0000 7fedd52b8600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:33.472 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:33.470+0000 7fedd52b8600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:33.489 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:16:33.489 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:16:33.490 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:16:33.490 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:16:33.490 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:16:33.491 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:16:33.491 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:33.491 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:33.491 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:16:33.492 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:33.492 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:16:33.492 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:33.492 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:33.493 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:16:33.493 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:33.494 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:16:33.494 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:33.494 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:16:33.494 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:33.495 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:33.495 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:33.495 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:33.495 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:16:33.496 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:33.496 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:33.496 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:16:33.497 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:33.497 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:33.498 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:33.498 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:16:33.498 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:33.498 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:33.499 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:33.499 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:33.499 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:16:33.500 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:33.500 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:33.500 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:16:33.500 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:33.501 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:33.501 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:33.501 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:16:33.501 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:33.502 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:33.502 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:33.502 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:33.502 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:16:33.503 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:33.503 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:33.503 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:16:33.503 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:33.504 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:33.504 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:33.505 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:16:33.505 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:33.505 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:33.505 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:33.506 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:16:33.506 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:33.506 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:16:33.506 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:16:33.507 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:16:33.507 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:33.507 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:16:33Z" 2022-09-13T08:16:33.507 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:16:33.509 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:16:33Z" } 2022-09-13T08:16:33.509 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:16:33.510 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:33.511 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:33.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:16:33.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~275s 2022-09-13T08:16:38.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:38.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:38.514 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:16:38.572 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:16:38.572 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:16:38.607 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:38.603+0000 7f14d0df1600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:38.607 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:38.604+0000 7f14d0df1600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:38.624 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:16:38.624 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:16:38.624 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:16:38.625 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:16:38.625 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:16:38.625 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:16:38.626 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:38.626 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:38.626 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:16:38.626 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:38.627 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:16:38.627 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:38.627 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:38.627 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:16:38.628 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:38.628 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:16:38.629 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:38.629 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:16:38.629 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:38.629 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:38.630 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:38.630 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:38.630 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:16:38.630 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:38.631 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:38.631 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:16:38.631 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:38.632 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:38.632 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:38.632 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:16:38.633 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:38.633 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:38.634 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:38.634 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:38.634 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:16:38.635 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:38.635 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:38.636 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:16:38.637 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:38.638 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:38.638 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:38.639 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:16:38.639 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:38.640 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:38.640 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:38.640 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:38.641 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:16:38.641 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:38.642 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:38.642 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:16:38.643 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:38.643 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:38.644 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:38.644 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:16:38.645 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:38.645 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:38.646 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:38.646 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:16:38.646 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:38.647 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:16:38.647 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:16:38.648 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:16:38.648 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:38.649 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:16:38Z" 2022-09-13T08:16:38.649 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:16:38.651 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:16:38Z" } 2022-09-13T08:16:38.652 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:16:38.653 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:38.654 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:38.655 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:16:39.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~280s 2022-09-13T08:16:43.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:43.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:43.658 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:16:43.716 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:16:43.716 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:16:43.753 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:43.748+0000 7f8cfac40600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:43.753 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:43.749+0000 7f8cfac40600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:43.772 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:16:43.773 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:16:43.773 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:16:43.773 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:16:43.774 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:16:43.774 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:16:43.774 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:43.774 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:43.775 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:16:43.775 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:43.775 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:16:43.775 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:43.776 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:43.776 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:16:43.777 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:43.777 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:16:43.778 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:43.778 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:16:43.779 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:43.779 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:43.780 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:43.780 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:43.781 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:16:43.781 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:43.782 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:43.782 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:16:43.783 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:43.783 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:43.784 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:43.784 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:16:43.785 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:43.785 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:43.786 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:43.786 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:43.787 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:16:43.787 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:43.788 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:43.788 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:16:43.789 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:43.789 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:43.789 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:43.790 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:16:43.790 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:43.791 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:43.791 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:43.792 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:43.792 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:16:43.793 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:43.793 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:43.793 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:16:43.794 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:43.794 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:43.795 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:43.795 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:16:43.796 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:43.796 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:43.796 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:43.797 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:16:43.797 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:43.798 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:16:43.798 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:16:43.798 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:16:43.799 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:43.800 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:16:43Z" 2022-09-13T08:16:43.800 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:16:43.802 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:16:43Z" } 2022-09-13T08:16:43.803 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:16:43.804 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:43.805 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:43.806 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:16:45.123 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~286s 2022-09-13T08:16:48.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:48.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:48.809 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:16:48.866 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:16:48.866 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:16:48.915 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:48.911+0000 7fba68d31600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:48.917 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:48.913+0000 7fba68d31600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:48.932 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:16:48.932 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:16:48.932 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:16:48.933 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:16:48.933 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:16:48.933 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:16:48.934 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:48.934 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:48.934 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:16:48.935 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:48.935 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:16:48.935 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:48.936 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:48.936 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:16:48.936 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:48.936 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:16:48.937 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:48.937 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:16:48.938 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:48.938 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:48.939 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:48.939 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:48.940 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:16:48.940 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:48.940 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:48.941 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:16:48.941 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:48.941 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:48.941 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:48.942 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:16:48.942 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:48.942 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:48.943 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:48.943 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:48.944 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:16:48.944 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:48.944 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:48.945 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:16:48.945 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:48.945 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:48.946 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:48.946 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:16:48.946 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:48.947 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:48.947 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:48.947 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:48.948 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:16:48.948 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:48.948 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:48.948 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:16:48.949 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:48.949 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:48.949 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:48.950 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:16:48.950 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:48.950 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:48.951 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:48.952 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:16:48.952 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:48.952 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:16:48.953 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:16:48.953 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:16:48.953 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:48.953 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:16:48Z" 2022-09-13T08:16:48.954 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:16:48.955 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:16:48Z" } 2022-09-13T08:16:48.956 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:16:48.957 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:48.957 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:48.958 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:16:50.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~292s 2022-09-13T08:16:53.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:53.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:53.961 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:16:54.018 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:16:54.019 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:16:54.057 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:54.053+0000 7f700e4ff600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:54.058 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:54.055+0000 7f700e4ff600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:54.077 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:16:54.077 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:16:54.078 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:16:54.078 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:16:54.079 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:16:54.079 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:16:54.079 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:54.080 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:54.080 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:16:54.080 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:54.081 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:16:54.081 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:54.082 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:54.082 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:16:54.083 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:54.083 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:16:54.083 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:54.084 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:16:54.084 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:54.085 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:54.085 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:54.085 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:54.086 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:16:54.086 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:54.087 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:54.087 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:16:54.087 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:16:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:54.090 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:54.090 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:54.091 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:16:54.091 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:54.092 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:54.092 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:16:54.092 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:54.093 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:54.093 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:54.094 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:16:54.094 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:54.094 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:54.095 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:54.095 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:54.096 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:16:54.096 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:54.096 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:54.097 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:16:54.097 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:54.098 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:54.098 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:54.098 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:16:54.099 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:54.099 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:54.100 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:54.100 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:16:54.101 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:54.101 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:16:54.101 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:16:54.102 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:16:54.102 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:54.102 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:16:54Z" 2022-09-13T08:16:54.102 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:16:54.104 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:16:54Z" } 2022-09-13T08:16:54.105 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:16:54.105 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:54.106 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:54.106 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:16:56.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~297s 2022-09-13T08:16:59.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:59.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:16:59.113 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:16:59.183 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:16:59.184 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:16:59.219 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:59.215+0000 7f7c9de4b600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:59.220 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:16:59.216+0000 7f7c9de4b600 20 RGW-SYNC:meta: read sync status 2022-09-13T08:16:59.236 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2022-09-13T08:16:59.237 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_status": { 2022-09-13T08:16:59.237 INFO:teuthology.orchestra.run.smithi083.stdout: "info": { 2022-09-13T08:16:59.238 INFO:teuthology.orchestra.run.smithi083.stdout: "status": "sync", 2022-09-13T08:16:59.238 INFO:teuthology.orchestra.run.smithi083.stdout: "num_shards": 4, 2022-09-13T08:16:59.238 INFO:teuthology.orchestra.run.smithi083.stdout: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T08:16:59.239 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:59.239 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:59.239 INFO:teuthology.orchestra.run.smithi083.stdout: "markers": [ 2022-09-13T08:16:59.240 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:59.240 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 0, 2022-09-13T08:16:59.240 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:59.240 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:59.241 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:16:59.241 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:59.241 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 1, 2022-09-13T08:16:59.241 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:59.242 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T08:16:59.242 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:59.243 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:59.243 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:59.244 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:59.244 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 1, 2022-09-13T08:16:59.245 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:59.245 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:59.246 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "", 2022-09-13T08:16:59.246 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:59.247 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:59.247 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:59.248 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "0.000000", 2022-09-13T08:16:59.248 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:59.249 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:59.249 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:59.250 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:59.250 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 2, 2022-09-13T08:16:59.251 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:59.251 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:59.252 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663055099.141531_226.1", 2022-09-13T08:16:59.252 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:59.253 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:59.253 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:59.254 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T08:16:59.254 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:59.255 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:59.255 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:59.256 INFO:teuthology.orchestra.run.smithi083.stdout: { 2022-09-13T08:16:59.257 INFO:teuthology.orchestra.run.smithi083.stdout: "key": 3, 2022-09-13T08:16:59.257 INFO:teuthology.orchestra.run.smithi083.stdout: "val": { 2022-09-13T08:16:59.258 INFO:teuthology.orchestra.run.smithi083.stdout: "state": 1, 2022-09-13T08:16:59.258 INFO:teuthology.orchestra.run.smithi083.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:16:59.259 INFO:teuthology.orchestra.run.smithi083.stdout: "next_step_marker": "", 2022-09-13T08:16:59.259 INFO:teuthology.orchestra.run.smithi083.stdout: "total_entries": 0, 2022-09-13T08:16:59.260 INFO:teuthology.orchestra.run.smithi083.stdout: "pos": 0, 2022-09-13T08:16:59.260 INFO:teuthology.orchestra.run.smithi083.stdout: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T08:16:59.261 INFO:teuthology.orchestra.run.smithi083.stdout: "realm_epoch": 2 2022-09-13T08:16:59.261 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:59.262 INFO:teuthology.orchestra.run.smithi083.stdout: } 2022-09-13T08:16:59.262 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2022-09-13T08:16:59.263 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:59.263 INFO:teuthology.orchestra.run.smithi083.stdout: "full_sync": { 2022-09-13T08:16:59.264 INFO:teuthology.orchestra.run.smithi083.stdout: "total": 1, 2022-09-13T08:16:59.264 INFO:teuthology.orchestra.run.smithi083.stdout: "complete": 1 2022-09-13T08:16:59.265 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2022-09-13T08:16:59.265 INFO:teuthology.orchestra.run.smithi083.stdout: "current_time": "2022-09-13T08:16:59Z" 2022-09-13T08:16:59.266 INFO:teuthology.orchestra.run.smithi083.stdout:} 2022-09-13T08:16:59.268 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "3255cc65-e34b-4e83-a436-e06431413191", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663053952.976234_34.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-13T07:25:52.976234Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663055099.141531_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:44:59.141531Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663054100.288347_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-13T07:28:20.288347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-13T08:16:59Z" } 2022-09-13T08:16:59.269 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T08:16:59.270 DEBUG:rgw_multi.tests:log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:59.270 DEBUG:rgw_multi.tests:sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:16:59.271 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T08:17:01.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~303s 2022-09-13T08:17:01.945 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2022-09-13T08:17:02.552 INFO:tasks.ceph.c2.osd.0:Sent signal 15 2022-09-13T08:17:02.555 INFO:tasks.ceph.c2.osd.1:Sent signal 15 2022-09-13T08:17:02.555 INFO:tasks.ceph.c2.osd.2:Sent signal 15 2022-09-13T08:17:02.556 INFO:tasks.ceph.c2.mon.a:Sent signal 15 2022-09-13T08:17:02.556 INFO:tasks.rgw.c2.client.1:Sent signal 15 2022-09-13T08:17:02.557 INFO:tasks.ceph.c2.mgr.x:Sent signal 15 2022-09-13T08:17:02.558 INFO:tasks.ceph.c2.osd.0.smithi083.stderr:2022-09-13T08:17:02.549+0000 7f4466a78700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 0 (PID: 101783) UID: 0 2022-09-13T08:17:02.558 INFO:tasks.ceph.c2.osd.0.smithi083.stderr:2022-09-13T08:17:02.549+0000 7f4466a78700 -1 osd.0 41 *** Got signal Terminated *** 2022-09-13T08:17:02.559 INFO:tasks.ceph.c2.osd.0.smithi083.stderr:2022-09-13T08:17:02.549+0000 7f4466a78700 -1 osd.0 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-13T08:17:02.559 INFO:tasks.ceph.c2.osd.1.smithi083.stderr:2022-09-13T08:17:02.552+0000 7fe2ef316700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 1 (PID: 101790) UID: 0 2022-09-13T08:17:02.560 INFO:tasks.ceph.c2.osd.1.smithi083.stderr:2022-09-13T08:17:02.552+0000 7fe2ef316700 -1 osd.1 41 *** Got signal Terminated *** 2022-09-13T08:17:02.560 INFO:tasks.ceph.c2.osd.1.smithi083.stderr:2022-09-13T08:17:02.552+0000 7fe2ef316700 -1 osd.1 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-13T08:17:02.561 INFO:tasks.ceph.c2.osd.2.smithi083.stderr:2022-09-13T08:17:02.553+0000 7f4ed4ed2700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 2 (PID: 101793) UID: 0 2022-09-13T08:17:02.561 INFO:tasks.ceph.c2.osd.2.smithi083.stderr:2022-09-13T08:17:02.553+0000 7f4ed4ed2700 -1 osd.2 41 *** Got signal Terminated *** 2022-09-13T08:17:02.561 INFO:tasks.ceph.c2.osd.2.smithi083.stderr:2022-09-13T08:17:02.553+0000 7f4ed4ed2700 -1 osd.2 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-13T08:17:02.562 INFO:tasks.ceph.c2.mon.a.smithi083.stderr:2022-09-13T08:17:02.553+0000 7fe0d6cbc700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c2 -i a (PID: 101373) UID: 0 2022-09-13T08:17:02.562 INFO:tasks.ceph.c2.mon.a.smithi083.stderr:2022-09-13T08:17:02.553+0000 7fe0d6cbc700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-09-13T08:17:02.563 INFO:tasks.rgw.c2.client.1.smithi083.stdout:2022-09-13T08:17:02.554+0000 7fd974c11700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 107345) UID: 0 2022-09-13T08:17:02.563 INFO:tasks.rgw.c2.client.1.smithi083.stdout:2022-09-13T08:17:02.554+0000 7fd978e7c600 -1 shutting down 2022-09-13T08:17:02.564 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T08:17:02.555+0000 7fe76d000700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster c2 -i x (PID: 101380) UID: 0 2022-09-13T08:17:02.564 INFO:tasks.ceph.c2.mgr.x.smithi083.stderr:2022-09-13T08:17:02.555+0000 7fe76d000700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2022-09-13T08:17:04.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... FAIL 2022-09-13T08:17:04.287 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-55 2022-09-13T08:17:04.288 DEBUG:boto:path=/yxvvuh-55/ 2022-09-13T08:17:04.289 DEBUG:boto:auth_path=/yxvvuh-55/ 2022-09-13T08:17:04.290 DEBUG:boto:Method: PUT 2022-09-13T08:17:04.291 DEBUG:boto:Path: /yxvvuh-55/ 2022-09-13T08:17:04.292 DEBUG:boto:Data: 2022-09-13T08:17:04.293 DEBUG:boto:Headers: {} 2022-09-13T08:17:04.293 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T08:17:04.294 DEBUG:boto:Port: 8000 2022-09-13T08:17:04.294 DEBUG:boto:Params: {} 2022-09-13T08:17:04.295 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:17:04.296 DEBUG:boto:Token: None 2022-09-13T08:17:04.297 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:04 GMT /yxvvuh-55/ 2022-09-13T08:17:04.298 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:dxFxdzvpERSbiiKAYUOBs59shtE= 2022-09-13T08:17:04.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dxFxdzvpERSbiiKAYUOBs59shtE=', 'Content-Length': '0'} 2022-09-13T08:17:04.324 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000071139a133783feb8-0063203c80-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:17:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:17:04.325 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-56 2022-09-13T08:17:04.326 DEBUG:boto:path=/yxvvuh-56/ 2022-09-13T08:17:04.326 DEBUG:boto:auth_path=/yxvvuh-56/ 2022-09-13T08:17:04.327 DEBUG:boto:Method: PUT 2022-09-13T08:17:04.327 DEBUG:boto:Path: /yxvvuh-56/ 2022-09-13T08:17:04.328 DEBUG:boto:Data: 2022-09-13T08:17:04.328 DEBUG:boto:Headers: {} 2022-09-13T08:17:04.329 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T08:17:04.329 DEBUG:boto:Port: 8000 2022-09-13T08:17:04.330 DEBUG:boto:Params: {} 2022-09-13T08:17:04.330 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:17:04.331 DEBUG:boto:Token: None 2022-09-13T08:17:04.331 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:04 GMT /yxvvuh-56/ 2022-09-13T08:17:04.332 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc= 2022-09-13T08:17:04.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc=', 'Content-Length': '0'} 2022-09-13T08:17:04.335 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:17:04.341 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:17:04.730 DEBUG:boto:Token: None 2022-09-13T08:17:04.731 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:04 GMT /yxvvuh-56/ 2022-09-13T08:17:04.732 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc= 2022-09-13T08:17:04.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc=', 'Content-Length': '0'} 2022-09-13T08:17:04.737 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:17:04.738 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:17:04.769 DEBUG:boto:Token: None 2022-09-13T08:17:04.771 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:04 GMT /yxvvuh-56/ 2022-09-13T08:17:04.772 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc= 2022-09-13T08:17:04.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc=', 'Content-Length': '0'} 2022-09-13T08:17:04.775 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:17:04.776 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:17:08.412 DEBUG:boto:Token: None 2022-09-13T08:17:08.413 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:04 GMT /yxvvuh-56/ 2022-09-13T08:17:08.414 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc= 2022-09-13T08:17:08.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc=', 'Content-Length': '0'} 2022-09-13T08:17:08.417 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:17:08.419 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:17:14.259 DEBUG:boto:Token: None 2022-09-13T08:17:14.260 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:04 GMT /yxvvuh-56/ 2022-09-13T08:17:14.260 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc= 2022-09-13T08:17:14.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc=', 'Content-Length': '0'} 2022-09-13T08:17:14.263 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:17:14.264 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:17:23.073 DEBUG:boto:Token: None 2022-09-13T08:17:23.074 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:04 GMT /yxvvuh-56/ 2022-09-13T08:17:23.075 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc= 2022-09-13T08:17:23.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc=', 'Content-Length': '0'} 2022-09-13T08:17:23.078 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:17:23.079 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:17:26.442 DEBUG:boto:Token: None 2022-09-13T08:17:26.443 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:04 GMT /yxvvuh-56/ 2022-09-13T08:17:26.444 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc= 2022-09-13T08:17:26.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc=', 'Content-Length': '0'} 2022-09-13T08:17:26.446 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:17:26.446 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:17:40.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ERROR 2022-09-13T08:17:40.235 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-57 2022-09-13T08:17:40.235 DEBUG:boto:path=/yxvvuh-57/ 2022-09-13T08:17:40.236 DEBUG:boto:auth_path=/yxvvuh-57/ 2022-09-13T08:17:40.236 DEBUG:boto:Method: PUT 2022-09-13T08:17:40.237 DEBUG:boto:Path: /yxvvuh-57/ 2022-09-13T08:17:40.237 DEBUG:boto:Data: 2022-09-13T08:17:40.238 DEBUG:boto:Headers: {} 2022-09-13T08:17:40.238 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T08:17:40.239 DEBUG:boto:Port: 8000 2022-09-13T08:17:40.239 DEBUG:boto:Params: {} 2022-09-13T08:17:40.240 DEBUG:boto:Token: None 2022-09-13T08:17:40.240 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:40 GMT /yxvvuh-57/ 2022-09-13T08:17:40.241 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:X7K9bemZQ0pV5EkIAlDlqjlp5ME= 2022-09-13T08:17:40.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:X7K9bemZQ0pV5EkIAlDlqjlp5ME=', 'Content-Length': '0'} 2022-09-13T08:17:40.272 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000032187529ff0712d2-0063203ca4-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:17:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:17:40.273 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-58 2022-09-13T08:17:40.273 DEBUG:boto:path=/yxvvuh-58/ 2022-09-13T08:17:40.274 DEBUG:boto:auth_path=/yxvvuh-58/ 2022-09-13T08:17:40.274 DEBUG:boto:Method: PUT 2022-09-13T08:17:40.275 DEBUG:boto:Path: /yxvvuh-58/ 2022-09-13T08:17:40.275 DEBUG:boto:Data: 2022-09-13T08:17:40.276 DEBUG:boto:Headers: {} 2022-09-13T08:17:40.276 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T08:17:40.277 DEBUG:boto:Port: 8000 2022-09-13T08:17:40.277 DEBUG:boto:Params: {} 2022-09-13T08:17:40.278 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:17:40.278 DEBUG:boto:Token: None 2022-09-13T08:17:40.279 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:40 GMT /yxvvuh-58/ 2022-09-13T08:17:40.279 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ= 2022-09-13T08:17:40.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ=', 'Content-Length': '0'} 2022-09-13T08:17:40.282 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:17:40.282 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:17:41.000 DEBUG:boto:Token: None 2022-09-13T08:17:41.001 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:40 GMT /yxvvuh-58/ 2022-09-13T08:17:41.002 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ= 2022-09-13T08:17:41.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ=', 'Content-Length': '0'} 2022-09-13T08:17:41.004 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:17:41.005 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:17:42.943 DEBUG:boto:Token: None 2022-09-13T08:17:42.945 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:40 GMT /yxvvuh-58/ 2022-09-13T08:17:42.946 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ= 2022-09-13T08:17:42.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ=', 'Content-Length': '0'} 2022-09-13T08:17:42.949 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:17:42.950 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:17:44.873 DEBUG:boto:Token: None 2022-09-13T08:17:44.873 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:40 GMT /yxvvuh-58/ 2022-09-13T08:17:44.874 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ= 2022-09-13T08:17:44.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ=', 'Content-Length': '0'} 2022-09-13T08:17:44.876 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:17:44.878 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:17:45.546 DEBUG:boto:Token: None 2022-09-13T08:17:45.547 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:40 GMT /yxvvuh-58/ 2022-09-13T08:17:45.548 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ= 2022-09-13T08:17:45.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ=', 'Content-Length': '0'} 2022-09-13T08:17:45.550 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:17:45.551 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:17:57.440 DEBUG:boto:Token: None 2022-09-13T08:17:57.441 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:40 GMT /yxvvuh-58/ 2022-09-13T08:17:57.441 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ= 2022-09-13T08:17:57.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ=', 'Content-Length': '0'} 2022-09-13T08:17:57.444 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:17:57.444 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:18:16.493 DEBUG:boto:Token: None 2022-09-13T08:18:16.493 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:17:40 GMT /yxvvuh-58/ 2022-09-13T08:18:16.494 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ= 2022-09-13T08:18:16.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ=', 'Content-Length': '0'} 2022-09-13T08:18:16.497 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:18:16.498 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:19:08.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ERROR 2022-09-13T08:19:08.619 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-59 2022-09-13T08:19:08.620 DEBUG:boto:path=/yxvvuh-59/ 2022-09-13T08:19:08.621 DEBUG:boto:auth_path=/yxvvuh-59/ 2022-09-13T08:19:08.621 DEBUG:boto:Method: PUT 2022-09-13T08:19:08.622 DEBUG:boto:Path: /yxvvuh-59/ 2022-09-13T08:19:08.622 DEBUG:boto:Data: 2022-09-13T08:19:08.623 DEBUG:boto:Headers: {} 2022-09-13T08:19:08.623 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T08:19:08.624 DEBUG:boto:Port: 8000 2022-09-13T08:19:08.624 DEBUG:boto:Params: {} 2022-09-13T08:19:08.625 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:19:08.626 DEBUG:boto:Token: None 2022-09-13T08:19:08.626 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:19:08 GMT /yxvvuh-59/ 2022-09-13T08:19:08.627 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nl+JKK0AEGRlWhLQroyzkYMbbEU= 2022-09-13T08:19:08.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nl+JKK0AEGRlWhLQroyzkYMbbEU=', 'Content-Length': '0'} 2022-09-13T08:19:08.655 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e674866abe7b18ce-0063203cfc-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:19:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:19:08.657 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yxvvuh-60 2022-09-13T08:19:08.657 DEBUG:boto:path=/yxvvuh-60/ 2022-09-13T08:19:08.658 DEBUG:boto:auth_path=/yxvvuh-60/ 2022-09-13T08:19:08.659 DEBUG:boto:Method: PUT 2022-09-13T08:19:08.660 DEBUG:boto:Path: /yxvvuh-60/ 2022-09-13T08:19:08.660 DEBUG:boto:Data: 2022-09-13T08:19:08.661 DEBUG:boto:Headers: {} 2022-09-13T08:19:08.661 DEBUG:boto:Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T08:19:08.662 DEBUG:boto:Port: 8000 2022-09-13T08:19:08.663 DEBUG:boto:Params: {} 2022-09-13T08:19:08.663 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:19:08.664 DEBUG:boto:Token: None 2022-09-13T08:19:08.664 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:19:08 GMT /yxvvuh-60/ 2022-09-13T08:19:08.665 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds= 2022-09-13T08:19:08.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds=', 'Content-Length': '0'} 2022-09-13T08:19:08.668 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:19:08.668 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:19:08.838 DEBUG:boto:Token: None 2022-09-13T08:19:08.839 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:19:08 GMT /yxvvuh-60/ 2022-09-13T08:19:08.839 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds= 2022-09-13T08:19:08.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds=', 'Content-Length': '0'} 2022-09-13T08:19:08.842 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:19:08.843 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:19:09.645 DEBUG:boto:Token: None 2022-09-13T08:19:09.647 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:19:08 GMT /yxvvuh-60/ 2022-09-13T08:19:09.647 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds= 2022-09-13T08:19:09.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds=', 'Content-Length': '0'} 2022-09-13T08:19:09.649 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:19:09.650 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:19:12.379 DEBUG:boto:Token: None 2022-09-13T08:19:12.380 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:19:08 GMT /yxvvuh-60/ 2022-09-13T08:19:12.381 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds= 2022-09-13T08:19:12.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds=', 'Content-Length': '0'} 2022-09-13T08:19:12.384 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:19:12.385 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:19:18.125 DEBUG:boto:Token: None 2022-09-13T08:19:18.126 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:19:08 GMT /yxvvuh-60/ 2022-09-13T08:19:18.127 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds= 2022-09-13T08:19:18.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds=', 'Content-Length': '0'} 2022-09-13T08:19:18.130 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:19:18.130 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:19:28.631 DEBUG:boto:Token: None 2022-09-13T08:19:28.632 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:19:08 GMT /yxvvuh-60/ 2022-09-13T08:19:28.633 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds= 2022-09-13T08:19:28.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds=', 'Content-Length': '0'} 2022-09-13T08:19:28.635 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:19:28.636 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:19:55.613 DEBUG:boto:Token: None 2022-09-13T08:19:55.614 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:19:08 GMT /yxvvuh-60/ 2022-09-13T08:19:55.615 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds= 2022-09-13T08:19:55.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds=', 'Content-Length': '0'} 2022-09-13T08:19:55.617 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T08:19:55.618 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:20:46.008 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ERROR 2022-09-13T08:20:46.010 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-61 2022-09-13T08:20:46.011 DEBUG:boto:path=/yxvvuh-61/ 2022-09-13T08:20:46.012 DEBUG:boto:auth_path=/yxvvuh-61/ 2022-09-13T08:20:46.012 DEBUG:boto:Method: PUT 2022-09-13T08:20:46.013 DEBUG:boto:Path: /yxvvuh-61/ 2022-09-13T08:20:46.014 DEBUG:boto:Data: 2022-09-13T08:20:46.014 DEBUG:boto:Headers: {} 2022-09-13T08:20:46.015 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T08:20:46.016 DEBUG:boto:Port: 8000 2022-09-13T08:20:46.016 DEBUG:boto:Params: {} 2022-09-13T08:20:46.018 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T08:20:46.019 DEBUG:boto:Token: None 2022-09-13T08:20:46.020 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 08:20:46 GMT /yxvvuh-61/ 2022-09-13T08:20:46.021 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:Rwrs7Ck2frcabdpY5LsIbzfai7U= 2022-09-13T08:20:46.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:20:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Rwrs7Ck2frcabdpY5LsIbzfai7U=', 'Content-Length': '0'} 2022-09-13T08:20:46.050 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bd454e1dad04be01-0063203d5e-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:20:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:20:46.051 DEBUG:boto:path=/yxvvuh-61/testobj-sse-c 2022-09-13T08:20:46.052 DEBUG:boto:auth_path=/yxvvuh-61/testobj-sse-c 2022-09-13T08:20:46.052 DEBUG:boto:Method: PUT 2022-09-13T08:20:46.053 DEBUG:boto:Path: /yxvvuh-61/testobj-sse-c 2022-09-13T08:20:46.053 DEBUG:boto:Data: 2022-09-13T08:20:46.054 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-13T08:20:46.055 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T08:20:46.055 DEBUG:boto:Port: 8000 2022-09-13T08:20:46.056 DEBUG:boto:Params: {} 2022-09-13T08:20:46.057 DEBUG:boto:Token: None 2022-09-13T08:20:46.057 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Tue, 13 Sep 2022 08:20:46 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /yxvvuh-61/testobj-sse-c 2022-09-13T08:20:46.058 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RgxR8eyCDc310zX1uDs6hF1Nt7E= 2022-09-13T08:20:46.058 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 08:20:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RgxR8eyCDc310zX1uDs6hF1Nt7E='} 2022-09-13T08:20:46.068 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000d664d4c11207af30-0063203d5e-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 08:20:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:20:46.070 DEBUG:boto:path=/yxvvuh-61/testobj-sse-kms 2022-09-13T08:20:46.070 DEBUG:boto:auth_path=/yxvvuh-61/testobj-sse-kms 2022-09-13T08:20:46.071 DEBUG:boto:Method: PUT 2022-09-13T08:20:46.072 DEBUG:boto:Path: /yxvvuh-61/testobj-sse-kms 2022-09-13T08:20:46.073 DEBUG:boto:Data: 2022-09-13T08:20:46.074 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-13T08:20:46.075 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T08:20:46.075 DEBUG:boto:Port: 8000 2022-09-13T08:20:46.076 DEBUG:boto:Params: {} 2022-09-13T08:20:46.077 DEBUG:boto:Token: None 2022-09-13T08:20:46.078 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Tue, 13 Sep 2022 08:20:46 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /yxvvuh-61/testobj-sse-kms 2022-09-13T08:20:46.079 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:wvx68MHJhfoKMgf2XSCRLYvbE5w= 2022-09-13T08:20:46.080 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 08:20:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wvx68MHJhfoKMgf2XSCRLYvbE5w='} 2022-09-13T08:20:46.088 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6eXh2dnVoLTYxL3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx000006eef3a7a7c1550f1-0063203d5e-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 08:20:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T08:20:46.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:20:46.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:20:46.091 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:20:46.147 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:20:46.148 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:20:46.188 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T08:20:46.202 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T08:20:46.203 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:20:46.203 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T08:20:46.203 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T08:20:46.203 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:20:46.204 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:20:46.204 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T08:20:46.204 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T08:20:46.205 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:20:46.205 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:20:46.206 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663057246.045157_471.1", 2022-09-13T08:20:46.206 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T08:20:46.045157Z" 2022-09-13T08:20:46.206 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T08:20:46.206 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T08:20:46.207 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T08:20:46.207 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T08:20:46.207 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T08:20:46.207 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T08:20:46.209 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663057246.045157_471.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T08:20:46.209 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T08:20:46.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:20:46.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T08:20:46.211 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T08:20:46.267 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T08:20:46.267 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T08:25:46.270 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:25:46.265+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T08:30:46.270 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:30:46.265+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T08:35:46.270 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:35:46.265+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T08:40:46.284 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:40:46.265+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T08:45:46.270 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:45:46.266+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T08:50:46.272 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:50:46.267+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T08:55:46.271 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T08:55:46.266+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T09:00:46.271 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T09:00:46.267+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T09:05:46.272 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T09:05:46.268+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T09:10:46.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T09:10:46.317 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T09:10:46.268+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T09:10:46.318 INFO:teuthology.orchestra.run.smithi083.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T09:10:46.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... FAIL 2022-09-13T09:10:46.323 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-62 2022-09-13T09:10:46.323 DEBUG:boto:path=/yxvvuh-62/ 2022-09-13T09:10:46.329 DEBUG:boto:auth_path=/yxvvuh-62/ 2022-09-13T09:10:46.330 DEBUG:boto:Method: PUT 2022-09-13T09:10:46.331 DEBUG:boto:Path: /yxvvuh-62/ 2022-09-13T09:10:46.332 DEBUG:boto:Data: 2022-09-13T09:10:46.333 DEBUG:boto:Headers: {} 2022-09-13T09:10:46.333 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T09:10:46.334 DEBUG:boto:Port: 8000 2022-09-13T09:10:46.335 DEBUG:boto:Params: {} 2022-09-13T09:10:46.337 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T09:10:46.338 DEBUG:boto:Token: None 2022-09-13T09:10:46.340 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 09:10:46 GMT /yxvvuh-62/ 2022-09-13T09:10:46.341 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pNqW/41uNV1dKuiTrKwumWMMV8A= 2022-09-13T09:10:46.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pNqW/41uNV1dKuiTrKwumWMMV8A=', 'Content-Length': '0'} 2022-09-13T09:10:46.375 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006d53d55180661671-0063204916-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T09:10:46.376 DEBUG:boto:path=/yxvvuh-62/ 2022-09-13T09:10:46.376 DEBUG:boto:auth_path=/yxvvuh-62/ 2022-09-13T09:10:46.377 DEBUG:boto:Method: HEAD 2022-09-13T09:10:46.378 DEBUG:boto:Path: /yxvvuh-62/ 2022-09-13T09:10:46.378 DEBUG:boto:Data: 2022-09-13T09:10:46.379 DEBUG:boto:Headers: {} 2022-09-13T09:10:46.379 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T09:10:46.379 DEBUG:boto:Port: 8000 2022-09-13T09:10:46.380 DEBUG:boto:Params: {} 2022-09-13T09:10:46.380 DEBUG:boto:Token: None 2022-09-13T09:10:46.381 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 09:10:46 GMT /yxvvuh-62/ 2022-09-13T09:10:46.382 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A= 2022-09-13T09:10:46.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A=', 'Content-Length': '0'} 2022-09-13T09:10:46.385 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000547efb11e7302678-0063204916-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T09:10:46.386 DEBUG:boto:path=/yxvvuh-62/a 2022-09-13T09:10:46.387 DEBUG:boto:auth_path=/yxvvuh-62/a 2022-09-13T09:10:46.387 DEBUG:boto:Method: PUT 2022-09-13T09:10:46.388 DEBUG:boto:Path: /yxvvuh-62/a 2022-09-13T09:10:46.389 DEBUG:boto:Data: 2022-09-13T09:10:46.389 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T09:10:46.390 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T09:10:46.390 DEBUG:boto:Port: 8000 2022-09-13T09:10:46.391 DEBUG:boto:Params: {} 2022-09-13T09:10:46.391 DEBUG:boto:Token: None 2022-09-13T09:10:46.392 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 13 Sep 2022 09:10:46 GMT /yxvvuh-62/a 2022-09-13T09:10:46.395 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:/TxJ28dQWmlNUubCzqhq7GAyh7k= 2022-09-13T09:10:46.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/TxJ28dQWmlNUubCzqhq7GAyh7k='} 2022-09-13T09:10:46.412 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000defe9c77bfae17e9-0063204916-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T09:10:46.414 DEBUG:boto:path=/yxvvuh-62/ 2022-09-13T09:10:46.415 DEBUG:boto:auth_path=/yxvvuh-62/ 2022-09-13T09:10:46.416 DEBUG:boto:Method: HEAD 2022-09-13T09:10:46.419 DEBUG:boto:Path: /yxvvuh-62/ 2022-09-13T09:10:46.420 DEBUG:boto:Data: 2022-09-13T09:10:46.420 DEBUG:boto:Headers: {} 2022-09-13T09:10:46.421 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T09:10:46.421 DEBUG:boto:Port: 8000 2022-09-13T09:10:46.422 DEBUG:boto:Params: {} 2022-09-13T09:10:46.422 DEBUG:boto:Token: None 2022-09-13T09:10:46.423 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 09:10:46 GMT /yxvvuh-62/ 2022-09-13T09:10:46.423 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A= 2022-09-13T09:10:46.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A=', 'Content-Length': '0'} 2022-09-13T09:10:46.427 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023f90cdfd794bf4c-0063204916-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T09:10:46.428 DEBUG:boto:path=/yxvvuh-62/b 2022-09-13T09:10:46.428 DEBUG:boto:auth_path=/yxvvuh-62/b 2022-09-13T09:10:46.429 DEBUG:boto:Method: PUT 2022-09-13T09:10:46.429 DEBUG:boto:Path: /yxvvuh-62/b 2022-09-13T09:10:46.430 DEBUG:boto:Data: 2022-09-13T09:10:46.430 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T09:10:46.431 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T09:10:46.431 DEBUG:boto:Port: 8000 2022-09-13T09:10:46.432 DEBUG:boto:Params: {} 2022-09-13T09:10:46.432 DEBUG:boto:Token: None 2022-09-13T09:10:46.433 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 13 Sep 2022 09:10:46 GMT /yxvvuh-62/b 2022-09-13T09:10:46.434 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:muhhWkSqWbrUauBK3Eynmy+6juU= 2022-09-13T09:10:46.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:muhhWkSqWbrUauBK3Eynmy+6juU='} 2022-09-13T09:10:46.483 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000432d817334668fbe-0063204916-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T09:10:46.484 DEBUG:boto:path=/yxvvuh-62/ 2022-09-13T09:10:46.485 DEBUG:boto:auth_path=/yxvvuh-62/ 2022-09-13T09:10:46.486 DEBUG:boto:Method: HEAD 2022-09-13T09:10:46.486 DEBUG:boto:Path: /yxvvuh-62/ 2022-09-13T09:10:46.487 DEBUG:boto:Data: 2022-09-13T09:10:46.488 DEBUG:boto:Headers: {} 2022-09-13T09:10:46.489 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T09:10:46.490 DEBUG:boto:Port: 8000 2022-09-13T09:10:46.490 DEBUG:boto:Params: {} 2022-09-13T09:10:46.493 DEBUG:boto:Token: None 2022-09-13T09:10:46.493 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 09:10:46 GMT /yxvvuh-62/ 2022-09-13T09:10:46.494 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A= 2022-09-13T09:10:46.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A=', 'Content-Length': '0'} 2022-09-13T09:10:46.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e308ed8db227ce6a-0063204916-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T09:10:46.499 DEBUG:boto:path=/yxvvuh-62/c 2022-09-13T09:10:46.499 DEBUG:boto:auth_path=/yxvvuh-62/c 2022-09-13T09:10:46.500 DEBUG:boto:Method: PUT 2022-09-13T09:10:46.500 DEBUG:boto:Path: /yxvvuh-62/c 2022-09-13T09:10:46.501 DEBUG:boto:Data: 2022-09-13T09:10:46.501 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T09:10:46.502 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T09:10:46.502 DEBUG:boto:Port: 8000 2022-09-13T09:10:46.503 DEBUG:boto:Params: {} 2022-09-13T09:10:46.503 DEBUG:boto:Token: None 2022-09-13T09:10:46.504 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 13 Sep 2022 09:10:46 GMT /yxvvuh-62/c 2022-09-13T09:10:46.505 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:RZfOC1j82BxDftMKcsXZz7SbCwg= 2022-09-13T09:10:46.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RZfOC1j82BxDftMKcsXZz7SbCwg='} 2022-09-13T09:10:46.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000621f4700de617488-0063204916-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T09:10:46.515 DEBUG:boto:path=/yxvvuh-62/ 2022-09-13T09:10:46.516 DEBUG:boto:auth_path=/yxvvuh-62/ 2022-09-13T09:10:46.517 DEBUG:boto:Method: HEAD 2022-09-13T09:10:46.518 DEBUG:boto:Path: /yxvvuh-62/ 2022-09-13T09:10:46.519 DEBUG:boto:Data: 2022-09-13T09:10:46.519 DEBUG:boto:Headers: {} 2022-09-13T09:10:46.521 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T09:10:46.523 DEBUG:boto:Port: 8000 2022-09-13T09:10:46.524 DEBUG:boto:Params: {} 2022-09-13T09:10:46.525 DEBUG:boto:Token: None 2022-09-13T09:10:46.526 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 09:10:46 GMT /yxvvuh-62/ 2022-09-13T09:10:46.527 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A= 2022-09-13T09:10:46.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A=', 'Content-Length': '0'} 2022-09-13T09:10:46.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036902a2b0b40ecb1-0063204916-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T09:10:46.533 DEBUG:boto:path=/yxvvuh-62/d 2022-09-13T09:10:46.533 DEBUG:boto:auth_path=/yxvvuh-62/d 2022-09-13T09:10:46.534 DEBUG:boto:Method: PUT 2022-09-13T09:10:46.535 DEBUG:boto:Path: /yxvvuh-62/d 2022-09-13T09:10:46.535 DEBUG:boto:Data: 2022-09-13T09:10:46.536 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T09:10:46.536 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T09:10:46.537 DEBUG:boto:Port: 8000 2022-09-13T09:10:46.537 DEBUG:boto:Params: {} 2022-09-13T09:10:46.538 DEBUG:boto:Token: None 2022-09-13T09:10:46.538 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 13 Sep 2022 09:10:46 GMT /yxvvuh-62/d 2022-09-13T09:10:46.539 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:7hqFbE7X/sjRvjiIqEXOZ016QyA= 2022-09-13T09:10:46.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7hqFbE7X/sjRvjiIqEXOZ016QyA='} 2022-09-13T09:10:46.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc5c8c528e927b84-0063204916-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T09:10:46.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T09:10:46.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T09:10:46.587 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T09:10:46.648 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T09:10:46.649 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T09:10:46.696 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T09:10:46.709 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T09:10:46.709 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T09:10:46.710 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T09:10:46.710 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T09:10:46.711 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T09:10:46.711 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T09:10:46.711 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T09:10:46.711 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T09:10:46.712 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T09:10:46.712 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T09:10:46.712 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663060246.371724_555.1", 2022-09-13T09:10:46.713 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T09:10:46.371724Z" 2022-09-13T09:10:46.713 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T09:10:46.713 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T09:10:46.713 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T09:10:46.714 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T09:10:46.714 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T09:10:46.714 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T09:10:46.716 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663060246.371724_555.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T09:10:46.716 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T09:10:46.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T09:10:46.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T09:10:46.718 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T09:10:46.772 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T09:10:46.773 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T09:15:46.775 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T09:15:46.772+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T09:20:46.776 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T09:20:46.772+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T09:25:46.776 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T09:25:46.773+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T09:30:46.777 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T09:30:46.773+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T09:35:46.777 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T09:35:46.774+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T09:40:46.777 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T09:40:46.775+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T09:45:46.779 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T09:45:46.775+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T09:50:46.779 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T09:50:46.775+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T09:55:46.778 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T09:55:46.775+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T10:00:46.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T10:00:46.804 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T10:00:46.776+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T10:00:46.805 INFO:teuthology.orchestra.run.smithi083.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T10:00:46.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... FAIL 2022-09-13T10:00:46.809 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yxvvuh-63 2022-09-13T10:00:46.810 DEBUG:boto:path=/yxvvuh-63/ 2022-09-13T10:00:46.814 DEBUG:boto:auth_path=/yxvvuh-63/ 2022-09-13T10:00:46.815 DEBUG:boto:Method: PUT 2022-09-13T10:00:46.816 DEBUG:boto:Path: /yxvvuh-63/ 2022-09-13T10:00:46.816 DEBUG:boto:Data: 2022-09-13T10:00:46.817 DEBUG:boto:Headers: {} 2022-09-13T10:00:46.817 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T10:00:46.818 DEBUG:boto:Port: 8000 2022-09-13T10:00:46.818 DEBUG:boto:Params: {} 2022-09-13T10:00:46.819 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T10:00:46.820 DEBUG:boto:Token: None 2022-09-13T10:00:46.821 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 10:00:46 GMT /yxvvuh-63/ 2022-09-13T10:00:46.822 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:tut0dZSKDsA4fcmEqBSHplNZybQ= 2022-09-13T10:00:46.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 10:00:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tut0dZSKDsA4fcmEqBSHplNZybQ=', 'Content-Length': '0'} 2022-09-13T10:00:46.856 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e9ff849664b91929-00632054ce-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 10:00:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T10:00:46.857 DEBUG:boto:path=/yxvvuh-63/ 2022-09-13T10:00:46.858 DEBUG:boto:auth_path=/yxvvuh-63/ 2022-09-13T10:00:46.859 DEBUG:boto:Method: HEAD 2022-09-13T10:00:46.860 DEBUG:boto:Path: /yxvvuh-63/ 2022-09-13T10:00:46.861 DEBUG:boto:Data: 2022-09-13T10:00:46.862 DEBUG:boto:Headers: {} 2022-09-13T10:00:46.863 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T10:00:46.864 DEBUG:boto:Port: 8000 2022-09-13T10:00:46.865 DEBUG:boto:Params: {} 2022-09-13T10:00:46.866 DEBUG:boto:Token: None 2022-09-13T10:00:46.867 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 10:00:46 GMT /yxvvuh-63/ 2022-09-13T10:00:46.868 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oRZBJNrKgFogjVioiIcukiSXsNg= 2022-09-13T10:00:46.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 10:00:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oRZBJNrKgFogjVioiIcukiSXsNg=', 'Content-Length': '0'} 2022-09-13T10:00:46.872 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094687016d78e8f6b-00632054ce-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 10:00:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T10:00:46.873 DEBUG:boto:path=/yxvvuh-63/a 2022-09-13T10:00:46.873 DEBUG:boto:auth_path=/yxvvuh-63/a 2022-09-13T10:00:46.874 DEBUG:boto:Method: PUT 2022-09-13T10:00:46.874 DEBUG:boto:Path: /yxvvuh-63/a 2022-09-13T10:00:46.875 DEBUG:boto:Data: 2022-09-13T10:00:46.875 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T10:00:46.876 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T10:00:46.876 DEBUG:boto:Port: 8000 2022-09-13T10:00:46.877 DEBUG:boto:Params: {} 2022-09-13T10:00:46.877 DEBUG:boto:Token: None 2022-09-13T10:00:46.878 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 13 Sep 2022 10:00:46 GMT /yxvvuh-63/a 2022-09-13T10:00:46.878 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qvqtOVyzKUmk/eboboxYh7EGQGM= 2022-09-13T10:00:46.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 10:00:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qvqtOVyzKUmk/eboboxYh7EGQGM='} 2022-09-13T10:00:46.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbb9b57feb925875-00632054ce-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 10:00:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T10:00:46.890 DEBUG:boto:path=/yxvvuh-63/ 2022-09-13T10:00:46.890 DEBUG:boto:auth_path=/yxvvuh-63/ 2022-09-13T10:00:46.891 DEBUG:boto:Method: HEAD 2022-09-13T10:00:46.892 DEBUG:boto:Path: /yxvvuh-63/ 2022-09-13T10:00:46.893 DEBUG:boto:Data: 2022-09-13T10:00:46.894 DEBUG:boto:Headers: {} 2022-09-13T10:00:46.895 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T10:00:46.896 DEBUG:boto:Port: 8000 2022-09-13T10:00:46.897 DEBUG:boto:Params: {} 2022-09-13T10:00:46.898 DEBUG:boto:Token: None 2022-09-13T10:00:46.899 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 10:00:46 GMT /yxvvuh-63/ 2022-09-13T10:00:46.900 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oRZBJNrKgFogjVioiIcukiSXsNg= 2022-09-13T10:00:46.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 10:00:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oRZBJNrKgFogjVioiIcukiSXsNg=', 'Content-Length': '0'} 2022-09-13T10:00:46.904 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb25daf77800bf32-00632054ce-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 10:00:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T10:00:46.905 DEBUG:boto:path=/yxvvuh-63/b 2022-09-13T10:00:46.905 DEBUG:boto:auth_path=/yxvvuh-63/b 2022-09-13T10:00:46.906 DEBUG:boto:Method: PUT 2022-09-13T10:00:46.906 DEBUG:boto:Path: /yxvvuh-63/b 2022-09-13T10:00:46.907 DEBUG:boto:Data: 2022-09-13T10:00:46.907 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T10:00:46.908 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T10:00:46.908 DEBUG:boto:Port: 8000 2022-09-13T10:00:46.909 DEBUG:boto:Params: {} 2022-09-13T10:00:46.910 DEBUG:boto:Token: None 2022-09-13T10:00:46.911 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 13 Sep 2022 10:00:46 GMT /yxvvuh-63/b 2022-09-13T10:00:46.912 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BG3Uzf14Tk2x6lOcIaGriVz03H0= 2022-09-13T10:00:46.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 10:00:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BG3Uzf14Tk2x6lOcIaGriVz03H0='} 2022-09-13T10:00:46.925 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006da5117129a5965f-00632054ce-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 10:00:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T10:00:46.926 DEBUG:boto:path=/yxvvuh-63/ 2022-09-13T10:00:46.926 DEBUG:boto:auth_path=/yxvvuh-63/ 2022-09-13T10:00:46.927 DEBUG:boto:Method: HEAD 2022-09-13T10:00:46.927 DEBUG:boto:Path: /yxvvuh-63/ 2022-09-13T10:00:46.928 DEBUG:boto:Data: 2022-09-13T10:00:46.929 DEBUG:boto:Headers: {} 2022-09-13T10:00:46.929 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T10:00:46.930 DEBUG:boto:Port: 8000 2022-09-13T10:00:46.930 DEBUG:boto:Params: {} 2022-09-13T10:00:46.932 DEBUG:boto:Token: None 2022-09-13T10:00:46.933 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 10:00:46 GMT /yxvvuh-63/ 2022-09-13T10:00:46.934 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oRZBJNrKgFogjVioiIcukiSXsNg= 2022-09-13T10:00:46.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 10:00:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oRZBJNrKgFogjVioiIcukiSXsNg=', 'Content-Length': '0'} 2022-09-13T10:00:46.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6cb192104949359-00632054ce-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 10:00:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T10:00:46.939 DEBUG:boto:path=/yxvvuh-63/c 2022-09-13T10:00:46.940 DEBUG:boto:auth_path=/yxvvuh-63/c 2022-09-13T10:00:46.941 DEBUG:boto:Method: PUT 2022-09-13T10:00:46.942 DEBUG:boto:Path: /yxvvuh-63/c 2022-09-13T10:00:46.943 DEBUG:boto:Data: 2022-09-13T10:00:46.944 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T10:00:46.945 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T10:00:46.946 DEBUG:boto:Port: 8000 2022-09-13T10:00:46.947 DEBUG:boto:Params: {} 2022-09-13T10:00:46.948 DEBUG:boto:Token: None 2022-09-13T10:00:46.949 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 13 Sep 2022 10:00:46 GMT /yxvvuh-63/c 2022-09-13T10:00:46.950 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:BrZ4Rwp89tvi/xJFPV7I6/14RQo= 2022-09-13T10:00:46.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 10:00:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BrZ4Rwp89tvi/xJFPV7I6/14RQo='} 2022-09-13T10:00:46.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051dd400d61bcc6f8-00632054ce-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 10:00:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T10:00:47.000 DEBUG:boto:path=/yxvvuh-63/ 2022-09-13T10:00:47.000 DEBUG:boto:auth_path=/yxvvuh-63/ 2022-09-13T10:00:47.001 DEBUG:boto:Method: HEAD 2022-09-13T10:00:47.001 DEBUG:boto:Path: /yxvvuh-63/ 2022-09-13T10:00:47.002 DEBUG:boto:Data: 2022-09-13T10:00:47.002 DEBUG:boto:Headers: {} 2022-09-13T10:00:47.003 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T10:00:47.004 DEBUG:boto:Port: 8000 2022-09-13T10:00:47.004 DEBUG:boto:Params: {} 2022-09-13T10:00:47.005 DEBUG:boto:Token: None 2022-09-13T10:00:47.005 DEBUG:boto:StringToSign: HEAD Tue, 13 Sep 2022 10:00:47 GMT /yxvvuh-63/ 2022-09-13T10:00:47.006 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:ZBXR2QXf6Cg+VjKzil/8M/D340M= 2022-09-13T10:00:47.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 10:00:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZBXR2QXf6Cg+VjKzil/8M/D340M=', 'Content-Length': '0'} 2022-09-13T10:00:47.009 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfd3513a15a93c5d-00632054cf-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 10:00:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T10:00:47.010 DEBUG:boto:path=/yxvvuh-63/d 2022-09-13T10:00:47.010 DEBUG:boto:auth_path=/yxvvuh-63/d 2022-09-13T10:00:47.011 DEBUG:boto:Method: PUT 2022-09-13T10:00:47.011 DEBUG:boto:Path: /yxvvuh-63/d 2022-09-13T10:00:47.012 DEBUG:boto:Data: 2022-09-13T10:00:47.012 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T10:00:47.013 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T10:00:47.014 DEBUG:boto:Port: 8000 2022-09-13T10:00:47.014 DEBUG:boto:Params: {} 2022-09-13T10:00:47.015 DEBUG:boto:Token: None 2022-09-13T10:00:47.016 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 13 Sep 2022 10:00:47 GMT /yxvvuh-63/d 2022-09-13T10:00:47.017 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:9Lz6Uu16XFX2yg2n1aih+cWj7Mc= 2022-09-13T10:00:47.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 10:00:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9Lz6Uu16XFX2yg2n1aih+cWj7Mc='} 2022-09-13T10:00:47.031 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b45666f057c1f5c-00632054cf-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 10:00:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T10:00:47.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T10:00:47.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T10:00:47.035 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T10:00:47.094 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T10:00:47.094 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T10:00:47.135 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T10:00:47.153 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T10:00:47.153 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T10:00:47.154 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T10:00:47.154 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T10:00:47.155 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T10:00:47.155 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T10:00:47.156 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T10:00:47.156 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T10:00:47.157 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T10:00:47.157 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T10:00:47.159 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663063246.853875_641.1", 2022-09-13T10:00:47.159 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T10:00:46.853875Z" 2022-09-13T10:00:47.161 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T10:00:47.161 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T10:00:47.162 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T10:00:47.162 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T10:00:47.163 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T10:00:47.163 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T10:00:47.165 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663063246.853875_641.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T10:00:47.166 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T10:00:47.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T10:00:47.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T10:00:47.169 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T10:00:47.224 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T10:00:47.225 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T10:05:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T10:05:47.225+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T10:10:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T10:10:47.226+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T10:15:47.229 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T10:15:47.225+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T10:20:47.229 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T10:20:47.227+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T10:25:47.231 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T10:25:47.227+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T10:30:47.230 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T10:30:47.228+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T10:35:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T10:35:47.228+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T10:40:47.231 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T10:40:47.229+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T10:45:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T10:45:47.230+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T10:50:47.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T10:50:47.257 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T10:50:47.229+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T10:50:47.258 INFO:teuthology.orchestra.run.smithi083.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T10:50:47.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2022-09-13T10:50:47.263 DEBUG:boto:path=/yxvvuh-64/ 2022-09-13T10:50:47.267 DEBUG:boto:auth_path=/yxvvuh-64/ 2022-09-13T10:50:47.268 DEBUG:boto:Method: PUT 2022-09-13T10:50:47.269 DEBUG:boto:Path: /yxvvuh-64/ 2022-09-13T10:50:47.270 DEBUG:boto:Data: 2022-09-13T10:50:47.271 DEBUG:boto:Headers: {} 2022-09-13T10:50:47.272 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T10:50:47.273 DEBUG:boto:Port: 8000 2022-09-13T10:50:47.274 DEBUG:boto:Params: {} 2022-09-13T10:50:47.276 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T10:50:47.277 DEBUG:boto:Token: None 2022-09-13T10:50:47.278 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 10:50:47 GMT /yxvvuh-64/ 2022-09-13T10:50:47.280 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:pAj3Gqg0qHalGIb5gy7l2SEtBhM= 2022-09-13T10:50:47.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 10:50:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pAj3Gqg0qHalGIb5gy7l2SEtBhM=', 'Content-Length': '0'} 2022-09-13T10:50:47.307 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009d8ba5ce80390655-0063206087-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 10:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T10:50:47.308 DEBUG:rgw_multi.tests:created bucket=yxvvuh-64 2022-09-13T10:50:47.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T10:50:47.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T10:50:47.313 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T10:50:47.374 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T10:50:47.375 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T10:50:47.415 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T10:50:47.440 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T10:50:47.441 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T10:50:47.441 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T10:50:47.442 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T10:50:47.442 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T10:50:47.442 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T10:50:47.443 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T10:50:47.443 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T10:50:47.444 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T10:50:47.444 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T10:50:47.445 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663066247.304693_728.1", 2022-09-13T10:50:47.445 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T10:50:47.304693Z" 2022-09-13T10:50:47.446 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T10:50:47.446 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T10:50:47.447 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T10:50:47.447 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T10:50:47.448 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T10:50:47.448 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T10:50:47.451 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663066247.304693_728.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T10:50:47.453 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T10:50:47.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T10:50:47.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T10:50:47.456 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T10:50:47.516 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T10:50:47.517 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T10:55:47.520 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T10:55:47.516+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T11:00:47.521 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T11:00:47.518+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T11:05:47.520 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T11:05:47.517+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T11:10:47.521 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T11:10:47.518+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T11:15:47.522 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T11:15:47.518+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T11:20:47.521 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T11:20:47.519+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T11:25:47.522 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T11:25:47.520+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T11:30:47.522 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T11:30:47.519+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T11:35:47.527 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T11:35:47.520+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T11:40:47.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T11:40:47.528 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T11:40:47.521+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T11:40:47.529 INFO:teuthology.orchestra.run.smithi083.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T11:40:47.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... FAIL 2022-09-13T11:40:47.533 DEBUG:boto:path=/yxvvuh-65/ 2022-09-13T11:40:47.534 DEBUG:boto:auth_path=/yxvvuh-65/ 2022-09-13T11:40:47.534 DEBUG:boto:Method: PUT 2022-09-13T11:40:47.535 DEBUG:boto:Path: /yxvvuh-65/ 2022-09-13T11:40:47.536 DEBUG:boto:Data: 2022-09-13T11:40:47.536 DEBUG:boto:Headers: {} 2022-09-13T11:40:47.537 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T11:40:47.537 DEBUG:boto:Port: 8000 2022-09-13T11:40:47.538 DEBUG:boto:Params: {} 2022-09-13T11:40:47.539 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T11:40:47.540 DEBUG:boto:Token: None 2022-09-13T11:40:47.541 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 11:40:47 GMT /yxvvuh-65/ 2022-09-13T11:40:47.541 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:hVXCgj6CC3K+I3jynIRFWWPmbNQ= 2022-09-13T11:40:47.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 11:40:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hVXCgj6CC3K+I3jynIRFWWPmbNQ=', 'Content-Length': '0'} 2022-09-13T11:40:47.574 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003691cd7ae24c11d3-0063206c3f-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 11:40:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T11:40:47.575 DEBUG:rgw_multi.tests:created bucket=yxvvuh-65 2022-09-13T11:40:47.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T11:40:47.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T11:40:47.577 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T11:40:47.824 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T11:40:47.824 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T11:40:47.892 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T11:40:47.907 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T11:40:47.908 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T11:40:47.908 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T11:40:47.908 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T11:40:47.908 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T11:40:47.909 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T11:40:47.909 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T11:40:47.909 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T11:40:47.910 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T11:40:47.910 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T11:40:47.910 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663069247.571769_814.1", 2022-09-13T11:40:47.910 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T11:40:47.571769Z" 2022-09-13T11:40:47.911 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T11:40:47.911 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T11:40:47.911 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T11:40:47.912 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T11:40:47.912 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T11:40:47.912 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T11:40:47.913 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663069247.571769_814.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T11:40:47.914 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T11:40:47.915 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T11:40:47.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T11:40:47.916 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T11:40:47.968 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T11:40:47.968 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T11:45:47.971 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T11:45:47.969+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T11:50:47.971 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T11:50:47.969+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T11:55:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T11:55:47.969+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T12:00:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T12:00:47.970+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T12:05:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T12:05:47.971+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T12:10:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T12:10:47.970+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T12:15:47.974 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T12:15:47.971+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T12:20:47.975 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T12:20:47.972+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T12:25:47.976 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T12:25:47.972+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T12:30:47.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T12:30:47.984 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T12:30:47.973+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T12:30:47.985 INFO:teuthology.orchestra.run.smithi083.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T12:30:47.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... FAIL 2022-09-13T12:30:47.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T12:30:47.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T12:30:47.991 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T12:30:48.053 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T12:30:48.053 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T12:30:48.088 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T12:30:48.102 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T12:30:48.103 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T12:30:48.103 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T12:30:48.103 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T12:30:48.103 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T12:30:48.104 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T12:30:48.104 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T12:30:48.104 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T12:30:48.104 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T12:30:48.105 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T12:30:48.105 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663069247.571769_814.1", 2022-09-13T12:30:48.105 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T11:40:47.571769Z" 2022-09-13T12:30:48.105 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T12:30:48.106 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T12:30:48.106 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T12:30:48.106 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T12:30:48.107 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T12:30:48.107 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T12:30:48.108 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663069247.571769_814.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T12:30:48.109 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T12:30:48.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T12:30:48.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T12:30:48.111 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T12:30:48.167 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T12:30:48.168 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T12:35:48.171 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T12:35:48.167+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T12:40:48.172 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T12:40:48.167+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T12:45:48.173 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T12:45:48.167+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T12:50:48.173 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T12:50:48.167+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T12:55:48.175 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T12:55:48.169+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T13:00:48.174 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T13:00:48.169+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T13:05:48.175 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T13:05:48.169+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T13:10:48.175 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T13:10:48.170+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T13:15:48.175 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T13:15:48.170+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T13:20:48.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T13:20:48.195 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T13:20:48.170+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T13:20:48.195 INFO:teuthology.orchestra.run.smithi083.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T13:20:48.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... FAIL 2022-09-13T13:20:48.199 DEBUG:boto:path=/yxvvuh-66/ 2022-09-13T13:20:48.205 DEBUG:boto:auth_path=/yxvvuh-66/ 2022-09-13T13:20:48.205 DEBUG:boto:Method: PUT 2022-09-13T13:20:48.206 DEBUG:boto:Path: /yxvvuh-66/ 2022-09-13T13:20:48.207 DEBUG:boto:Data: 2022-09-13T13:20:48.207 DEBUG:boto:Headers: {} 2022-09-13T13:20:48.208 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T13:20:48.208 DEBUG:boto:Port: 8000 2022-09-13T13:20:48.209 DEBUG:boto:Params: {} 2022-09-13T13:20:48.210 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T13:20:48.211 DEBUG:boto:Token: None 2022-09-13T13:20:48.212 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 13:20:48 GMT /yxvvuh-66/ 2022-09-13T13:20:48.213 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:rQtwI+SV5b/INxSvNTw32SCHKxg= 2022-09-13T13:20:48.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 13:20:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rQtwI+SV5b/INxSvNTw32SCHKxg=', 'Content-Length': '0'} 2022-09-13T13:20:48.241 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002d4e53e0ad90d229-00632083b0-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 13:20:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T13:20:48.242 DEBUG:rgw_multi.tests:created bucket=yxvvuh-66 2022-09-13T13:20:48.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T13:20:48.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T13:20:48.244 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T13:20:48.324 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T13:20:48.324 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T13:20:48.369 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T13:20:48.384 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T13:20:48.384 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T13:20:48.384 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T13:20:48.385 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T13:20:48.385 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T13:20:48.385 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T13:20:48.385 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T13:20:48.386 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T13:20:48.386 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T13:20:48.386 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T13:20:48.386 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663075248.236352_982.1", 2022-09-13T13:20:48.387 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T13:20:48.236352Z" 2022-09-13T13:20:48.387 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T13:20:48.387 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T13:20:48.387 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T13:20:48.388 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T13:20:48.388 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T13:20:48.388 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T13:20:48.389 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663075248.236352_982.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T13:20:48.390 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T13:20:48.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T13:20:48.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T13:20:48.392 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T13:20:48.446 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T13:20:48.447 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T13:25:48.451 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T13:25:48.445+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T13:30:48.450 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T13:30:48.445+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T13:35:48.451 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T13:35:48.446+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T13:40:48.451 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T13:40:48.445+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T13:45:48.451 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T13:45:48.446+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T13:50:48.451 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T13:50:48.446+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T13:55:48.452 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T13:55:48.447+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T14:00:48.453 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T14:00:48.447+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T14:05:48.453 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T14:05:48.448+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T14:10:48.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T14:10:48.479 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T14:10:48.449+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T14:10:48.480 INFO:teuthology.orchestra.run.smithi083.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T14:10:48.483 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... FAIL 2022-09-13T14:10:48.484 DEBUG:boto:path=/yxvvuh-67/ 2022-09-13T14:10:48.485 DEBUG:boto:auth_path=/yxvvuh-67/ 2022-09-13T14:10:48.486 DEBUG:boto:Method: PUT 2022-09-13T14:10:48.487 DEBUG:boto:Path: /yxvvuh-67/ 2022-09-13T14:10:48.487 DEBUG:boto:Data: 2022-09-13T14:10:48.488 DEBUG:boto:Headers: {} 2022-09-13T14:10:48.488 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T14:10:48.489 DEBUG:boto:Port: 8000 2022-09-13T14:10:48.489 DEBUG:boto:Params: {} 2022-09-13T14:10:48.490 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T14:10:48.491 DEBUG:boto:Token: None 2022-09-13T14:10:48.492 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 14:10:48 GMT /yxvvuh-67/ 2022-09-13T14:10:48.493 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:qR2tlxB6Z7JKf5yhsDRfve/RbqA= 2022-09-13T14:10:48.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 14:10:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qR2tlxB6Z7JKf5yhsDRfve/RbqA=', 'Content-Length': '0'} 2022-09-13T14:10:48.525 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fdc965eaeb628d62-0063208f68-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 14:10:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T14:10:48.526 DEBUG:rgw_multi.tests:created bucket=yxvvuh-67 2022-09-13T14:10:48.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T14:10:48.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T14:10:48.528 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T14:10:48.652 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T14:10:48.653 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T14:10:48.689 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T14:10:48.703 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T14:10:48.703 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T14:10:48.704 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T14:10:48.704 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T14:10:48.704 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T14:10:48.705 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T14:10:48.705 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T14:10:48.705 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T14:10:48.705 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T14:10:48.706 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T14:10:48.706 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663078248.520924_1066.1", 2022-09-13T14:10:48.706 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T14:10:48.520924Z" 2022-09-13T14:10:48.707 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T14:10:48.707 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T14:10:48.707 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663054100.288347_77.1", 2022-09-13T14:10:48.707 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T14:10:48.708 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T14:10:48.708 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T14:10:48.762 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663078248.520924_1066.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T14:10:48.763 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T14:10:48.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T14:10:48.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T14:10:48.765 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T14:10:48.824 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T14:10:48.824 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T14:15:48.827 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T14:15:48.822+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T14:20:48.829 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T14:20:48.823+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T14:25:48.828 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T14:25:48.824+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T14:30:48.829 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T14:30:48.824+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T14:35:48.829 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T14:35:48.825+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T14:40:48.830 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T14:40:48.826+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T14:45:48.830 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T14:45:48.826+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T14:50:48.830 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T14:50:48.826+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T14:55:48.831 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T14:55:48.826+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T15:00:48.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T15:00:48.838 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T15:00:48.828+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T15:00:48.839 INFO:teuthology.orchestra.run.smithi083.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T15:00:48.841 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... FAIL 2022-09-13T15:00:48.843 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-yxvvuh-1 2022-09-13T15:00:48.844 DEBUG:boto:Method: POST 2022-09-13T15:00:48.844 DEBUG:boto:Path: / 2022-09-13T15:00:48.845 DEBUG:boto:Data: 2022-09-13T15:00:48.846 DEBUG:boto:Headers: {} 2022-09-13T15:00:48.846 DEBUG:boto:Host: smithi036.front.sepia.ceph.com 2022-09-13T15:00:48.847 DEBUG:boto:Port: 8000 2022-09-13T15:00:48.847 DEBUG:boto:Params: {'RoleName': 'roles-yxvvuh-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2022-09-13T15:00:48.848 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T15:00:48.849 DEBUG:boto:Token: None 2022-09-13T15:00:48.851 DEBUG:boto:CanonicalRequest: POST / host:smithi036.front.sepia.ceph.com:8000 x-amz-date:20220913T150048Z host;x-amz-date 182fd3870aca11fdeeaf60d0f516606f850cd90dee9c18712b959899cb186515 2022-09-13T15:00:48.851 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220913T150048Z 20220913/front/smithi036/aws4_request 6a665b1d9881145a9e2573eb5d7f8c7c6c1bca3265f639221fd0aa604fe78d03 2022-09-13T15:00:48.852 DEBUG:boto:Signature: b115ef567229cc27b92e48e6241ecda26eb9d808086dfdeb62f669050d5060f0 2022-09-13T15:00:48.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150048Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi036/aws4_request,SignedHeaders=host;x-amz-date,Signature=b115ef567229cc27b92e48e6241ecda26eb9d808086dfdeb62f669050d5060f0', 'Host': 'smithi036.front.sepia.ceph.com:8000'} 2022-09-13T15:00:48.872 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a32d5bf7612eaef6-0063209b20-1452-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '731'), ('Date', 'Tue, 13 Sep 2022 15:00:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T15:00:48.873 DEBUG:boto:b'417d2fc0-e214-46af-8c1e-90aa63c986e4roles-yxvvuh-1/arn:aws:iam:::role/roles-yxvvuh-12022-09-13T15:00:48.859Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000a32d5bf7612eaef6-0063209b20-1452-test-zone1' 2022-09-13T15:00:48.876 INFO:rgw_multi.tests:create role zone=test-zone2 name=roles-yxvvuh-2 2022-09-13T15:00:48.876 DEBUG:boto:Method: POST 2022-09-13T15:00:48.877 DEBUG:boto:Path: / 2022-09-13T15:00:48.878 DEBUG:boto:Data: 2022-09-13T15:00:48.879 DEBUG:boto:Headers: {} 2022-09-13T15:00:48.880 DEBUG:boto:Host: smithi083.front.sepia.ceph.com 2022-09-13T15:00:48.881 DEBUG:boto:Port: 8000 2022-09-13T15:00:48.882 DEBUG:boto:Params: {'RoleName': 'roles-yxvvuh-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2022-09-13T15:00:48.883 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T15:00:48.884 DEBUG:boto:Token: None 2022-09-13T15:00:48.884 DEBUG:boto:CanonicalRequest: POST / host:smithi083.front.sepia.ceph.com:8000 x-amz-date:20220913T150048Z host;x-amz-date 0ac290b3ddeda42772c78a2b5e46b0854cedd10a8dd42ffefe9fc0d46fcbcfa1 2022-09-13T15:00:48.885 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220913T150048Z 20220913/front/smithi083/aws4_request 519cb6647ef7cd4b9862dfba68ac2b1ee4c57c590356602e42c701384f2d651c 2022-09-13T15:00:48.886 DEBUG:boto:Signature: bdc91ea2ec0e32a5787b11360264ffcccb87bd8310134fba29acd76b70bc25ac 2022-09-13T15:00:48.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150048Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi083/aws4_request,SignedHeaders=host;x-amz-date,Signature=bdc91ea2ec0e32a5787b11360264ffcccb87bd8310134fba29acd76b70bc25ac', 'Host': 'smithi083.front.sepia.ceph.com:8000'} 2022-09-13T15:00:48.889 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T15:00:48.889 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T15:00:49.174 DEBUG:boto:Token: None 2022-09-13T15:00:49.175 DEBUG:boto:CanonicalRequest: POST / host:smithi083.front.sepia.ceph.com:8000 x-amz-date:20220913T150049Z host;x-amz-date 0ac290b3ddeda42772c78a2b5e46b0854cedd10a8dd42ffefe9fc0d46fcbcfa1 2022-09-13T15:00:49.176 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220913T150049Z 20220913/front/smithi083/aws4_request 423774d31646d183af46cdbcbd6f450d4a47b0ac0f1c47e051ae23fb4ead72eb 2022-09-13T15:00:49.177 DEBUG:boto:Signature: aa8ec3b2d90d7ff0a91f1ad46ea6c66b794d3587f2d79730a94a3b7d34351a8a 2022-09-13T15:00:49.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150049Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi083/aws4_request,SignedHeaders=host;x-amz-date,Signature=aa8ec3b2d90d7ff0a91f1ad46ea6c66b794d3587f2d79730a94a3b7d34351a8a', 'Host': 'smithi083.front.sepia.ceph.com:8000'} 2022-09-13T15:00:49.180 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T15:00:49.181 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T15:00:50.187 DEBUG:boto:Token: None 2022-09-13T15:00:50.188 DEBUG:boto:CanonicalRequest: POST / host:smithi083.front.sepia.ceph.com:8000 x-amz-date:20220913T150050Z host;x-amz-date 0ac290b3ddeda42772c78a2b5e46b0854cedd10a8dd42ffefe9fc0d46fcbcfa1 2022-09-13T15:00:50.189 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220913T150050Z 20220913/front/smithi083/aws4_request 5621475449910c5b0fd59decfd542b4015175c32f52186241c3ee89f5e4796e2 2022-09-13T15:00:50.190 DEBUG:boto:Signature: 5d7474e3e0a90693e75bb0901c80d86d54352a8c62a89cfd820d0ba1daecce3d 2022-09-13T15:00:50.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150050Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi083/aws4_request,SignedHeaders=host;x-amz-date,Signature=5d7474e3e0a90693e75bb0901c80d86d54352a8c62a89cfd820d0ba1daecce3d', 'Host': 'smithi083.front.sepia.ceph.com:8000'} 2022-09-13T15:00:50.192 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T15:00:50.193 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T15:00:52.420 DEBUG:boto:Token: None 2022-09-13T15:00:52.421 DEBUG:boto:CanonicalRequest: POST / host:smithi083.front.sepia.ceph.com:8000 x-amz-date:20220913T150052Z host;x-amz-date 0ac290b3ddeda42772c78a2b5e46b0854cedd10a8dd42ffefe9fc0d46fcbcfa1 2022-09-13T15:00:52.422 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220913T150052Z 20220913/front/smithi083/aws4_request fafc66dc6cf5f1ceb71d562e70817708451517ebedadd68f94ab312809f71aed 2022-09-13T15:00:52.423 DEBUG:boto:Signature: 2f67f5a102a06fc6783ec35acc03f00be9df1a095cb4f4e578c58b385d702e25 2022-09-13T15:00:52.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150052Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi083/aws4_request,SignedHeaders=host;x-amz-date,Signature=2f67f5a102a06fc6783ec35acc03f00be9df1a095cb4f4e578c58b385d702e25', 'Host': 'smithi083.front.sepia.ceph.com:8000'} 2022-09-13T15:00:52.425 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T15:00:52.427 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T15:00:53.080 DEBUG:boto:Token: None 2022-09-13T15:00:53.081 DEBUG:boto:CanonicalRequest: POST / host:smithi083.front.sepia.ceph.com:8000 x-amz-date:20220913T150053Z host;x-amz-date 0ac290b3ddeda42772c78a2b5e46b0854cedd10a8dd42ffefe9fc0d46fcbcfa1 2022-09-13T15:00:53.082 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220913T150053Z 20220913/front/smithi083/aws4_request 0f6c44b0c23a1e092ebef6d9beeb891f66b49b74ce509b64ff17bdbdce3831ab 2022-09-13T15:00:53.082 DEBUG:boto:Signature: 550f517afdf231f748ff1fb29af8dadd1c063d3c43b1f2529be3947bb78868f9 2022-09-13T15:00:53.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150053Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi083/aws4_request,SignedHeaders=host;x-amz-date,Signature=550f517afdf231f748ff1fb29af8dadd1c063d3c43b1f2529be3947bb78868f9', 'Host': 'smithi083.front.sepia.ceph.com:8000'} 2022-09-13T15:00:53.085 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T15:00:53.085 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T15:01:06.537 DEBUG:boto:Token: None 2022-09-13T15:01:06.539 DEBUG:boto:CanonicalRequest: POST / host:smithi083.front.sepia.ceph.com:8000 x-amz-date:20220913T150106Z host;x-amz-date 0ac290b3ddeda42772c78a2b5e46b0854cedd10a8dd42ffefe9fc0d46fcbcfa1 2022-09-13T15:01:06.540 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220913T150106Z 20220913/front/smithi083/aws4_request 4a177d9abc7fd81e9344faac4d181e6bcb05893f098af4557a534b99a1ca9aaf 2022-09-13T15:01:06.540 DEBUG:boto:Signature: 0942373b6f91725b3f1c392ffdf93b6698f7044aa9f8db0bb101f098df7d9436 2022-09-13T15:01:06.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150106Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi083/aws4_request,SignedHeaders=host;x-amz-date,Signature=0942373b6f91725b3f1c392ffdf93b6698f7044aa9f8db0bb101f098df7d9436', 'Host': 'smithi083.front.sepia.ceph.com:8000'} 2022-09-13T15:01:06.543 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T15:01:06.544 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T15:01:25.237 DEBUG:boto:Token: None 2022-09-13T15:01:25.238 DEBUG:boto:CanonicalRequest: POST / host:smithi083.front.sepia.ceph.com:8000 x-amz-date:20220913T150125Z host;x-amz-date 0ac290b3ddeda42772c78a2b5e46b0854cedd10a8dd42ffefe9fc0d46fcbcfa1 2022-09-13T15:01:25.239 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220913T150125Z 20220913/front/smithi083/aws4_request ec3b126eb66666510ea3860a2ac85d159bfdb6a9fe6355c7d1114e1ae80fa7fe 2022-09-13T15:01:25.240 DEBUG:boto:Signature: 28026bfb974d3348a071116a256f742c3bceceb5d490e989b10ba7935a96456f 2022-09-13T15:01:25.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150125Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi083/aws4_request,SignedHeaders=host;x-amz-date,Signature=28026bfb974d3348a071116a256f742c3bceceb5d490e989b10ba7935a96456f', 'Host': 'smithi083.front.sepia.ceph.com:8000'} 2022-09-13T15:01:25.243 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-13T15:01:25.244 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T15:02:09.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ERROR 2022-09-13T15:02:09.432 DEBUG:boto:path=/yxvvuh-68/ 2022-09-13T15:02:09.433 DEBUG:boto:auth_path=/yxvvuh-68/ 2022-09-13T15:02:09.434 DEBUG:boto:Method: PUT 2022-09-13T15:02:09.434 DEBUG:boto:Path: /yxvvuh-68/ 2022-09-13T15:02:09.435 DEBUG:boto:Data: 2022-09-13T15:02:09.435 DEBUG:boto:Headers: {} 2022-09-13T15:02:09.436 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T15:02:09.436 DEBUG:boto:Port: 8000 2022-09-13T15:02:09.437 DEBUG:boto:Params: {} 2022-09-13T15:02:09.438 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T15:02:09.439 DEBUG:boto:Token: None 2022-09-13T15:02:09.439 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 15:02:09 GMT /yxvvuh-68/ 2022-09-13T15:02:09.440 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:u0tWWDjh9hMJPd85SxEiEkC2ANQ= 2022-09-13T15:02:09.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 15:02:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:u0tWWDjh9hMJPd85SxEiEkC2ANQ=', 'Content-Length': '0'} 2022-09-13T15:02:09.478 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000086b3e5c57e96d14d-0063209b71-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 15:02:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T15:02:09.479 DEBUG:rgw_multi.tests:created bucket=yxvvuh-68 2022-09-13T15:02:09.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T15:02:09.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T15:02:09.482 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T15:02:09.540 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T15:02:09.556 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T15:02:09.585 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T15:02:09.599 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T15:02:09.599 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T15:02:09.599 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T15:02:09.600 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T15:02:09.600 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T15:02:09.600 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T15:02:09.600 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T15:02:09.601 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T15:02:09.601 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T15:02:09.601 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T15:02:09.601 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663081329.473165_1154.1", 2022-09-13T15:02:09.602 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T15:02:09.473165Z" 2022-09-13T15:02:09.602 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T15:02:09.602 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T15:02:09.603 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663081248.863419_600.1", 2022-09-13T15:02:09.603 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T15:00:48.863419Z" 2022-09-13T15:02:09.603 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T15:02:09.604 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T15:02:09.605 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663081329.473165_1154.1', 3: '1_1663081248.863419_600.1'} 2022-09-13T15:02:09.606 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T15:02:09.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T15:02:09.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T15:02:09.608 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T15:02:09.660 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T15:02:09.661 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T15:07:09.665 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T15:07:09.660+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T15:12:09.664 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T15:12:09.660+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T15:17:09.665 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T15:17:09.660+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T15:22:09.665 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T15:22:09.660+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T15:27:09.666 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T15:27:09.661+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T15:32:09.666 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T15:32:09.661+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T15:37:09.666 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T15:37:09.662+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T15:42:09.667 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T15:42:09.663+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T15:47:09.667 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T15:47:09.663+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T15:52:09.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T15:52:09.673 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T15:52:09.664+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T15:52:09.673 INFO:teuthology.orchestra.run.smithi083.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T15:52:09.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... FAIL 2022-09-13T15:52:09.678 DEBUG:boto:path=/yxvvuh-69/ 2022-09-13T15:52:09.679 DEBUG:boto:auth_path=/yxvvuh-69/ 2022-09-13T15:52:09.680 DEBUG:boto:Method: PUT 2022-09-13T15:52:09.680 DEBUG:boto:Path: /yxvvuh-69/ 2022-09-13T15:52:09.681 DEBUG:boto:Data: 2022-09-13T15:52:09.682 DEBUG:boto:Headers: {} 2022-09-13T15:52:09.682 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T15:52:09.683 DEBUG:boto:Port: 8000 2022-09-13T15:52:09.684 DEBUG:boto:Params: {} 2022-09-13T15:52:09.685 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T15:52:09.686 DEBUG:boto:Token: None 2022-09-13T15:52:09.687 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 15:52:09 GMT /yxvvuh-69/ 2022-09-13T15:52:09.688 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:oN2jh2TiWO+XBGv9YrJtt7COK48= 2022-09-13T15:52:09.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 15:52:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oN2jh2TiWO+XBGv9YrJtt7COK48=', 'Content-Length': '0'} 2022-09-13T15:52:09.717 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002299a2329e6960e2-006320a729-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 15:52:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T15:52:09.717 DEBUG:rgw_multi.tests:created bucket=yxvvuh-69 2022-09-13T15:52:09.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T15:52:09.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T15:52:09.720 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T15:52:09.955 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T15:52:09.956 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T15:52:09.997 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T15:52:10.018 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T15:52:10.019 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T15:52:10.019 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T15:52:10.019 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T15:52:10.020 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T15:52:10.020 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T15:52:10.020 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T15:52:10.020 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T15:52:10.021 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T15:52:10.021 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T15:52:10.021 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663084329.711421_1240.1", 2022-09-13T15:52:10.021 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T15:52:09.711421Z" 2022-09-13T15:52:10.021 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T15:52:10.022 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T15:52:10.022 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663081248.863419_600.1", 2022-09-13T15:52:10.022 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T15:00:48.863419Z" 2022-09-13T15:52:10.023 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T15:52:10.023 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T15:52:10.024 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663084329.711421_1240.1', 3: '1_1663081248.863419_600.1'} 2022-09-13T15:52:10.025 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T15:52:10.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T15:52:10.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T15:52:10.027 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T15:52:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T15:52:10.080 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T15:57:10.083 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T15:57:10.078+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T16:02:10.084 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T16:02:10.079+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T16:07:10.084 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T16:07:10.080+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T16:12:10.085 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T16:12:10.080+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T16:17:10.085 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T16:17:10.081+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T16:22:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T16:22:10.081+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T16:27:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T16:27:10.082+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T16:32:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T16:32:10.082+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T16:37:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T16:37:10.082+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T16:42:10.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T16:42:10.091 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T16:42:10.083+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T16:42:10.092 INFO:teuthology.orchestra.run.smithi083.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T16:42:10.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... FAIL 2022-09-13T16:42:10.096 DEBUG:boto:path=/yxvvuh-70/ 2022-09-13T16:42:10.097 DEBUG:boto:auth_path=/yxvvuh-70/ 2022-09-13T16:42:10.098 DEBUG:boto:Method: PUT 2022-09-13T16:42:10.098 DEBUG:boto:Path: /yxvvuh-70/ 2022-09-13T16:42:10.099 DEBUG:boto:Data: 2022-09-13T16:42:10.099 DEBUG:boto:Headers: {} 2022-09-13T16:42:10.100 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T16:42:10.101 DEBUG:boto:Port: 8000 2022-09-13T16:42:10.101 DEBUG:boto:Params: {} 2022-09-13T16:42:10.102 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T16:42:10.103 DEBUG:boto:Token: None 2022-09-13T16:42:10.104 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 16:42:10 GMT /yxvvuh-70/ 2022-09-13T16:42:10.105 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:1RZM0P9gEvvGqkUn+Sj+GyG0IYI= 2022-09-13T16:42:10.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 16:42:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1RZM0P9gEvvGqkUn+Sj+GyG0IYI=', 'Content-Length': '0'} 2022-09-13T16:42:10.134 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d0ff00ece732f25f-006320b2e2-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 16:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T16:42:10.135 DEBUG:rgw_multi.tests:created bucket=yxvvuh-70 2022-09-13T16:42:10.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T16:42:10.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T16:42:10.138 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T16:42:10.205 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T16:42:10.206 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T16:42:10.256 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T16:42:10.278 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T16:42:10.279 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T16:42:10.279 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T16:42:10.279 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T16:42:10.280 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T16:42:10.280 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T16:42:10.280 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T16:42:10.281 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T16:42:10.281 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T16:42:10.281 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T16:42:10.281 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663087330.130711_1326.1", 2022-09-13T16:42:10.282 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T16:42:10.130711Z" 2022-09-13T16:42:10.282 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T16:42:10.282 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T16:42:10.282 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663081248.863419_600.1", 2022-09-13T16:42:10.283 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T15:00:48.863419Z" 2022-09-13T16:42:10.283 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T16:42:10.283 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T16:42:10.284 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663087330.130711_1326.1', 3: '1_1663081248.863419_600.1'} 2022-09-13T16:42:10.285 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T16:42:10.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T16:42:10.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T16:42:10.287 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T16:42:10.340 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T16:42:10.341 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T16:47:10.344 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T16:47:10.339+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T16:52:10.346 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T16:52:10.340+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T16:57:10.346 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T16:57:10.341+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T17:02:10.346 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T17:02:10.341+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T17:07:10.346 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T17:07:10.342+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T17:12:10.347 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T17:12:10.341+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T17:17:10.347 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T17:17:10.342+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T17:22:10.347 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T17:22:10.342+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T17:27:10.348 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T17:27:10.343+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T17:32:10.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T17:32:10.355 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T17:32:10.343+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T17:32:10.355 INFO:teuthology.orchestra.run.smithi083.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T17:32:10.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... FAIL 2022-09-13T17:32:10.360 DEBUG:boto:path=/yxvvuh-71/ 2022-09-13T17:32:10.371 DEBUG:boto:auth_path=/yxvvuh-71/ 2022-09-13T17:32:10.372 DEBUG:boto:Method: PUT 2022-09-13T17:32:10.373 DEBUG:boto:Path: /yxvvuh-71/ 2022-09-13T17:32:10.374 DEBUG:boto:Data: 2022-09-13T17:32:10.375 DEBUG:boto:Headers: {} 2022-09-13T17:32:10.375 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T17:32:10.376 DEBUG:boto:Port: 8000 2022-09-13T17:32:10.377 DEBUG:boto:Params: {} 2022-09-13T17:32:10.379 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T17:32:10.380 DEBUG:boto:Token: None 2022-09-13T17:32:10.381 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 17:32:10 GMT /yxvvuh-71/ 2022-09-13T17:32:10.382 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:nWafIa8uhVz6iOLhHgYxvod2AS8= 2022-09-13T17:32:10.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 17:32:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nWafIa8uhVz6iOLhHgYxvod2AS8=', 'Content-Length': '0'} 2022-09-13T17:32:10.410 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000100a34e1dc03416e-006320be9a-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 17:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T17:32:10.411 DEBUG:rgw_multi.tests:created bucket=yxvvuh-71 2022-09-13T17:32:10.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T17:32:10.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T17:32:10.415 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T17:32:10.491 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T17:32:10.491 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T17:32:10.524 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T17:32:10.559 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T17:32:10.559 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T17:32:10.559 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T17:32:10.560 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T17:32:10.560 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T17:32:10.560 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T17:32:10.561 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T17:32:10.561 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T17:32:10.562 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T17:32:10.562 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T17:32:10.563 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663090330.404695_1412.1", 2022-09-13T17:32:10.563 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T17:32:10.404695Z" 2022-09-13T17:32:10.564 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T17:32:10.564 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T17:32:10.564 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663081248.863419_600.1", 2022-09-13T17:32:10.564 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T15:00:48.863419Z" 2022-09-13T17:32:10.565 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T17:32:10.565 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T17:32:10.567 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663090330.404695_1412.1', 3: '1_1663081248.863419_600.1'} 2022-09-13T17:32:10.567 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T17:32:10.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T17:32:10.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T17:32:10.569 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T17:32:10.622 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T17:32:10.622 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T17:37:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T17:37:10.621+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T17:42:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T17:42:10.622+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T17:47:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T17:47:10.623+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T17:52:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T17:52:10.623+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T17:57:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T17:57:10.624+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T18:02:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T18:02:10.623+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T18:07:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T18:07:10.624+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T18:12:10.630 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T18:12:10.624+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T18:17:10.630 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T18:17:10.625+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T18:22:10.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T18:22:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T18:22:10.626+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T18:22:10.642 INFO:teuthology.orchestra.run.smithi083.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T18:22:10.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2022-09-13T18:22:10.649 DEBUG:boto:path=/yxvvuh-72/ 2022-09-13T18:22:10.651 DEBUG:boto:auth_path=/yxvvuh-72/ 2022-09-13T18:22:10.652 DEBUG:boto:Method: PUT 2022-09-13T18:22:10.653 DEBUG:boto:Path: /yxvvuh-72/ 2022-09-13T18:22:10.654 DEBUG:boto:Data: 2022-09-13T18:22:10.655 DEBUG:boto:Headers: {} 2022-09-13T18:22:10.656 DEBUG:boto:Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T18:22:10.657 DEBUG:boto:Port: 8000 2022-09-13T18:22:10.658 DEBUG:boto:Params: {} 2022-09-13T18:22:10.660 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T18:22:10.661 DEBUG:boto:Token: None 2022-09-13T18:22:10.662 DEBUG:boto:StringToSign: PUT Tue, 13 Sep 2022 18:22:10 GMT /yxvvuh-72/ 2022-09-13T18:22:10.663 DEBUG:boto:Signature: AWS RICCPJUVKDGV9UEGFN6Z:a0urf2N/1m5kw9uSKoSjlHj+AHQ= 2022-09-13T18:22:10.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 18:22:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:a0urf2N/1m5kw9uSKoSjlHj+AHQ=', 'Content-Length': '0'} 2022-09-13T18:22:10.690 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009617ebeba0a61a12-006320ca52-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 18:22:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T18:22:10.691 DEBUG:rgw_multi.tests:created bucket=yxvvuh-72 2022-09-13T18:22:10.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T18:22:10.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T18:22:10.694 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T18:22:10.789 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2022-09-13T18:22:10.790 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T18:22:10.824 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T18:22:10.838 INFO:teuthology.orchestra.run.smithi036.stdout:[ 2022-09-13T18:22:10.838 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T18:22:10.839 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663053952.976234_34.1", 2022-09-13T18:22:10.839 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T18:22:10.840 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T18:22:10.840 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T18:22:10.840 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "", 2022-09-13T18:22:10.840 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "0.000000" 2022-09-13T18:22:10.841 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T18:22:10.841 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T18:22:10.841 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663093330.686299_1498.1", 2022-09-13T18:22:10.842 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T18:22:10.686299Z" 2022-09-13T18:22:10.842 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2022-09-13T18:22:10.842 INFO:teuthology.orchestra.run.smithi036.stdout: { 2022-09-13T18:22:10.842 INFO:teuthology.orchestra.run.smithi036.stdout: "marker": "1_1663081248.863419_600.1", 2022-09-13T18:22:10.843 INFO:teuthology.orchestra.run.smithi036.stdout: "last_update": "2022-09-13T15:00:48.863419Z" 2022-09-13T18:22:10.843 INFO:teuthology.orchestra.run.smithi036.stdout: } 2022-09-13T18:22:10.843 INFO:teuthology.orchestra.run.smithi036.stdout:] 2022-09-13T18:22:10.845 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663093330.686299_1498.1', 3: '1_1663081248.863419_600.1'} 2022-09-13T18:22:10.845 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-13T18:22:10.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T18:22:10.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T18:22:10.847 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T18:22:10.907 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setuser ceph since I am not root 2022-09-13T18:22:10.907 INFO:teuthology.orchestra.run.smithi083.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T18:27:10.910 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T18:27:10.906+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T18:32:10.911 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T18:32:10.905+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T18:37:10.912 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T18:37:10.907+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T18:42:10.912 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T18:42:10.907+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T18:47:10.912 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T18:47:10.907+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T18:52:10.925 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T18:52:10.908+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T18:57:10.914 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T18:57:10.908+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:02:10.914 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T19:02:10.909+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:07:10.914 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T19:07:10.910+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:10.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T19:12:10.922 INFO:teuthology.orchestra.run.smithi083.stderr:2022-09-13T19:12:10.910+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:10.923 INFO:teuthology.orchestra.run.smithi083.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T19:12:10.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2022-09-13T19:12:10.927 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.928 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:10.928 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_sync_enable_right_after_disable 2022-09-13T19:12:10.929 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:10.929 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:10.930 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:10.930 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:10.930 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1153, in test_bucket_sync_enable_right_after_disable 2022-09-13T19:12:10.931 INFO:tasks.rgw_multisite_tests: buckets, zone_bucket = create_bucket_per_zone(zonegroup_conns) 2022-09-13T19:12:10.931 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 523, in create_bucket_per_zone 2022-09-13T19:12:10.932 INFO:tasks.rgw_multisite_tests: bucket = zone.create_bucket(bucket_name) 2022-09-13T19:12:10.932 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/rgw_multi/zone_rados.py", line 55, in create_bucket 2022-09-13T19:12:10.933 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2022-09-13T19:12:10.933 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 619, in create_bucket 2022-09-13T19:12:10.933 INFO:tasks.rgw_multisite_tests: data=data) 2022-09-13T19:12:10.934 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 671, in make_request 2022-09-13T19:12:10.934 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2022-09-13T19:12:10.935 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2022-09-13T19:12:10.935 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2022-09-13T19:12:10.936 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1030, in _mexe 2022-09-13T19:12:10.936 INFO:tasks.rgw_multisite_tests: raise ex 2022-09-13T19:12:10.937 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 943, in _mexe 2022-09-13T19:12:10.937 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2022-09-13T19:12:10.937 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1285, in request 2022-09-13T19:12:10.938 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2022-09-13T19:12:10.938 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1331, in _send_request 2022-09-13T19:12:10.939 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2022-09-13T19:12:10.939 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1280, in endheaders 2022-09-13T19:12:10.939 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2022-09-13T19:12:10.940 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1046, in _send_output 2022-09-13T19:12:10.940 INFO:tasks.rgw_multisite_tests: self.send(msg) 2022-09-13T19:12:10.941 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 984, in send 2022-09-13T19:12:10.941 INFO:tasks.rgw_multisite_tests: self.connect() 2022-09-13T19:12:10.941 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 956, in connect 2022-09-13T19:12:10.942 INFO:tasks.rgw_multisite_tests: (self.host,self.port), self.timeout, self.source_address) 2022-09-13T19:12:10.942 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/gevent/socket.py", line 101, in create_connection 2022-09-13T19:12:10.943 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2022-09-13T19:12:10.943 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/gevent/_socketcommon.py", line 607, in connect 2022-09-13T19:12:10.944 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2022-09-13T19:12:10.944 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2022-09-13T19:12:10.944 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:10.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=yxvvuh-55 2022-09-13T19:12:10.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-55/ 2022-09-13T19:12:10.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-55/ 2022-09-13T19:12:10.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:10.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-55/ 2022-09-13T19:12:10.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:10.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:10.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:10.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:10.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:10.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:10.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:10.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:10.951 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:10.952 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.952 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.952 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:04 GMT 2022-09-13T19:12:10.953 INFO:tasks.rgw_multisite_tests:/yxvvuh-55/ 2022-09-13T19:12:10.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:10.954 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:dxFxdzvpERSbiiKAYUOBs59shtE= 2022-09-13T19:12:10.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:dxFxdzvpERSbiiKAYUOBs59shtE=', 'Content-Length': '0'} 2022-09-13T19:12:10.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000071139a133783feb8-0063203c80-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:17:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:10.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=yxvvuh-56 2022-09-13T19:12:10.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-56/ 2022-09-13T19:12:10.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-56/ 2022-09-13T19:12:10.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:10.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-56/ 2022-09-13T19:12:10.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:10.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:10.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:10.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:10.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:10.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:10.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:10.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:10.961 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:10.961 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.962 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.962 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:04 GMT 2022-09-13T19:12:10.962 INFO:tasks.rgw_multisite_tests:/yxvvuh-56/ 2022-09-13T19:12:10.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:10.963 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc= 2022-09-13T19:12:10.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc=', 'Content-Length': '0'} 2022-09-13T19:12:10.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:10.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:10.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:10.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:10.966 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:10.966 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.966 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.967 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:04 GMT 2022-09-13T19:12:10.967 INFO:tasks.rgw_multisite_tests:/yxvvuh-56/ 2022-09-13T19:12:10.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:10.968 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc= 2022-09-13T19:12:10.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc=', 'Content-Length': '0'} 2022-09-13T19:12:10.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:10.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:10.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:10.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:10.971 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:10.971 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.972 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.972 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:04 GMT 2022-09-13T19:12:10.973 INFO:tasks.rgw_multisite_tests:/yxvvuh-56/ 2022-09-13T19:12:10.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:10.974 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc= 2022-09-13T19:12:10.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc=', 'Content-Length': '0'} 2022-09-13T19:12:10.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:10.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:10.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:10.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:10.976 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:10.976 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.977 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.977 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:04 GMT 2022-09-13T19:12:10.978 INFO:tasks.rgw_multisite_tests:/yxvvuh-56/ 2022-09-13T19:12:10.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:10.979 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc= 2022-09-13T19:12:10.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc=', 'Content-Length': '0'} 2022-09-13T19:12:10.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:10.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:10.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:10.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:10.981 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:10.981 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.982 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.982 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:04 GMT 2022-09-13T19:12:10.983 INFO:tasks.rgw_multisite_tests:/yxvvuh-56/ 2022-09-13T19:12:10.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:10.983 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc= 2022-09-13T19:12:10.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc=', 'Content-Length': '0'} 2022-09-13T19:12:10.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:10.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:10.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:10.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:10.986 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:10.986 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.987 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.987 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:04 GMT 2022-09-13T19:12:10.988 INFO:tasks.rgw_multisite_tests:/yxvvuh-56/ 2022-09-13T19:12:10.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:10.988 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc= 2022-09-13T19:12:10.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc=', 'Content-Length': '0'} 2022-09-13T19:12:10.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:10.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:10.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:10.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:10.991 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:10.991 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.992 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.992 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:04 GMT 2022-09-13T19:12:10.993 INFO:tasks.rgw_multisite_tests:/yxvvuh-56/ 2022-09-13T19:12:10.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:10.994 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc= 2022-09-13T19:12:10.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:04 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:IEBsyxNsjLTviuaZocRbPDSB/oc=', 'Content-Length': '0'} 2022-09-13T19:12:10.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:10.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:10.996 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:10.996 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:10.997 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:10.997 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_sync_disable_enable 2022-09-13T19:12:10.998 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:10.998 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:10.999 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:10.999 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:11.000 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1188, in test_bucket_sync_disable_enable 2022-09-13T19:12:11.000 INFO:tasks.rgw_multisite_tests: buckets, zone_bucket = create_bucket_per_zone(zonegroup_conns) 2022-09-13T19:12:11.000 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 523, in create_bucket_per_zone 2022-09-13T19:12:11.001 INFO:tasks.rgw_multisite_tests: bucket = zone.create_bucket(bucket_name) 2022-09-13T19:12:11.001 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/rgw_multi/zone_rados.py", line 55, in create_bucket 2022-09-13T19:12:11.002 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2022-09-13T19:12:11.002 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 619, in create_bucket 2022-09-13T19:12:11.003 INFO:tasks.rgw_multisite_tests: data=data) 2022-09-13T19:12:11.003 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 671, in make_request 2022-09-13T19:12:11.004 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2022-09-13T19:12:11.004 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2022-09-13T19:12:11.005 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2022-09-13T19:12:11.005 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1030, in _mexe 2022-09-13T19:12:11.006 INFO:tasks.rgw_multisite_tests: raise ex 2022-09-13T19:12:11.006 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 943, in _mexe 2022-09-13T19:12:11.006 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2022-09-13T19:12:11.007 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1285, in request 2022-09-13T19:12:11.007 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2022-09-13T19:12:11.008 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1331, in _send_request 2022-09-13T19:12:11.008 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2022-09-13T19:12:11.008 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1280, in endheaders 2022-09-13T19:12:11.009 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2022-09-13T19:12:11.009 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1046, in _send_output 2022-09-13T19:12:11.010 INFO:tasks.rgw_multisite_tests: self.send(msg) 2022-09-13T19:12:11.010 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 984, in send 2022-09-13T19:12:11.011 INFO:tasks.rgw_multisite_tests: self.connect() 2022-09-13T19:12:11.011 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 956, in connect 2022-09-13T19:12:11.012 INFO:tasks.rgw_multisite_tests: (self.host,self.port), self.timeout, self.source_address) 2022-09-13T19:12:11.012 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/gevent/socket.py", line 101, in create_connection 2022-09-13T19:12:11.012 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2022-09-13T19:12:11.013 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/gevent/_socketcommon.py", line 607, in connect 2022-09-13T19:12:11.013 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2022-09-13T19:12:11.014 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2022-09-13T19:12:11.014 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:11.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=yxvvuh-57 2022-09-13T19:12:11.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-57/ 2022-09-13T19:12:11.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-57/ 2022-09-13T19:12:11.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:11.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-57/ 2022-09-13T19:12:11.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.020 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.020 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.020 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.021 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:40 GMT 2022-09-13T19:12:11.021 INFO:tasks.rgw_multisite_tests:/yxvvuh-57/ 2022-09-13T19:12:11.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.022 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:X7K9bemZQ0pV5EkIAlDlqjlp5ME= 2022-09-13T19:12:11.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:X7K9bemZQ0pV5EkIAlDlqjlp5ME=', 'Content-Length': '0'} 2022-09-13T19:12:11.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000032187529ff0712d2-0063203ca4-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:17:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=yxvvuh-58 2022-09-13T19:12:11.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-58/ 2022-09-13T19:12:11.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-58/ 2022-09-13T19:12:11.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:11.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-58/ 2022-09-13T19:12:11.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.030 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.030 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.031 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.031 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:40 GMT 2022-09-13T19:12:11.031 INFO:tasks.rgw_multisite_tests:/yxvvuh-58/ 2022-09-13T19:12:11.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.032 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ= 2022-09-13T19:12:11.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ=', 'Content-Length': '0'} 2022-09-13T19:12:11.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.035 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.035 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.036 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.036 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:40 GMT 2022-09-13T19:12:11.036 INFO:tasks.rgw_multisite_tests:/yxvvuh-58/ 2022-09-13T19:12:11.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.037 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ= 2022-09-13T19:12:11.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ=', 'Content-Length': '0'} 2022-09-13T19:12:11.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.040 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.040 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.041 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.041 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:40 GMT 2022-09-13T19:12:11.041 INFO:tasks.rgw_multisite_tests:/yxvvuh-58/ 2022-09-13T19:12:11.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.042 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ= 2022-09-13T19:12:11.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ=', 'Content-Length': '0'} 2022-09-13T19:12:11.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.045 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.045 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.046 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.046 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:40 GMT 2022-09-13T19:12:11.047 INFO:tasks.rgw_multisite_tests:/yxvvuh-58/ 2022-09-13T19:12:11.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.047 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ= 2022-09-13T19:12:11.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ=', 'Content-Length': '0'} 2022-09-13T19:12:11.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.050 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.051 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.051 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.052 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:40 GMT 2022-09-13T19:12:11.052 INFO:tasks.rgw_multisite_tests:/yxvvuh-58/ 2022-09-13T19:12:11.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.053 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ= 2022-09-13T19:12:11.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ=', 'Content-Length': '0'} 2022-09-13T19:12:11.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.055 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.056 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.057 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.057 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:40 GMT 2022-09-13T19:12:11.057 INFO:tasks.rgw_multisite_tests:/yxvvuh-58/ 2022-09-13T19:12:11.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.058 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ= 2022-09-13T19:12:11.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ=', 'Content-Length': '0'} 2022-09-13T19:12:11.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.061 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.062 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.063 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.063 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:17:40 GMT 2022-09-13T19:12:11.064 INFO:tasks.rgw_multisite_tests:/yxvvuh-58/ 2022-09-13T19:12:11.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.065 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ= 2022-09-13T19:12:11.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:17:40 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:mj4M5i0S+AgCsJr8j2iNWqGMmUQ=', 'Content-Length': '0'} 2022-09-13T19:12:11.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.066 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:11.067 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.067 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:11.068 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_multipart_object_sync 2022-09-13T19:12:11.068 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:11.069 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:11.069 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:11.070 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:11.070 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1226, in test_multipart_object_sync 2022-09-13T19:12:11.071 INFO:tasks.rgw_multisite_tests: buckets, zone_bucket = create_bucket_per_zone(zonegroup_conns) 2022-09-13T19:12:11.071 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 523, in create_bucket_per_zone 2022-09-13T19:12:11.073 INFO:tasks.rgw_multisite_tests: bucket = zone.create_bucket(bucket_name) 2022-09-13T19:12:11.074 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/rgw_multi/zone_rados.py", line 55, in create_bucket 2022-09-13T19:12:11.074 INFO:tasks.rgw_multisite_tests: return self.conn.create_bucket(name) 2022-09-13T19:12:11.075 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 619, in create_bucket 2022-09-13T19:12:11.076 INFO:tasks.rgw_multisite_tests: data=data) 2022-09-13T19:12:11.076 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 671, in make_request 2022-09-13T19:12:11.077 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2022-09-13T19:12:11.077 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2022-09-13T19:12:11.078 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2022-09-13T19:12:11.078 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1030, in _mexe 2022-09-13T19:12:11.079 INFO:tasks.rgw_multisite_tests: raise ex 2022-09-13T19:12:11.079 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 943, in _mexe 2022-09-13T19:12:11.080 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2022-09-13T19:12:11.081 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1285, in request 2022-09-13T19:12:11.082 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2022-09-13T19:12:11.082 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1331, in _send_request 2022-09-13T19:12:11.083 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2022-09-13T19:12:11.084 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1280, in endheaders 2022-09-13T19:12:11.085 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2022-09-13T19:12:11.086 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1046, in _send_output 2022-09-13T19:12:11.087 INFO:tasks.rgw_multisite_tests: self.send(msg) 2022-09-13T19:12:11.087 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 984, in send 2022-09-13T19:12:11.088 INFO:tasks.rgw_multisite_tests: self.connect() 2022-09-13T19:12:11.089 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 956, in connect 2022-09-13T19:12:11.090 INFO:tasks.rgw_multisite_tests: (self.host,self.port), self.timeout, self.source_address) 2022-09-13T19:12:11.091 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/gevent/socket.py", line 101, in create_connection 2022-09-13T19:12:11.092 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2022-09-13T19:12:11.092 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/gevent/_socketcommon.py", line 607, in connect 2022-09-13T19:12:11.093 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2022-09-13T19:12:11.093 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2022-09-13T19:12:11.094 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:11.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=yxvvuh-59 2022-09-13T19:12:11.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-59/ 2022-09-13T19:12:11.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-59/ 2022-09-13T19:12:11.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:11.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-59/ 2022-09-13T19:12:11.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.101 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.101 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.101 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.102 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:19:08 GMT 2022-09-13T19:12:11.102 INFO:tasks.rgw_multisite_tests:/yxvvuh-59/ 2022-09-13T19:12:11.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.104 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:nl+JKK0AEGRlWhLQroyzkYMbbEU= 2022-09-13T19:12:11.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nl+JKK0AEGRlWhLQroyzkYMbbEU=', 'Content-Length': '0'} 2022-09-13T19:12:11.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e674866abe7b18ce-0063203cfc-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:19:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=yxvvuh-60 2022-09-13T19:12:11.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-60/ 2022-09-13T19:12:11.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-60/ 2022-09-13T19:12:11.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:11.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-60/ 2022-09-13T19:12:11.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.111 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.111 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.112 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.112 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:19:08 GMT 2022-09-13T19:12:11.113 INFO:tasks.rgw_multisite_tests:/yxvvuh-60/ 2022-09-13T19:12:11.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.114 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds= 2022-09-13T19:12:11.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds=', 'Content-Length': '0'} 2022-09-13T19:12:11.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.116 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.117 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.117 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.118 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:19:08 GMT 2022-09-13T19:12:11.118 INFO:tasks.rgw_multisite_tests:/yxvvuh-60/ 2022-09-13T19:12:11.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.119 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds= 2022-09-13T19:12:11.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds=', 'Content-Length': '0'} 2022-09-13T19:12:11.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.122 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.122 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.122 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.123 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:19:08 GMT 2022-09-13T19:12:11.123 INFO:tasks.rgw_multisite_tests:/yxvvuh-60/ 2022-09-13T19:12:11.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.124 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds= 2022-09-13T19:12:11.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds=', 'Content-Length': '0'} 2022-09-13T19:12:11.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.127 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.127 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.128 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.128 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:19:08 GMT 2022-09-13T19:12:11.129 INFO:tasks.rgw_multisite_tests:/yxvvuh-60/ 2022-09-13T19:12:11.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.130 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds= 2022-09-13T19:12:11.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds=', 'Content-Length': '0'} 2022-09-13T19:12:11.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.132 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.133 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.133 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.134 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:19:08 GMT 2022-09-13T19:12:11.134 INFO:tasks.rgw_multisite_tests:/yxvvuh-60/ 2022-09-13T19:12:11.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.135 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds= 2022-09-13T19:12:11.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds=', 'Content-Length': '0'} 2022-09-13T19:12:11.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.138 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.138 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.138 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.139 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:19:08 GMT 2022-09-13T19:12:11.139 INFO:tasks.rgw_multisite_tests:/yxvvuh-60/ 2022-09-13T19:12:11.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.140 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds= 2022-09-13T19:12:11.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds=', 'Content-Length': '0'} 2022-09-13T19:12:11.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.143 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.143 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.144 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.144 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:19:08 GMT 2022-09-13T19:12:11.145 INFO:tasks.rgw_multisite_tests:/yxvvuh-60/ 2022-09-13T19:12:11.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.145 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds= 2022-09-13T19:12:11.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:19:08 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Jlpt953ucaXQIflRTj2nN1rk0ds=', 'Content-Length': '0'} 2022-09-13T19:12:11.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.147 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:11.148 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.148 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:11.149 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_role_sync 2022-09-13T19:12:11.149 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:11.150 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:11.150 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:11.151 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:11.151 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1696, in test_role_sync 2022-09-13T19:12:11.152 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 510, in create_role_per_zone 2022-09-13T19:12:11.152 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/rgw_multi/zone_rados.py", line 130, in create_role 2022-09-13T19:12:11.152 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/iam/connection.py", line 1155, in create_role 2022-09-13T19:12:11.153 INFO:tasks.rgw_multisite_tests: return self.get_response('CreateRole', params) 2022-09-13T19:12:11.153 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/iam/connection.py", line 85, in get_response 2022-09-13T19:12:11.154 INFO:tasks.rgw_multisite_tests: response = self.make_request(action, params, path, verb) 2022-09-13T19:12:11.154 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1116, in make_request 2022-09-13T19:12:11.155 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request) 2022-09-13T19:12:11.155 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1030, in _mexe 2022-09-13T19:12:11.156 INFO:tasks.rgw_multisite_tests: raise ex 2022-09-13T19:12:11.156 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 943, in _mexe 2022-09-13T19:12:11.156 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2022-09-13T19:12:11.157 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1285, in request 2022-09-13T19:12:11.157 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2022-09-13T19:12:11.158 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1331, in _send_request 2022-09-13T19:12:11.158 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2022-09-13T19:12:11.159 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1280, in endheaders 2022-09-13T19:12:11.159 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2022-09-13T19:12:11.160 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 1046, in _send_output 2022-09-13T19:12:11.160 INFO:tasks.rgw_multisite_tests: self.send(msg) 2022-09-13T19:12:11.160 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 984, in send 2022-09-13T19:12:11.161 INFO:tasks.rgw_multisite_tests: self.connect() 2022-09-13T19:12:11.161 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.6/http/client.py", line 956, in connect 2022-09-13T19:12:11.162 INFO:tasks.rgw_multisite_tests: (self.host,self.port), self.timeout, self.source_address) 2022-09-13T19:12:11.162 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/gevent/socket.py", line 101, in create_connection 2022-09-13T19:12:11.163 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2022-09-13T19:12:11.163 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/gevent/_socketcommon.py", line 607, in connect 2022-09-13T19:12:11.164 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2022-09-13T19:12:11.164 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2022-09-13T19:12:11.164 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:11.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create role zone=test-zone1 name=roles-yxvvuh-1 2022-09-13T19:12:11.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2022-09-13T19:12:11.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: / 2022-09-13T19:12:11.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com 2022-09-13T19:12:11.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {'RoleName': 'roles-yxvvuh-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2022-09-13T19:12:11.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2022-09-13T19:12:11.170 INFO:tasks.rgw_multisite_tests:POST 2022-09-13T19:12:11.170 INFO:tasks.rgw_multisite_tests:/ 2022-09-13T19:12:11.171 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.171 INFO:tasks.rgw_multisite_tests:host:smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.172 INFO:tasks.rgw_multisite_tests:x-amz-date:20220913T150048Z 2022-09-13T19:12:11.172 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.173 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2022-09-13T19:12:11.173 INFO:tasks.rgw_multisite_tests:182fd3870aca11fdeeaf60d0f516606f850cd90dee9c18712b959899cb186515 2022-09-13T19:12:11.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.174 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2022-09-13T19:12:11.175 INFO:tasks.rgw_multisite_tests:20220913T150048Z 2022-09-13T19:12:11.175 INFO:tasks.rgw_multisite_tests:20220913/front/smithi036/aws4_request 2022-09-13T19:12:11.175 INFO:tasks.rgw_multisite_tests:6a665b1d9881145a9e2573eb5d7f8c7c6c1bca3265f639221fd0aa604fe78d03 2022-09-13T19:12:11.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.176 INFO:tasks.rgw_multisite_tests:b115ef567229cc27b92e48e6241ecda26eb9d808086dfdeb62f669050d5060f0 2022-09-13T19:12:11.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150048Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi036/aws4_request,SignedHeaders=host;x-amz-date,Signature=b115ef567229cc27b92e48e6241ecda26eb9d808086dfdeb62f669050d5060f0', 'Host': 'smithi036.front.sepia.ceph.com:8000'} 2022-09-13T19:12:11.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000a32d5bf7612eaef6-0063209b20-1452-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '731'), ('Date', 'Tue, 13 Sep 2022 15:00:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'417d2fc0-e214-46af-8c1e-90aa63c986e4roles-yxvvuh-1/arn:aws:iam:::role/roles-yxvvuh-12022-09-13T15:00:48.859Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000a32d5bf7612eaef6-0063209b20-1452-test-zone1' 2022-09-13T19:12:11.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create role zone=test-zone2 name=roles-yxvvuh-2 2022-09-13T19:12:11.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2022-09-13T19:12:11.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: / 2022-09-13T19:12:11.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com 2022-09-13T19:12:11.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {'RoleName': 'roles-yxvvuh-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2022-09-13T19:12:11.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2022-09-13T19:12:11.183 INFO:tasks.rgw_multisite_tests:POST 2022-09-13T19:12:11.183 INFO:tasks.rgw_multisite_tests:/ 2022-09-13T19:12:11.184 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.184 INFO:tasks.rgw_multisite_tests:host:smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.185 INFO:tasks.rgw_multisite_tests:x-amz-date:20220913T150048Z 2022-09-13T19:12:11.185 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.186 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2022-09-13T19:12:11.186 INFO:tasks.rgw_multisite_tests:0ac290b3ddeda42772c78a2b5e46b0854cedd10a8dd42ffefe9fc0d46fcbcfa1 2022-09-13T19:12:11.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.187 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2022-09-13T19:12:11.187 INFO:tasks.rgw_multisite_tests:20220913T150048Z 2022-09-13T19:12:11.188 INFO:tasks.rgw_multisite_tests:20220913/front/smithi083/aws4_request 2022-09-13T19:12:11.188 INFO:tasks.rgw_multisite_tests:519cb6647ef7cd4b9862dfba68ac2b1ee4c57c590356602e42c701384f2d651c 2022-09-13T19:12:11.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.189 INFO:tasks.rgw_multisite_tests:bdc91ea2ec0e32a5787b11360264ffcccb87bd8310134fba29acd76b70bc25ac 2022-09-13T19:12:11.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150048Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi083/aws4_request,SignedHeaders=host;x-amz-date,Signature=bdc91ea2ec0e32a5787b11360264ffcccb87bd8310134fba29acd76b70bc25ac', 'Host': 'smithi083.front.sepia.ceph.com:8000'} 2022-09-13T19:12:11.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2022-09-13T19:12:11.192 INFO:tasks.rgw_multisite_tests:POST 2022-09-13T19:12:11.192 INFO:tasks.rgw_multisite_tests:/ 2022-09-13T19:12:11.193 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.193 INFO:tasks.rgw_multisite_tests:host:smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.194 INFO:tasks.rgw_multisite_tests:x-amz-date:20220913T150049Z 2022-09-13T19:12:11.194 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.194 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2022-09-13T19:12:11.195 INFO:tasks.rgw_multisite_tests:0ac290b3ddeda42772c78a2b5e46b0854cedd10a8dd42ffefe9fc0d46fcbcfa1 2022-09-13T19:12:11.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.196 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2022-09-13T19:12:11.196 INFO:tasks.rgw_multisite_tests:20220913T150049Z 2022-09-13T19:12:11.197 INFO:tasks.rgw_multisite_tests:20220913/front/smithi083/aws4_request 2022-09-13T19:12:11.198 INFO:tasks.rgw_multisite_tests:423774d31646d183af46cdbcbd6f450d4a47b0ac0f1c47e051ae23fb4ead72eb 2022-09-13T19:12:11.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.199 INFO:tasks.rgw_multisite_tests:aa8ec3b2d90d7ff0a91f1ad46ea6c66b794d3587f2d79730a94a3b7d34351a8a 2022-09-13T19:12:11.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150049Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi083/aws4_request,SignedHeaders=host;x-amz-date,Signature=aa8ec3b2d90d7ff0a91f1ad46ea6c66b794d3587f2d79730a94a3b7d34351a8a', 'Host': 'smithi083.front.sepia.ceph.com:8000'} 2022-09-13T19:12:11.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2022-09-13T19:12:11.201 INFO:tasks.rgw_multisite_tests:POST 2022-09-13T19:12:11.202 INFO:tasks.rgw_multisite_tests:/ 2022-09-13T19:12:11.202 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.203 INFO:tasks.rgw_multisite_tests:host:smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.203 INFO:tasks.rgw_multisite_tests:x-amz-date:20220913T150050Z 2022-09-13T19:12:11.204 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.205 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2022-09-13T19:12:11.205 INFO:tasks.rgw_multisite_tests:0ac290b3ddeda42772c78a2b5e46b0854cedd10a8dd42ffefe9fc0d46fcbcfa1 2022-09-13T19:12:11.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.206 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2022-09-13T19:12:11.207 INFO:tasks.rgw_multisite_tests:20220913T150050Z 2022-09-13T19:12:11.207 INFO:tasks.rgw_multisite_tests:20220913/front/smithi083/aws4_request 2022-09-13T19:12:11.208 INFO:tasks.rgw_multisite_tests:5621475449910c5b0fd59decfd542b4015175c32f52186241c3ee89f5e4796e2 2022-09-13T19:12:11.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.208 INFO:tasks.rgw_multisite_tests:5d7474e3e0a90693e75bb0901c80d86d54352a8c62a89cfd820d0ba1daecce3d 2022-09-13T19:12:11.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150050Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi083/aws4_request,SignedHeaders=host;x-amz-date,Signature=5d7474e3e0a90693e75bb0901c80d86d54352a8c62a89cfd820d0ba1daecce3d', 'Host': 'smithi083.front.sepia.ceph.com:8000'} 2022-09-13T19:12:11.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2022-09-13T19:12:11.211 INFO:tasks.rgw_multisite_tests:POST 2022-09-13T19:12:11.212 INFO:tasks.rgw_multisite_tests:/ 2022-09-13T19:12:11.212 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.212 INFO:tasks.rgw_multisite_tests:host:smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.213 INFO:tasks.rgw_multisite_tests:x-amz-date:20220913T150052Z 2022-09-13T19:12:11.213 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.214 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2022-09-13T19:12:11.214 INFO:tasks.rgw_multisite_tests:0ac290b3ddeda42772c78a2b5e46b0854cedd10a8dd42ffefe9fc0d46fcbcfa1 2022-09-13T19:12:11.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.215 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2022-09-13T19:12:11.215 INFO:tasks.rgw_multisite_tests:20220913T150052Z 2022-09-13T19:12:11.216 INFO:tasks.rgw_multisite_tests:20220913/front/smithi083/aws4_request 2022-09-13T19:12:11.216 INFO:tasks.rgw_multisite_tests:fafc66dc6cf5f1ceb71d562e70817708451517ebedadd68f94ab312809f71aed 2022-09-13T19:12:11.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.217 INFO:tasks.rgw_multisite_tests:2f67f5a102a06fc6783ec35acc03f00be9df1a095cb4f4e578c58b385d702e25 2022-09-13T19:12:11.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150052Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi083/aws4_request,SignedHeaders=host;x-amz-date,Signature=2f67f5a102a06fc6783ec35acc03f00be9df1a095cb4f4e578c58b385d702e25', 'Host': 'smithi083.front.sepia.ceph.com:8000'} 2022-09-13T19:12:11.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2022-09-13T19:12:11.220 INFO:tasks.rgw_multisite_tests:POST 2022-09-13T19:12:11.220 INFO:tasks.rgw_multisite_tests:/ 2022-09-13T19:12:11.221 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.221 INFO:tasks.rgw_multisite_tests:host:smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.222 INFO:tasks.rgw_multisite_tests:x-amz-date:20220913T150053Z 2022-09-13T19:12:11.222 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.223 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2022-09-13T19:12:11.223 INFO:tasks.rgw_multisite_tests:0ac290b3ddeda42772c78a2b5e46b0854cedd10a8dd42ffefe9fc0d46fcbcfa1 2022-09-13T19:12:11.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.224 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2022-09-13T19:12:11.224 INFO:tasks.rgw_multisite_tests:20220913T150053Z 2022-09-13T19:12:11.225 INFO:tasks.rgw_multisite_tests:20220913/front/smithi083/aws4_request 2022-09-13T19:12:11.225 INFO:tasks.rgw_multisite_tests:0f6c44b0c23a1e092ebef6d9beeb891f66b49b74ce509b64ff17bdbdce3831ab 2022-09-13T19:12:11.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.226 INFO:tasks.rgw_multisite_tests:550f517afdf231f748ff1fb29af8dadd1c063d3c43b1f2529be3947bb78868f9 2022-09-13T19:12:11.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150053Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi083/aws4_request,SignedHeaders=host;x-amz-date,Signature=550f517afdf231f748ff1fb29af8dadd1c063d3c43b1f2529be3947bb78868f9', 'Host': 'smithi083.front.sepia.ceph.com:8000'} 2022-09-13T19:12:11.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2022-09-13T19:12:11.229 INFO:tasks.rgw_multisite_tests:POST 2022-09-13T19:12:11.229 INFO:tasks.rgw_multisite_tests:/ 2022-09-13T19:12:11.230 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.230 INFO:tasks.rgw_multisite_tests:host:smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.231 INFO:tasks.rgw_multisite_tests:x-amz-date:20220913T150106Z 2022-09-13T19:12:11.231 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.232 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2022-09-13T19:12:11.232 INFO:tasks.rgw_multisite_tests:0ac290b3ddeda42772c78a2b5e46b0854cedd10a8dd42ffefe9fc0d46fcbcfa1 2022-09-13T19:12:11.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.233 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2022-09-13T19:12:11.233 INFO:tasks.rgw_multisite_tests:20220913T150106Z 2022-09-13T19:12:11.234 INFO:tasks.rgw_multisite_tests:20220913/front/smithi083/aws4_request 2022-09-13T19:12:11.234 INFO:tasks.rgw_multisite_tests:4a177d9abc7fd81e9344faac4d181e6bcb05893f098af4557a534b99a1ca9aaf 2022-09-13T19:12:11.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.235 INFO:tasks.rgw_multisite_tests:0942373b6f91725b3f1c392ffdf93b6698f7044aa9f8db0bb101f098df7d9436 2022-09-13T19:12:11.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150106Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi083/aws4_request,SignedHeaders=host;x-amz-date,Signature=0942373b6f91725b3f1c392ffdf93b6698f7044aa9f8db0bb101f098df7d9436', 'Host': 'smithi083.front.sepia.ceph.com:8000'} 2022-09-13T19:12:11.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: CanonicalRequest: 2022-09-13T19:12:11.238 INFO:tasks.rgw_multisite_tests:POST 2022-09-13T19:12:11.238 INFO:tasks.rgw_multisite_tests:/ 2022-09-13T19:12:11.239 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.239 INFO:tasks.rgw_multisite_tests:host:smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.240 INFO:tasks.rgw_multisite_tests:x-amz-date:20220913T150125Z 2022-09-13T19:12:11.240 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.240 INFO:tasks.rgw_multisite_tests:host;x-amz-date 2022-09-13T19:12:11.241 INFO:tasks.rgw_multisite_tests:0ac290b3ddeda42772c78a2b5e46b0854cedd10a8dd42ffefe9fc0d46fcbcfa1 2022-09-13T19:12:11.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.242 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2022-09-13T19:12:11.242 INFO:tasks.rgw_multisite_tests:20220913T150125Z 2022-09-13T19:12:11.243 INFO:tasks.rgw_multisite_tests:20220913/front/smithi083/aws4_request 2022-09-13T19:12:11.243 INFO:tasks.rgw_multisite_tests:ec3b126eb66666510ea3860a2ac85d159bfdb6a9fe6355c7d1114e1ae80fa7fe 2022-09-13T19:12:11.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.244 INFO:tasks.rgw_multisite_tests:28026bfb974d3348a071116a256f742c3bceceb5d490e989b10ba7935a96456f 2022-09-13T19:12:11.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220913T150125Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=RICCPJUVKDGV9UEGFN6Z/20220913/front/smithi083/aws4_request,SignedHeaders=host;x-amz-date,Signature=28026bfb974d3348a071116a256f742c3bceceb5d490e989b10ba7935a96456f', 'Host': 'smithi083.front.sepia.ceph.com:8000'} 2022-09-13T19:12:11.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2022-09-13T19:12:11.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:11.246 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:11.246 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.247 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:11.247 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_version_suspended_incremental_sync 2022-09-13T19:12:11.248 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:11.250 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:11.251 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:11.251 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:11.252 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 790, in test_version_suspended_incremental_sync 2022-09-13T19:12:11.252 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2022-09-13T19:12:11.253 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 397, in zonegroup_bucket_checkpoint 2022-09-13T19:12:11.253 INFO:tasks.rgw_multisite_tests: target_conn.check_bucket_eq(source_conn, bucket_name) 2022-09-13T19:12:11.253 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/rgw_multi/zone_rados.py", line 93, in check_bucket_eq 2022-09-13T19:12:11.254 INFO:tasks.rgw_multisite_tests: check_object_eq(k1_head, k2_head, False) 2022-09-13T19:12:11.254 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/rgw_multi/zone_rados.py", line 19, in check_object_eq 2022-09-13T19:12:11.255 INFO:tasks.rgw_multisite_tests: eq(k1.last_modified, k2.last_modified) 2022-09-13T19:12:11.255 INFO:tasks.rgw_multisite_tests:AssertionError: 'Tue, 13 Sep 2022 07:40:00 GMT' != 'Tue, 13 Sep 2022 07:41:36 GMT' 2022-09-13T19:12:11.256 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:11.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:11.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:11.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.261 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.262 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.262 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.263 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:39:54 GMT 2022-09-13T19:12:11.263 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.264 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:tvFUVCE7KmkdvadwImSgcVFD5Tc= 2022-09-13T19:12:11.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:39:54 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tvFUVCE7KmkdvadwImSgcVFD5Tc=', 'Content-Length': '0'} 2022-09-13T19:12:11.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000005cb1bb79be5104f1-00632033ca-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=yxvvuh-20 2022-09-13T19:12:11.266 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:11.266 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:11.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:11.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:11.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:11.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:11.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:11.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:11.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054794.798856_147.1", 2022-09-13T19:12:11.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:39:54.798856Z" 2022-09-13T19:12:11.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:11.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:11.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:11.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:11.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:11.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:11.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054794.798856_147.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:11.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:11.278 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:11.278 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:11.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:11.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:11.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:11.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:39:55.011+0000 7fa91f5e5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:11.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:39:55.012+0000 7fa91f5e5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:11.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:11.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:11.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:11.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:11.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:11.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:11.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:11.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:11.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:11.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:11.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:11.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:11.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:11.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:11.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:11.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:11.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:11.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:11.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:11.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:11.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:11.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:11.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:11.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:11.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:11.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:11.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:11.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:11.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:11.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:11.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:11.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:11.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:11.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:11.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:11.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054794.791007_142.1", 2022-09-13T19:12:11.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:11.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:11.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:11.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:39:54.791007Z", 2022-09-13T19:12:11.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:11.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:11.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:11.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:11.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:11.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:11.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:11.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:11.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:11.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:11.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:11.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:11.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:11.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:11.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:11.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:11.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:11.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:11.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:11.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:39:55Z" 2022-09-13T19:12:11.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:11.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:11.312 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:11.313 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:11.313 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:11.314 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:11.314 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:11.314 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:11.315 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.315 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:11.316 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:11.316 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:11.317 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:11.317 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:11.318 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:11.318 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:11.318 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:11.319 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:11.319 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:11.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:11.320 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:11.321 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.321 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:11.322 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:11.322 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:11.323 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:11.323 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:11.323 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:11.324 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:11.324 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:11.326 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:11.326 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:11.327 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:11.327 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.327 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:11.331 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:11.332 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:11.333 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:11.334 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054794.791007_142.1", 2022-09-13T19:12:11.334 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:11.335 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:11.336 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:11.337 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:39:54.791007Z", 2022-09-13T19:12:11.338 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:11.338 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:11.342 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.343 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:11.343 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:11.344 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:11.344 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:11.344 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:11.345 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:11.345 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:11.346 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:11.346 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:11.347 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:11.347 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:11.348 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:11.348 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:11.348 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.349 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:11.349 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:11.350 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:11.350 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.351 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:39:55Z" 2022-09-13T19:12:11.351 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:11.352 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054794.791007_142.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:39:54.791007Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:11.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054794.798856_147.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:11.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054794.791007_142.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:11.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663054794.798856_147.1 target=1_1663054794.791007_142.1 2022-09-13T19:12:11.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:11.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:11.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:11.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:11.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:11.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:00.172+0000 7f59cdc93600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:11.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:00.174+0000 7f59cdc93600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:11.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:11.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:11.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:11.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:11.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:11.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:11.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:11.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:11.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:11.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:11.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:11.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:11.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:11.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:11.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:11.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:11.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:11.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:11.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:11.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:11.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:11.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:11.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:11.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:11.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:11.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:11.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:11.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:11.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:11.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:11.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:11.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:11.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054794.798856_147.1", 2022-09-13T19:12:11.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:11.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:11.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:11.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:39:54.798856Z", 2022-09-13T19:12:11.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:11.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:11.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:11.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:11.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:11.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:11.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:11.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:11.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:11.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:11.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:11.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:11.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:11.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:11.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:11.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:11.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:11.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:11.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:11.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:40:00Z" 2022-09-13T19:12:11.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:11.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:11.388 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:11.388 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:11.388 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:11.389 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:11.389 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:11.390 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:11.390 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.391 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:11.391 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:11.392 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:11.392 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:11.393 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:11.393 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:11.394 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:11.394 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:11.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:11.395 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:11.395 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:11.396 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:11.396 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.397 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:11.397 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:11.398 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:11.398 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:11.398 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:11.399 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:11.399 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:11.400 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:11.400 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:11.401 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:11.401 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:11.402 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.402 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:11.402 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:11.403 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:11.403 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:11.404 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054794.798856_147.1", 2022-09-13T19:12:11.404 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:11.405 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:11.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:11.405 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:39:54.798856Z", 2022-09-13T19:12:11.406 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:11.406 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:11.407 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.407 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:11.408 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:11.408 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:11.409 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:11.409 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:11.410 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:11.410 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:11.410 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:11.411 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:11.411 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:11.412 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:11.412 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:11.413 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:11.413 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.413 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:11.414 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:11.414 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:11.415 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.415 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:40:00Z" 2022-09-13T19:12:11.416 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:11.416 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054794.798856_147.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:39:54.798856Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:11.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054794.798856_147.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:11.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054794.798856_147.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:11.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2022-09-13T19:12:11.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2022-09-13T19:12:11.419 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:11.419 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:11.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:11.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:11.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:11.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:00.335+0000 7fe067b39600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:11.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:00.336+0000 7fe067b39600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:11.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:11.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "sync_status": { 2022-09-13T19:12:11.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "info": { 2022-09-13T19:12:11.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "status": "sync", 2022-09-13T19:12:11.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:11.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "markers": [ 2022-09-13T19:12:11.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:11.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 0, 2022-09-13T19:12:11.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:11.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:11.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:11.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:11.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:11.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:11.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:11.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:11.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:11.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 1, 2022-09-13T19:12:11.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:11.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:11.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:11.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:11.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:11.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 2, 2022-09-13T19:12:11.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:11.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:11.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054794.798856_147.1", 2022-09-13T19:12:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:11.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:11.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "2022-09-13T07:39:54.798856Z", 2022-09-13T19:12:11.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:11.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:11.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:11.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:11.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 3, 2022-09-13T19:12:11.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:11.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:11.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:11.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "full_sync": { 2022-09-13T19:12:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total": 1, 2022-09-13T19:12:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "complete": 1 2022-09-13T19:12:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:11.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "current_time": "2022-09-13T07:40:00Z" 2022-09-13T19:12:11.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:11.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:11.453 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:11.453 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:11.453 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:11.454 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:11.454 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:11.455 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:11.455 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.456 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:11.456 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:11.457 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:11.457 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:11.457 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:11.458 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:11.458 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:11.459 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:11.459 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:11.460 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:11.460 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:11.461 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:11.461 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.461 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:11.462 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:11.462 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:11.463 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:11.463 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:11.464 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:11.464 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:11.465 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:11.465 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:11.466 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:11.466 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:11.467 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.467 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:11.467 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:11.468 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:11.468 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:11.469 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054794.798856_147.1", 2022-09-13T19:12:11.469 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:11.470 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:11.470 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:11.471 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:39:54.798856Z", 2022-09-13T19:12:11.471 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:11.471 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:11.472 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.472 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:11.473 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:11.473 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:11.474 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:11.474 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:11.475 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:11.475 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:11.476 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:11.476 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:11.477 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:11.477 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:11.478 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:11.478 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:11.479 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.479 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:11.480 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:11.480 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:11.480 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:11.481 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:40:00Z" 2022-09-13T19:12:11.481 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:11.482 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054794.798856_147.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:39:54.798856Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:11.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054794.798856_147.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:11.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054794.798856_147.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:11.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2022-09-13T19:12:11.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:11.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:11.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.490 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:11.490 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.490 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.491 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:00 GMT 2022-09-13T19:12:11.491 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.492 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:g6KCc9dFIwzxZw1yXQ9N1H+WNpM= 2022-09-13T19:12:11.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:00 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:g6KCc9dFIwzxZw1yXQ9N1H+WNpM=', 'Content-Length': '0'} 2022-09-13T19:12:11.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf8d066aa90fbfaa-00632033d0-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:11.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:11.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T19:12:11.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.499 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:11.499 INFO:tasks.rgw_multisite_tests:1B2M2Y8AsgTpgAmY7PhCfg== 2022-09-13T19:12:11.499 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:11.500 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:00 GMT 2022-09-13T19:12:11.500 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:11.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.501 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:yZDtonOKDJwfZcjbjhPDYzFtG8A= 2022-09-13T19:12:11.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:40:00 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:yZDtonOKDJwfZcjbjhPDYzFtG8A='} 2022-09-13T19:12:11.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f834321274da949-00632033d0-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created initial version id=None 2022-09-13T19:12:11.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-20 2022-09-13T19:12:11.504 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:11.504 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:11.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:11.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:00.563+0000 7fdd41a49600 1 waiting to reach incremental sync.. 2022-09-13T19:12:11.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:05.564+0000 7fdd41a49600 1 waiting to reach incremental sync.. 2022-09-13T19:12:11.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:10.565+0000 7fdd41a49600 1 bucket sync caught up with source: 2022-09-13T19:12:11.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T19:12:11.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T19:12:11.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:10.565+0000 7fdd41a49600 0 bucket checkpoint complete 2022-09-13T19:12:11.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-20 2022-09-13T19:12:11.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:11.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:11.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:11.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:11.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:11.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:10.695+0000 7f102b7de600 1 waiting to reach incremental sync.. 2022-09-13T19:12:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:15.697+0000 7f102b7de600 1 bucket sync caught up with source: 2022-09-13T19:12:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T19:12:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T19:12:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:15.697+0000 7f102b7de600 0 bucket checkpoint complete 2022-09-13T19:12:11.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-20 2022-09-13T19:12:11.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:11.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:11.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:11.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:15.831+0000 7f74bdc9d600 1 waiting to reach incremental sync.. 2022-09-13T19:12:11.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:20.832+0000 7f74bdc9d600 1 waiting to reach incremental sync.. 2022-09-13T19:12:11.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:25.832+0000 7f74bdc9d600 1 waiting to reach incremental sync.. 2022-09-13T19:12:11.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:30.834+0000 7f74bdc9d600 1 bucket sync caught up with source: 2022-09-13T19:12:11.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T19:12:11.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T19:12:11.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:30.834+0000 7f74bdc9d600 0 bucket checkpoint complete 2022-09-13T19:12:11.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-20 2022-09-13T19:12:11.520 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:11.520 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:11.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:11.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:11.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:11.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:30.971+0000 7fcf9662a600 1 bucket sync caught up with source: 2022-09-13T19:12:11.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T19:12:11.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T19:12:11.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:30.971+0000 7fcf9662a600 0 bucket checkpoint complete 2022-09-13T19:12:11.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-20 2022-09-13T19:12:11.524 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:11.525 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:31.081+0000 7f90ff6bd600 1 bucket sync caught up with source: 2022-09-13T19:12:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000002.64.5, , ] 2022-09-13T19:12:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000002.64.5, , ] 2022-09-13T19:12:11.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:31.081+0000 7f90ff6bd600 0 bucket checkpoint complete 2022-09-13T19:12:11.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-20 2022-09-13T19:12:11.529 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:11.530 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:11.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:11.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:11.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:11.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:31.220+0000 7fea0246e600 1 bucket sync caught up with source: 2022-09-13T19:12:11.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000002.64.5, , ] 2022-09-13T19:12:11.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000002.64.5, , ] 2022-09-13T19:12:11.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:31.220+0000 7fea0246e600 0 bucket checkpoint complete 2022-09-13T19:12:11.534 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=yxvvuh-20 zones={test-zone2, test-zone1} 2022-09-13T19:12:11.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:11.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:11.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.540 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:11.540 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.541 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.541 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.542 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.543 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U= 2022-09-13T19:12:11.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U=', 'Content-Length': '0'} 2022-09-13T19:12:11.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc1446275dc5dfbd-00632033ef-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:11.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:11.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.550 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:11.551 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.551 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.552 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.552 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.553 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U= 2022-09-13T19:12:11.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U=', 'Content-Length': '0'} 2022-09-13T19:12:11.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002271181554045bb1-00632033ef-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.555 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2022-09-13T19:12:11.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.562 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.563 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.563 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.564 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.564 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:11.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.565 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T19:12:11.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T19:12:11.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e987d43602f27a3-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.567 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:11.568 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2022-09-13T19:12:11.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.575 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.575 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.575 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.576 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.576 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:11.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.577 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T19:12:11.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T19:12:11.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000474bd447e98d848a-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.579 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:11.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.585 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.586 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.586 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.587 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.587 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:11.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.588 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T19:12:11.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T19:12:11.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac9687f56b18abaf-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.596 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.596 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.596 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.597 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.597 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:11.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.598 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T19:12:11.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T19:12:11.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095d0a18410f7a15e-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.600 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:11.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.606 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.607 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.607 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.607 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.608 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.609 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T19:12:11.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T19:12:11.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000071bab7a36bf4834-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.616 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.616 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.617 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.617 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.618 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.618 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T19:12:11.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T19:12:11.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040d38cddfe3d9b43-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:11.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.626 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:11.626 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.626 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.627 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.627 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.628 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ= 2022-09-13T19:12:11.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ=', 'Content-Length': '0'} 2022-09-13T19:12:11.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e69f3db726a03a97-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:11.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.635 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:11.636 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.636 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.637 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.637 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.638 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ= 2022-09-13T19:12:11.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ=', 'Content-Length': '0'} 2022-09-13T19:12:11.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000061a62398be3efffc-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.640 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:11.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.646 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.646 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.647 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.647 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.648 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.648 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T19:12:11.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T19:12:11.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003b60881f0f8ff629-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.656 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.656 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.656 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.657 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.657 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.658 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T19:12:11.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T19:12:11.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000921a22419b17ccd6-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.665 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.666 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.666 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.667 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.667 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.668 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T19:12:11.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T19:12:11.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009686484e39d37822-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.676 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.676 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.677 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.677 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.677 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.678 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T19:12:11.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T19:12:11.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b29448057fce256-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:11.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:11.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.685 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:11.685 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.686 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.686 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.687 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:11.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.688 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA= 2022-09-13T19:12:11.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA=', 'Content-Length': '0'} 2022-09-13T19:12:11.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000024a44c312a7bc20b-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:11.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:11.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.694 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:11.694 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.695 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.695 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.695 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:11.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.696 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA= 2022-09-13T19:12:11.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA=', 'Content-Length': '0'} 2022-09-13T19:12:11.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005ead47aa89c4a024-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.698 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:11.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:11.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.703 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.704 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.704 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.704 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.705 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:11.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.706 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y= 2022-09-13T19:12:11.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y=', 'Content-Length': '0'} 2022-09-13T19:12:11.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000056e4fdd3cc163b02-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:11.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.712 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.713 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.713 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.713 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.714 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:11.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.715 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y= 2022-09-13T19:12:11.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y=', 'Content-Length': '0'} 2022-09-13T19:12:11.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080c93ddb34d541f2-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.722 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.722 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.723 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.723 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.723 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.724 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T19:12:11.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T19:12:11.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f12184f1dadd221-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.732 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.732 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.733 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.733 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.734 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.734 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T19:12:11.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T19:12:11.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c350c2cc490a0bcc-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.736 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=yxvvuh-20 zones={test-zone2, test-zone1} 2022-09-13T19:12:11.737 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=yxvvuh-20 zones={test-zone3, test-zone1} 2022-09-13T19:12:11.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:11.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:11.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:11.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:11.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.742 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:11.743 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.743 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.743 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.744 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.745 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U= 2022-09-13T19:12:11.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U=', 'Content-Length': '0'} 2022-09-13T19:12:11.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d54da016e15f76ee-00632033ef-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:11.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:11.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.751 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:11.751 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.752 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.752 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.753 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.753 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U= 2022-09-13T19:12:11.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U=', 'Content-Length': '0'} 2022-09-13T19:12:11.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e2f2a917c6b753a-00632033ef-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.755 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2022-09-13T19:12:11.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:11.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:11.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.761 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.761 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.762 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.762 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.763 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:11.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.764 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T19:12:11.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T19:12:11.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082e6eb060106538c-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.765 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:11.766 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2022-09-13T19:12:11.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.772 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.772 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.773 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.773 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.774 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:11.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.775 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T19:12:11.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T19:12:11.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9e1fab51ea0c8f2-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.776 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:11.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:11.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:11.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.783 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.783 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.784 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.784 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.784 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:11.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.785 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T19:12:11.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T19:12:11.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acc03080c8628507-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.793 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.793 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.794 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.794 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.794 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:11.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.795 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T19:12:11.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T19:12:11.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e66b7f0e9d1ae42d-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.797 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:11.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:11.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:11.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.803 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.804 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.804 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.804 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.805 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.806 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T19:12:11.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T19:12:11.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003e9393209af32e06-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.813 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.813 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.814 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.814 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.815 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.815 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T19:12:11.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T19:12:11.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bdbfcbad7ead7879-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:11.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:11.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:11.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.822 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:11.823 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.823 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.824 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.824 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.825 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ= 2022-09-13T19:12:11.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ=', 'Content-Length': '0'} 2022-09-13T19:12:11.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036ed0c35adc5e53f-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:11.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.832 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:11.832 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.833 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.833 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.834 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.835 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ= 2022-09-13T19:12:11.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ=', 'Content-Length': '0'} 2022-09-13T19:12:11.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000806aac0fb4e3fa3e-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.836 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:11.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:11.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:11.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.842 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.843 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.843 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.844 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.844 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.845 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T19:12:11.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T19:12:11.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d1445664761efcb-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.852 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.852 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.853 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.853 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.854 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.855 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T19:12:11.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T19:12:11.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fe915d4ff8c8c061-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:11.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:11.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.862 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.862 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.862 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.863 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.863 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.864 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T19:12:11.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T19:12:11.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000659290c8dc7623e2-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.872 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.872 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.873 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.873 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.874 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.874 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T19:12:11.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T19:12:11.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f7279c1ab67e72c-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:11.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:11.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:11.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:11.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.881 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:11.881 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.882 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.882 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.883 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:11.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.884 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA= 2022-09-13T19:12:11.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA=', 'Content-Length': '0'} 2022-09-13T19:12:11.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000007996de83597c681-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:11.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:11.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.890 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:11.891 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.891 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.892 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.892 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:11.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.893 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA= 2022-09-13T19:12:11.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:snEkX+Of6wemfAWgzrOtsCqCyYA=', 'Content-Length': '0'} 2022-09-13T19:12:11.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000063936968378e5e71-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.894 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:11.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:11.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:11.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:11.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.900 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.900 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.901 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.901 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.902 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:11.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.903 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y= 2022-09-13T19:12:11.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y=', 'Content-Length': '0'} 2022-09-13T19:12:11.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa267b0a0019819d-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:11.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.909 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.909 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.910 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.910 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.910 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:11.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.911 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y= 2022-09-13T19:12:11.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RoOxG9iILnX609H7cOg+raHX+9Y=', 'Content-Length': '0'} 2022-09-13T19:12:11.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000015dee212e01d6a93-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:11.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:11.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.918 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.919 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.919 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.920 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.920 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.921 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T19:12:11.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T19:12:11.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080d92b896010004e-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:11.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:11.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.929 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.929 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.929 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.930 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.930 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.931 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T19:12:11.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T19:12:11.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030be43d9f3b8e04f-00632033ef-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.934 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=yxvvuh-20 zones={test-zone3, test-zone1} 2022-09-13T19:12:11.934 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=yxvvuh-20 zones={test-zone3, test-zone2} 2022-09-13T19:12:11.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:11.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:11.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:11.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:11.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.940 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:11.940 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.940 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.941 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.941 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.942 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U= 2022-09-13T19:12:11.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U=', 'Content-Length': '0'} 2022-09-13T19:12:11.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c500a12444a75af7-00632033ef-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:11.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:11.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.948 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:11.949 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.949 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.950 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.950 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:11.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.951 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U= 2022-09-13T19:12:11.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:HCNu2atMQ3Tfw11ThVF24HCQm3U=', 'Content-Length': '0'} 2022-09-13T19:12:11.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c62352cf5999388f-00632033ef-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.952 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2022-09-13T19:12:11.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:11.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:11.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.958 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.959 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.959 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.960 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.960 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:11.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.961 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T19:12:11.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T19:12:11.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e8d1852c83c4473-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.963 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:11.963 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2022-09-13T19:12:11.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.969 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.970 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.970 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.971 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.971 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:11.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.972 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T19:12:11.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T19:12:11.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d77de341b5ce19e-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.974 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:11.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:11.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:11.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.980 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.981 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.981 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.981 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.982 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:11.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.983 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T19:12:11.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T19:12:11.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca3e91577a7929bb-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:11.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:11.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:11.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:11.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:11.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:11.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:11.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:11.990 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:11.991 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.991 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:11.992 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:11.992 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:11.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:11.993 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40= 2022-09-13T19:12:11.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nLXAmhO+GjQLwLazGYEs56oni40=', 'Content-Length': '0'} 2022-09-13T19:12:11.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddcf14121524b8db-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:11.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:11.995 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:11.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:11.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:11.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:11.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:11.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:11.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:11.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:11.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.001 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.001 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.002 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.002 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:12.003 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.004 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T19:12:12.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T19:12:12.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002c53d97eb8894ad4-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.011 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.011 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.011 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.012 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:12.012 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.013 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T19:12:12.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T19:12:12.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e39d90a4ec93b8a8-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.021 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.021 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.021 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.022 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:12.022 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.023 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ= 2022-09-13T19:12:12.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ=', 'Content-Length': '0'} 2022-09-13T19:12:12.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f397e9f0a6a37d04-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.031 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.031 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.032 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.032 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:12.033 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.033 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ= 2022-09-13T19:12:12.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:bjnjUn2U2zO8Uof1po94nGCMTzQ=', 'Content-Length': '0'} 2022-09-13T19:12:12.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c81f93f381d9bca6-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.035 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:12.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.041 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.042 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.042 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.043 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:12.043 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.044 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T19:12:12.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T19:12:12.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a19f36e2311e364c-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.051 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.051 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.052 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.052 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:12.053 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.054 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8= 2022-09-13T19:12:12.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:6xa6BygyuGAN64JSysKpJW9ego8=', 'Content-Length': '0'} 2022-09-13T19:12:12.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd41594c6599690d-00632033ef-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.061 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.061 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.062 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.062 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:31 GMT 2022-09-13T19:12:12.063 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.063 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk= 2022-09-13T19:12:12.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YLja5/0XO61/hvge60qIUMxIohk=', 'Content-Length': '0'} 2022-09-13T19:12:12.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022ddb3c38713a812-00632033ef-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.071 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.072 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.072 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.073 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:32 GMT 2022-09-13T19:12:12.073 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.074 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:EQR0ru4msaGkW4+U60wpbykFfiM= 2022-09-13T19:12:12.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EQR0ru4msaGkW4+U60wpbykFfiM=', 'Content-Length': '0'} 2022-09-13T19:12:12.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b58daa4279160623-00632033f0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:12.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.081 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.081 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.082 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.082 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:32 GMT 2022-09-13T19:12:12.082 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:12.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.083 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:Up1ZYsmylWYV4SF1qH9sxKuRXuY= 2022-09-13T19:12:12.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Up1ZYsmylWYV4SF1qH9sxKuRXuY=', 'Content-Length': '0'} 2022-09-13T19:12:12.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000996bee48142421b9-00632033f0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:12.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.090 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.090 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.090 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.091 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:32 GMT 2022-09-13T19:12:12.091 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:12.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.092 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:Up1ZYsmylWYV4SF1qH9sxKuRXuY= 2022-09-13T19:12:12.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Up1ZYsmylWYV4SF1qH9sxKuRXuY=', 'Content-Length': '0'} 2022-09-13T19:12:12.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000033b9cc9d076a34bc-00632033f0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.093 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:12.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:12.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.099 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.100 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.100 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.101 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:32 GMT 2022-09-13T19:12:12.101 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:12.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.102 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:VGKw4BZsknAtOX3kJAhcyMhTRhI= 2022-09-13T19:12:12.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VGKw4BZsknAtOX3kJAhcyMhTRhI=', 'Content-Length': '0'} 2022-09-13T19:12:12.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000ef96f4078087b27-00632033f0-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.108 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.108 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.109 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.109 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:32 GMT 2022-09-13T19:12:12.110 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:12.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.111 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:VGKw4BZsknAtOX3kJAhcyMhTRhI= 2022-09-13T19:12:12.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:VGKw4BZsknAtOX3kJAhcyMhTRhI=', 'Content-Length': '0'} 2022-09-13T19:12:12.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d981d4444f29d8e5-00632033f0-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.118 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.118 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.119 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.119 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:32 GMT 2022-09-13T19:12:12.120 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.120 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:EQR0ru4msaGkW4+U60wpbykFfiM= 2022-09-13T19:12:12.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EQR0ru4msaGkW4+U60wpbykFfiM=', 'Content-Length': '0'} 2022-09-13T19:12:12.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064df19905293c355-00632033f0-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.128 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.128 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.129 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.129 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:32 GMT 2022-09-13T19:12:12.130 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.130 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:EQR0ru4msaGkW4+U60wpbykFfiM= 2022-09-13T19:12:12.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:EQR0ru4msaGkW4+U60wpbykFfiM=', 'Content-Length': '0'} 2022-09-13T19:12:12.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000557c70c2e3adbfa6-00632033f0-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.132 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=yxvvuh-20 zones={test-zone3, test-zone2} 2022-09-13T19:12:12.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versioning 2022-09-13T19:12:12.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versioning 2022-09-13T19:12:12.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:12.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versioning 2022-09-13T19:12:12.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.136 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.136 INFO:tasks.rgw_multisite_tests: Enabled 2022-09-13T19:12:12.137 INFO:tasks.rgw_multisite_tests: Disabled 2022-09-13T19:12:12.137 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.140 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:12.141 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.141 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.141 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:32 GMT 2022-09-13T19:12:12.142 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versioning 2022-09-13T19:12:12.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.143 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:eAomLi6Vz1w8+lC2LUkNRTe561s= 2022-09-13T19:12:12.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eAomLi6Vz1w8+lC2LUkNRTe561s=', 'Content-Length': '228'} 2022-09-13T19:12:12.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000ecf68a4e280da9be-00632033f0-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.144 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:12.145 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:12.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:12.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:12.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:12.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:12.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:12.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:12.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:12.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:12.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:12.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:12.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:12.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:12.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:12.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:12.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054832.207579_149.1", 2022-09-13T19:12:12.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:40:32.207579Z" 2022-09-13T19:12:12.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:12.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:12.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:12.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:12.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:12.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:12.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054832.207579_149.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:12.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:12.156 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:12.156 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:12.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:12.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:12.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:12.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:32.422+0000 7f37db3b2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:12.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:32.423+0000 7f37db3b2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:12.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:12.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:12.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:12.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:12.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:12.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:12.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:12.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:12.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:12.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:12.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054832.207579_149.1", 2022-09-13T19:12:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:40:32.207579Z", 2022-09-13T19:12:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:12.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:12.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:12.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:12.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:12.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:12.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:12.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:12.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:12.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:12.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:12.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:12.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:12.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:12.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:12.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:12.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:12.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:12.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:12.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:12.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:12.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:40:32Z" 2022-09-13T19:12:12.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:12.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:12.188 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:12.189 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:12.189 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:12.190 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:12.190 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:12.191 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:12.191 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:12.191 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:12.192 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:12.192 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:12.193 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:12.193 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:12.194 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:12.194 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:12.195 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:12.195 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:12.195 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:12.196 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:12.196 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:12.197 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:12.197 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:12.198 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:12.198 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:12.198 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:12.199 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:12.199 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:12.200 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:12.200 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:12.201 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:12.201 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:12.201 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:12.202 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:12.202 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:12.203 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:12.203 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:12.204 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:12.204 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054832.207579_149.1", 2022-09-13T19:12:12.205 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:12.205 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:12.206 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:12.206 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:40:32.207579Z", 2022-09-13T19:12:12.207 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:12.207 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:12.208 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:12.208 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:12.208 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:12.209 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:12.209 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:12.210 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:12.210 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:12.211 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:12.211 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:12.212 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:12.212 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:12.212 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:12.213 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:12.213 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:12.214 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:12.214 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:12.215 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:12.215 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:12.216 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:12.216 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:40:32Z" 2022-09-13T19:12:12.217 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:12.217 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054832.207579_149.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:40:32.207579Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:12.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054832.207579_149.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:12.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054832.207579_149.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:12.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2022-09-13T19:12:12.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2022-09-13T19:12:12.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:12.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:12.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:12.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:12.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:12.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:32.562+0000 7f850d0d5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:12.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:40:32.563+0000 7f850d0d5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:12.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:12.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "sync_status": { 2022-09-13T19:12:12.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "info": { 2022-09-13T19:12:12.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "status": "sync", 2022-09-13T19:12:12.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:12.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:12.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:12.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:12.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "markers": [ 2022-09-13T19:12:12.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:12.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 0, 2022-09-13T19:12:12.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:12.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:12.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:12.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:12.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:12.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:12.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:12.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:12.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:12.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:12.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:12.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 1, 2022-09-13T19:12:12.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:12.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:12.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:12.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:12.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:12.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:12.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:12.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:12.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:12.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:12.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:12.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 2, 2022-09-13T19:12:12.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:12.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:12.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054832.207579_149.1", 2022-09-13T19:12:12.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:12.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:12.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:12.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "2022-09-13T07:40:32.207579Z", 2022-09-13T19:12:12.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:12.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:12.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:12.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:12.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 3, 2022-09-13T19:12:12.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:12.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:12.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:12.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:12.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:12.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:12.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:12.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:12.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:12.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:12.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:12.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:12.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "full_sync": { 2022-09-13T19:12:12.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total": 1, 2022-09-13T19:12:12.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "complete": 1 2022-09-13T19:12:12.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:12.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "current_time": "2022-09-13T07:40:32Z" 2022-09-13T19:12:12.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:12.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:12.253 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:12.253 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:12.254 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:12.254 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:12.255 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:12.255 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:12.256 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:12.256 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:12.256 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:12.257 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:12.257 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:12.258 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:12.258 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:12.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:12.259 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:12.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:12.260 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:12.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:12.261 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:12.262 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:12.262 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:12.263 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:12.263 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:12.263 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:12.264 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:12.264 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:12.265 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:12.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:12.266 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:12.266 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:12.267 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:12.267 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:12.268 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:12.268 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:12.269 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:12.269 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:12.270 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054832.207579_149.1", 2022-09-13T19:12:12.270 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:12.271 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:12.271 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:12.272 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:40:32.207579Z", 2022-09-13T19:12:12.272 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:12.272 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:12.273 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:12.273 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:12.274 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:12.274 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:12.275 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:12.276 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:12.276 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:12.276 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:12.277 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:12.277 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:12.278 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:12.278 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:12.279 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:12.279 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:12.280 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:12.280 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:12.280 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:12.281 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:12.281 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:12.282 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:40:32Z" 2022-09-13T19:12:12.282 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:12.283 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054832.207579_149.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:40:32.207579Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:12.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054832.207579_149.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:12.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054832.207579_149.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:12.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2022-09-13T19:12:12.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:12.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.290 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.291 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.291 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.291 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:32 GMT 2022-09-13T19:12:12.292 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.293 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:FORhZGRUQbOyQrIevPhZRq1QEpM= 2022-09-13T19:12:12.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FORhZGRUQbOyQrIevPhZRq1QEpM=', 'Content-Length': '0'} 2022-09-13T19:12:12.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060adcb476143c2ee-00632033f0-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:12.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:12.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T19:12:12.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.299 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:12.299 INFO:tasks.rgw_multisite_tests:1B2M2Y8AsgTpgAmY7PhCfg== 2022-09-13T19:12:12.300 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:12.300 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:40:32 GMT 2022-09-13T19:12:12.301 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:12.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.302 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:WPhnKbuuHsG1m4vG8qHMH2zl2gE= 2022-09-13T19:12:12.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:40:32 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WPhnKbuuHsG1m4vG8qHMH2zl2gE='} 2022-09-13T19:12:12.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-amz-request-id', 'tx000001792f86485979461-00632033f0-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created new version id=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-20 2022-09-13T19:12:12.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:12.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:12.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:12.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:12.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:12.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:32.743+0000 7f4414aaa600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:12.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T19:12:12.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T19:12:12.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:37.744+0000 7f4414aaa600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:12.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T19:12:12.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T19:12:12.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:42.745+0000 7f4414aaa600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:12.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T19:12:12.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T19:12:12.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:47.746+0000 7f4414aaa600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:12.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T19:12:12.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T19:12:12.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:52.748+0000 7f4414aaa600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:12.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T19:12:12.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T19:12:12.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:40:57.749+0000 7f4414aaa600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:12.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T19:12:12.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T19:12:12.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:41:02.751+0000 7f4414aaa600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:12.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000001.37.5, , ] 2022-09-13T19:12:12.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T19:12:12.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:41:07.753+0000 7f4414aaa600 1 bucket sync caught up with source: 2022-09-13T19:12:12.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T19:12:12.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T19:12:12.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:41:07.753+0000 7f4414aaa600 0 bucket checkpoint complete 2022-09-13T19:12:12.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-20 2022-09-13T19:12:12.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:12.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:12.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:12.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:12.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:12.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:07.889+0000 7f0b138fa600 1 bucket sync caught up with source: 2022-09-13T19:12:12.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T19:12:12.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T19:12:12.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:07.889+0000 7f0b138fa600 0 bucket checkpoint complete 2022-09-13T19:12:12.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-20 2022-09-13T19:12:12.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:12.323 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:12.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:12.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:12.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:12.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:07.998+0000 7f275b5ab600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:12.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T19:12:12.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T19:12:12.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:12.998+0000 7f275b5ab600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:12.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T19:12:12.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T19:12:12.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:17.999+0000 7f275b5ab600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:12.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T19:12:12.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T19:12:12.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:23.001+0000 7f275b5ab600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:12.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000001.59.5, , ] 2022-09-13T19:12:12.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T19:12:12.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:28.002+0000 7f275b5ab600 1 bucket sync caught up with source: 2022-09-13T19:12:12.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T19:12:12.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T19:12:12.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:28.002+0000 7f275b5ab600 0 bucket checkpoint complete 2022-09-13T19:12:12.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-20 2022-09-13T19:12:12.332 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:12.332 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:12.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:12.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:12.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:12.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:28.127+0000 7faa3518d600 1 bucket sync caught up with source: 2022-09-13T19:12:12.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T19:12:12.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T19:12:12.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:28.127+0000 7faa3518d600 0 bucket checkpoint complete 2022-09-13T19:12:12.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-20 2022-09-13T19:12:12.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:12.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:12.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:12.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:12.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:12.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:28.247+0000 7fe052b24600 1 bucket sync caught up with source: 2022-09-13T19:12:12.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000005.70.9, , ] 2022-09-13T19:12:12.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000005.70.9, , ] 2022-09-13T19:12:12.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:28.247+0000 7fe052b24600 0 bucket checkpoint complete 2022-09-13T19:12:12.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-20 2022-09-13T19:12:12.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:12.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:12.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:12.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:12.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:12.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:41:29.373+0000 7f11e424a600 1 bucket sync caught up with source: 2022-09-13T19:12:12.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000005.70.9, , ] 2022-09-13T19:12:12.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000005.70.9, , ] 2022-09-13T19:12:12.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:41:29.373+0000 7f11e424a600 0 bucket checkpoint complete 2022-09-13T19:12:12.345 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=yxvvuh-20 zones={test-zone2, test-zone1} 2022-09-13T19:12:12.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:12.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.350 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.351 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.351 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.351 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.352 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.353 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo= 2022-09-13T19:12:12.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo=', 'Content-Length': '0'} 2022-09-13T19:12:12.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000beaa5cdecac7c639-0063203429-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:12.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.359 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.359 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.360 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.360 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.361 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.362 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo= 2022-09-13T19:12:12.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo=', 'Content-Length': '0'} 2022-09-13T19:12:12.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087126cd3f906475f-0063203429-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.363 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2022-09-13T19:12:12.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.369 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.370 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.370 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.371 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.371 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:12.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.372 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs= 2022-09-13T19:12:12.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs=', 'Content-Length': '0'} 2022-09-13T19:12:12.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edaf8c4f098ec1c6-0063203429-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.374 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:12.375 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:12.375 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2022-09-13T19:12:12.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.382 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.383 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.383 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.384 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.384 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:12.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.385 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs= 2022-09-13T19:12:12.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs=', 'Content-Length': '0'} 2022-09-13T19:12:12.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d92c87d410642e88-0063203429-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.387 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:12.388 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:12.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.395 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.396 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.396 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.397 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.397 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:12.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.398 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs= 2022-09-13T19:12:12.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs=', 'Content-Length': '0'} 2022-09-13T19:12:12.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fa264013e1f15a0-0063203429-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.407 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.407 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.408 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.408 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.409 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:12.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.410 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs= 2022-09-13T19:12:12.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs=', 'Content-Length': '0'} 2022-09-13T19:12:12.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcfeda408b5dedc9-0063203429-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.412 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:12.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.418 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.419 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.419 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.420 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.420 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.421 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T19:12:12.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T19:12:12.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000048246dad7ea8cd3f-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.428 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.428 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.429 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.429 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.430 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.431 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T19:12:12.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T19:12:12.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000006d4b220d0a5fea-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.438 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.438 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.439 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.439 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.440 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.440 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5cGa9OcM8H994XOwy0zwsP56cW4= 2022-09-13T19:12:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5cGa9OcM8H994XOwy0zwsP56cW4=', 'Content-Length': '0'} 2022-09-13T19:12:12.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a8aeedbc4650310-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.447 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.448 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.448 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.449 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.449 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.450 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5cGa9OcM8H994XOwy0zwsP56cW4= 2022-09-13T19:12:12.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5cGa9OcM8H994XOwy0zwsP56cW4=', 'Content-Length': '0'} 2022-09-13T19:12:12.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f78e302e51c58901-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.451 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:12.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.458 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.458 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.459 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.459 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.459 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.460 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T19:12:12.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T19:12:12.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec2038f43e7916de-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.467 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.468 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.468 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.468 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.469 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.470 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T19:12:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T19:12:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000243ec6f399f4c55c-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.477 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.478 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.478 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.479 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.479 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.480 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T19:12:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T19:12:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4a2ffbb3ffdf73d-0063203429-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.487 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.488 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.488 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.489 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.489 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.490 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T19:12:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T19:12:12.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bc188a8eee0498f-0063203429-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:12.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.497 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.497 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.498 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.498 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.498 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:12.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.499 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64= 2022-09-13T19:12:12.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64=', 'Content-Length': '0'} 2022-09-13T19:12:12.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000034c51afe1dca80d8-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:12.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.507 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.507 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.508 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.508 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.508 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:12.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.509 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64= 2022-09-13T19:12:12.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64=', 'Content-Length': '0'} 2022-09-13T19:12:12.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000adaf8a8808a462a8-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.511 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:12.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.518 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.519 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.519 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.520 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.520 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.521 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T19:12:12.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T19:12:12.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c12bce474144877-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.528 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.529 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.529 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.529 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.530 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.531 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T19:12:12.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T19:12:12.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080a788896b4de21b-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.538 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.538 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.539 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.539 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.540 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.541 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T19:12:12.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T19:12:12.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c26f0c4c87680b8a-0063203429-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.549 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.549 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.550 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.550 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.551 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.551 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T19:12:12.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T19:12:12.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce4f18181c34a30b-0063203429-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.553 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:12.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.560 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.560 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.561 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.561 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.562 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.562 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc= 2022-09-13T19:12:12.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc=', 'Content-Length': '0'} 2022-09-13T19:12:12.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000078adb442c5762865-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.569 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.570 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.570 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.571 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.571 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.572 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc= 2022-09-13T19:12:12.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc=', 'Content-Length': '0'} 2022-09-13T19:12:12.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7224b985c9c6127-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.591 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.592 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.592 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.593 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.593 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.594 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:pOs6vzl5rgaaVjpZ3US0Zl0YoGM= 2022-09-13T19:12:12.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pOs6vzl5rgaaVjpZ3US0Zl0YoGM=', 'Content-Length': '0'} 2022-09-13T19:12:12.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001571641238fcbc7e-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.602 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.602 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.602 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.603 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.603 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.604 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:pOs6vzl5rgaaVjpZ3US0Zl0YoGM= 2022-09-13T19:12:12.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pOs6vzl5rgaaVjpZ3US0Zl0YoGM=', 'Content-Length': '0'} 2022-09-13T19:12:12.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007ee829837c5c14ed-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.605 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:12.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.612 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.612 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.612 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.613 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.613 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.614 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc= 2022-09-13T19:12:12.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc=', 'Content-Length': '0'} 2022-09-13T19:12:12.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b80e2a64f3ccba2d-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.621 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.622 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.622 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.623 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.623 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.624 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc= 2022-09-13T19:12:12.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:WP8lp6Brczb8mXZx5q6xM1vROHc=', 'Content-Length': '0'} 2022-09-13T19:12:12.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000078405db93292c7b6-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.631 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.631 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.632 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.632 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.633 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.634 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T19:12:12.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T19:12:12.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd7e63e1f3a95b7b-0063203429-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.641 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.642 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.642 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.642 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.643 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.644 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T19:12:12.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T19:12:12.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000115acefa7c2a575c-0063203429-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:12.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.650 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.651 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.651 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.652 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.652 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:12.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.653 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64= 2022-09-13T19:12:12.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64=', 'Content-Length': '0'} 2022-09-13T19:12:12.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba6dea8da2cf4377-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:12.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.661 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.661 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.662 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.663 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.664 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:12.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.665 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64= 2022-09-13T19:12:12.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:uD5wfT8j/MV/rreEUN7FXPviR64=', 'Content-Length': '0'} 2022-09-13T19:12:12.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e33a4cf0843d0a3b-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.666 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:12.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.673 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.674 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.675 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.676 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.676 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.677 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T19:12:12.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T19:12:12.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a344a338515f8ad8-0063203429-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.685 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.685 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.686 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.686 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.687 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.688 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44= 2022-09-13T19:12:12.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:F7uF0Z9tKXXo310UfML8GujEm44=', 'Content-Length': '0'} 2022-09-13T19:12:12.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000de4231781e766acd-0063203429-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:12.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.695 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.695 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.696 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.696 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.697 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.698 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T19:12:12.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T19:12:12.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008682c58b4ce590b1-0063203429-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.706 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.706 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.707 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.707 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.707 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.708 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E= 2022-09-13T19:12:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:0QTzpm70ScZau1Z7GCUr1jk9F5E=', 'Content-Length': '0'} 2022-09-13T19:12:12.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001671d21f3b6d3447-0063203429-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.710 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=yxvvuh-20 zones={test-zone2, test-zone1} 2022-09-13T19:12:12.711 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=yxvvuh-20 zones={test-zone3, test-zone1} 2022-09-13T19:12:12.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:12.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.716 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.717 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.717 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.718 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.718 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.719 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo= 2022-09-13T19:12:12.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo=', 'Content-Length': '0'} 2022-09-13T19:12:12.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e1385daaf252966-0063203429-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:12.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.726 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.726 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.727 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.727 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.728 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.728 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo= 2022-09-13T19:12:12.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UrbC0CJ+qPTzrpFo4GIYPSgS+fo=', 'Content-Length': '0'} 2022-09-13T19:12:12.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045c821901507b1cc-0063203429-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.730 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2022-09-13T19:12:12.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.736 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.737 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.737 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.738 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.738 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:12.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.739 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs= 2022-09-13T19:12:12.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs=', 'Content-Length': '0'} 2022-09-13T19:12:12.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023ba5a6f30259907-0063203429-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.741 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:12.741 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:12.742 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2022-09-13T19:12:12.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.748 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.749 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.749 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.749 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:29 GMT 2022-09-13T19:12:12.750 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:12.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.751 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs= 2022-09-13T19:12:12.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:29 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:x/yOHaNthmtUMes07oM5JK5mCvs=', 'Content-Length': '0'} 2022-09-13T19:12:12.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f39c358f5ab612e-0063203429-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.753 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:12.753 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:12.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.759 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.760 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.760 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.761 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.761 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:12.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.762 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU= 2022-09-13T19:12:12.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU=', 'Content-Length': '0'} 2022-09-13T19:12:12.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089c41f4faeb91455-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:12.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.769 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.770 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.770 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.771 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.771 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:12.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.772 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU= 2022-09-13T19:12:12.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU=', 'Content-Length': '0'} 2022-09-13T19:12:12.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cfcb21c90326de5-006320342a-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.774 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:12.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.780 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.780 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.781 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.781 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.782 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.783 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:12.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:12.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036d676aa98b84854-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.789 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.790 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.790 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.791 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.791 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.792 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:12.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:12.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001536bb4d23aae2aa-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.799 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.800 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.800 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.801 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.801 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.802 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I= 2022-09-13T19:12:12.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I=', 'Content-Length': '0'} 2022-09-13T19:12:12.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005577e6148f665db0-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.809 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.810 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.810 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.810 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.811 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.812 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I= 2022-09-13T19:12:12.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I=', 'Content-Length': '0'} 2022-09-13T19:12:12.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fc428cb979436f02-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.813 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:12.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.819 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.820 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.820 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.821 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.821 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.822 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:12.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:12.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d2adace9c5b3223b-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.829 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.829 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.830 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.830 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.831 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.832 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:12.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:12.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3efa51f24c75db7-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.839 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.839 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.840 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.841 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.841 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.843 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:12.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:12.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020c794b59aca81e2-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.851 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.851 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.851 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.852 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.852 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.853 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:12.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:12.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052170e7d071581ee-006320342a-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:12.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.860 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.860 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.861 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.861 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.862 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:12.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.862 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T19:12:12.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T19:12:12.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000558a6fe0914596bb-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:12.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.869 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.869 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.869 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.870 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.870 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.871 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T19:12:12.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T19:12:12.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000396a9dcda792c5bd-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.872 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.879 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.879 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.880 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.880 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.880 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.881 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:12.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:12.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000945d43e79cb77b24-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.888 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.889 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.889 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.890 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.890 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:12.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.891 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:12.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:12.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bc460de5455f4326-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.898 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.899 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.899 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.899 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.900 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.901 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:12.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:12.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd24cbbb80e7ac7e-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.908 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.909 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.909 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.910 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.910 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.911 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:12.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:12.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba38aa67d3974644-006320342a-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.913 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:12.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.919 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.919 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.920 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.920 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.921 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.922 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T19:12:12.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T19:12:12.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003559b7a554e9805c-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.929 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.930 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.930 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.931 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.931 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.932 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T19:12:12.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T19:12:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006e838c5d42dbe37b-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.939 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.939 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.940 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.940 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.941 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.942 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M= 2022-09-13T19:12:12.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M=', 'Content-Length': '0'} 2022-09-13T19:12:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000031a17e0e7c3091e5-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.949 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.949 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.950 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.950 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.951 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.951 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M= 2022-09-13T19:12:12.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M=', 'Content-Length': '0'} 2022-09-13T19:12:12.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d4f6796a2f640338-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.953 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:12.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.959 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.960 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.960 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.961 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.961 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.962 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T19:12:12.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T19:12:12.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000716166009d5b7a2f-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.969 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.970 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.970 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.971 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.971 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.972 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T19:12:12.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T19:12:12.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dbdd6c77dc8634b3-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.979 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.979 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.980 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.980 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.981 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.981 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:12.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:12.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdf2b06eb872ef2d-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:12.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:12.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:12.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:12.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:12.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:12.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.989 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:12.990 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.990 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:12.991 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:12.991 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:12.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:12.992 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:12.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:12.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e694b7fdc127a66d-006320342a-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:12.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:12.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:12.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:12.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:12.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:12.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:12.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:12.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:12.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:12.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:12.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:12.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:12.999 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:12.999 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.000 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.000 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.001 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:13.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.002 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T19:12:13.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T19:12:13.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004d3e91f658c8aa95-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:13.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:13.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.008 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.008 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.009 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.009 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.010 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:13.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.010 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T19:12:13.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T19:12:13.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dc18f588e4df86da-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.012 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:13.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.018 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.019 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.019 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.019 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.020 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.021 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:13.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:13.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006a3cad874257f33c-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:13.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.028 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.029 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.029 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.030 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.030 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.031 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:13.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:13.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005da626c317dea5ab-006320342a-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.039 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.039 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.040 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.040 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.041 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:13.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.042 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:13.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:13.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eb2c079582a30c2-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:13.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.049 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.050 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.050 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.051 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.052 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:13.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.053 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:13.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:13.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023b5a17ac42304d5-006320342a-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.054 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=yxvvuh-20 zones={test-zone3, test-zone1} 2022-09-13T19:12:13.055 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=yxvvuh-20 zones={test-zone3, test-zone2} 2022-09-13T19:12:13.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:13.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.060 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.061 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.061 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.062 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.062 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:13.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.063 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:cfkVsbBaPWLCGHlKc9e3SyLA+dA= 2022-09-13T19:12:13.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cfkVsbBaPWLCGHlKc9e3SyLA+dA=', 'Content-Length': '0'} 2022-09-13T19:12:13.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e073bf541c70a73-006320342a-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:13.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.069 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.070 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.070 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.071 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.071 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:13.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.072 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:cfkVsbBaPWLCGHlKc9e3SyLA+dA= 2022-09-13T19:12:13.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:cfkVsbBaPWLCGHlKc9e3SyLA+dA=', 'Content-Length': '0'} 2022-09-13T19:12:13.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e5b661bcbada1f2-006320342a-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.073 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2022-09-13T19:12:13.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.080 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.080 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.080 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.081 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.081 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:13.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.082 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU= 2022-09-13T19:12:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU=', 'Content-Length': '0'} 2022-09-13T19:12:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1505b70c007b2ac-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.084 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:13.084 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:13.085 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2022-09-13T19:12:13.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.091 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.091 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.092 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.092 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.093 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:13.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.094 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU= 2022-09-13T19:12:13.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU=', 'Content-Length': '0'} 2022-09-13T19:12:13.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d99f48f46a63a3e3-006320342a-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.096 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:13.096 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:13.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.102 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.103 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.103 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.103 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.104 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:13.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.105 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU= 2022-09-13T19:12:13.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU=', 'Content-Length': '0'} 2022-09-13T19:12:13.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005107f9ba0119b77a-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.112 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.113 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.113 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.114 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.114 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:13.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.115 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU= 2022-09-13T19:12:13.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:aBCDwnL0qedAXhM/biVUiV/qnhU=', 'Content-Length': '0'} 2022-09-13T19:12:13.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004095cd4f535e0ead-006320342a-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vtrue2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-13T07:40:00.402Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.117 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:13.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.123 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.123 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.124 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.124 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.125 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.125 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:13.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:13.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000823160c066ca8895-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.133 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.133 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.133 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.134 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.134 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.135 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:13.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:13.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cf44f5863395efd3-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.142 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.143 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.143 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.144 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.144 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.145 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I= 2022-09-13T19:12:13.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I=', 'Content-Length': '0'} 2022-09-13T19:12:13.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a7df7f3de4ba3b4e-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.152 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.152 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.153 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.153 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.154 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.155 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I= 2022-09-13T19:12:13.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:vLKHvd4CCOYRai7F4BLTsAs8u0I=', 'Content-Length': '0'} 2022-09-13T19:12:13.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bb51d38c84ead971-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.156 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:13.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.162 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.163 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.164 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.164 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.165 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.165 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:13.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:13.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009f17046fc3ef071a-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.173 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.173 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.174 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.174 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.175 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.175 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:13.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:13.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000025c3b0170dad55ae-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.182 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.183 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.183 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.184 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.184 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:13.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.185 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:13.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:13.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003131d164b68daa4e-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.192 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.193 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.193 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.194 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.194 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:13.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.195 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:13.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:13.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000196e4ceb4b3f1a4f-006320342a-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:13.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.202 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.202 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.202 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.203 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.203 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:13.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.204 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T19:12:13.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T19:12:13.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ce1639f6f2198a47-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:13.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.210 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.211 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.211 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.212 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.212 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:13.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.213 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T19:12:13.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T19:12:13.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b98df274c61cc057-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.214 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:13.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.220 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.221 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.221 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.222 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.222 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.223 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:13.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:13.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006764680cc4462b68-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.230 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.231 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.231 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.231 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.232 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.233 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:13.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:13.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c355747b315d3de-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.240 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.240 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.241 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.241 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.241 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:13.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.242 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:13.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:13.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c35235e3876a7ef-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.250 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.250 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.251 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.251 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.251 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:13.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.252 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:13.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:13.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f2fd5155ddde9f1-006320342a-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.254 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:13.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.260 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.261 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.261 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.261 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.262 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.263 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T19:12:13.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T19:12:13.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000074632187ded208f0-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.270 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.270 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.271 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.271 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.272 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.273 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T19:12:13.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T19:12:13.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cccc5c7dc66a5755-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.280 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.280 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.280 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.281 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.281 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.282 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M= 2022-09-13T19:12:13.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M=', 'Content-Length': '0'} 2022-09-13T19:12:13.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007b5e630975f84408-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.289 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.289 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.290 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.290 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.291 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.292 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M= 2022-09-13T19:12:13.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eCt+MKVTKVsKAz8ewlfKik1Rf3M=', 'Content-Length': '0'} 2022-09-13T19:12:13.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d02dc8921ae2163a-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.293 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:13.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.300 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.300 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.301 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.301 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.302 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.303 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T19:12:13.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T19:12:13.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000974b78775a785fc6-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.310 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.310 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.311 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.311 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.311 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.312 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0= 2022-09-13T19:12:13.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:eFiyBp8a22cWHO5pacH3Xm/rVB0=', 'Content-Length': '0'} 2022-09-13T19:12:13.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002ca3a0d5229a2da7-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.319 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.320 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.320 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.321 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.321 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:13.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.322 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:13.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:13.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be846e69d6caf940-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.330 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.330 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.331 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.331 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.332 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:13.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.333 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:13.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:13.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1a5ddba3a6b4ad2-006320342a-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:13.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.344 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.345 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.346 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.347 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.347 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:13.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.349 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T19:12:13.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T19:12:13.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a44b9f5130653620-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:13.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.358 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.358 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.359 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.359 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.360 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:13.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.361 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y= 2022-09-13T19:12:13.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:trAvquUiAj5ZE459RWJko1F0s0Y=', 'Content-Length': '0'} 2022-09-13T19:12:13.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004fc460460c08fea6-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.363 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:13.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.374 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.375 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.375 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.376 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.377 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.379 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:13.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:13.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000089df5bfe26a5ea62-006320342a-10f3-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.392 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.392 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.393 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.394 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.395 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=z8eAuq6jQubRLW50nSXtjtNqYKt6T4V 2022-09-13T19:12:13.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.397 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA= 2022-09-13T19:12:13.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:5ZHerOQj5KnUyuQFolXJIVkvOrA=', 'Content-Length': '0'} 2022-09-13T19:12:13.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:32 GMT'), ('x-amz-version-id', 'z8eAuq6jQubRLW50nSXtjtNqYKt6T4V'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e7e3add489db3a3-006320342a-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.404 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.405 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.405 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.406 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.406 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:13.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.407 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:13.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:13.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd31c40f27b96a3c-006320342a-10f3-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?prefix=obj 2022-09-13T19:12:13.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.414 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.415 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.415 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.416 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.416 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:13.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.417 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU= 2022-09-13T19:12:13.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ImMCZyH6JeBqdLHkA+fwXbR6smU=', 'Content-Length': '0'} 2022-09-13T19:12:13.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7dd4170be753bfc-006320342a-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:41:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20obj1000falseobj2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.419 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=yxvvuh-20 zones={test-zone3, test-zone2} 2022-09-13T19:12:13.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versioning 2022-09-13T19:12:13.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versioning 2022-09-13T19:12:13.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:13.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versioning 2022-09-13T19:12:13.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.424 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.425 INFO:tasks.rgw_multisite_tests: Suspended 2022-09-13T19:12:13.426 INFO:tasks.rgw_multisite_tests: Disabled 2022-09-13T19:12:13.426 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:13.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.431 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:13.432 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.432 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.432 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:30 GMT 2022-09-13T19:12:13.433 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versioning 2022-09-13T19:12:13.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.434 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:rVm4VllGmmxpekZ1BWdIOIx3gqk= 2022-09-13T19:12:13.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:30 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rVm4VllGmmxpekZ1BWdIOIx3gqk=', 'Content-Length': '230'} 2022-09-13T19:12:13.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000005cf9947917207d96-006320342a-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:13.436 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:13.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:13.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:13.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:13.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:13.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:13.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:13.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:13.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:13.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054891.002339_151.1", 2022-09-13T19:12:13.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:41:31.002339Z" 2022-09-13T19:12:13.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:13.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:13.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:13.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:13.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054891.002339_151.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:13.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:13.446 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:13.447 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:13.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:13.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:13.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:41:31.234+0000 7f5427eae600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:41:31.235+0000 7f5427eae600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:13.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:13.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:13.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:13.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:13.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:13.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:13.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:13.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:13.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:13.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:13.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:13.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:13.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:13.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:13.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:13.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:13.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:13.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:13.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:13.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:13.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:13.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:13.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:13.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:13.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:13.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054891.002339_151.1", 2022-09-13T19:12:13.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:13.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:41:31.002339Z", 2022-09-13T19:12:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:13.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:13.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:13.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:13.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:13.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:13.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:13.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:13.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:13.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:13.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:13.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:41:31Z" 2022-09-13T19:12:13.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:13.478 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:13.479 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:13.479 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:13.480 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:13.480 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:13.481 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.481 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.481 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:13.482 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.482 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:13.483 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.483 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.484 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:13.484 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.485 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:13.485 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.485 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:13.486 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.486 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.487 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.487 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.488 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:13.488 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.488 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.489 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:13.489 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.490 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:13.490 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.491 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:13.491 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.492 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.492 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.492 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.493 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:13.493 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.494 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.494 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054891.002339_151.1", 2022-09-13T19:12:13.495 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.495 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:13.496 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.496 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:41:31.002339Z", 2022-09-13T19:12:13.496 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.497 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.497 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.498 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.498 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:13.499 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.499 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.499 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:13.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.500 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:13.501 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.501 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:13.502 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.502 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.502 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.503 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:13.503 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.504 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:13.504 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:13.505 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:13.505 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.505 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:41:31Z" 2022-09-13T19:12:13.506 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:13.506 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054891.002339_151.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:41:31.002339Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:13.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054891.002339_151.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:13.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054891.002339_151.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:13.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2022-09-13T19:12:13.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2022-09-13T19:12:13.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:13.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:13.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:13.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:13.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:13.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:31.355+0000 7fae2eab1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:13.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:31.356+0000 7fae2eab1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:13.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "sync_status": { 2022-09-13T19:12:13.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "info": { 2022-09-13T19:12:13.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "status": "sync", 2022-09-13T19:12:13.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:13.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:13.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "markers": [ 2022-09-13T19:12:13.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 0, 2022-09-13T19:12:13.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:13.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:13.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:13.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:13.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:13.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:13.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 1, 2022-09-13T19:12:13.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:13.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:13.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:13.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:13.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:13.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:13.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 2, 2022-09-13T19:12:13.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:13.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:13.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054890.998697_150.1", 2022-09-13T19:12:13.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:13.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "2022-09-13T07:41:30.998697Z", 2022-09-13T19:12:13.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:13.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 3, 2022-09-13T19:12:13.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:13.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:13.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:13.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:13.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:13.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:13.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:13.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:13.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "full_sync": { 2022-09-13T19:12:13.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total": 1, 2022-09-13T19:12:13.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "complete": 1 2022-09-13T19:12:13.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "current_time": "2022-09-13T07:41:31Z" 2022-09-13T19:12:13.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:13.540 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:13.541 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:13.541 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:13.542 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:13.542 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:13.543 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:13.544 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.544 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.547 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:13.548 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.548 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:13.549 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.550 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.550 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:13.551 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.551 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:13.551 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.552 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:13.552 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.553 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.553 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.553 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.554 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:13.554 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.555 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.555 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:13.556 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.556 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:13.556 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.557 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:13.557 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.558 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.558 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.558 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.559 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:13.559 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.560 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.560 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054890.998697_150.1", 2022-09-13T19:12:13.561 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.561 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:13.562 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.562 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:41:30.998697Z", 2022-09-13T19:12:13.563 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.563 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.564 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.564 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.565 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:13.565 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.566 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.566 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:13.567 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.567 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:13.567 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.568 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:13.568 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.569 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.569 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.570 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:13.570 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.570 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:13.571 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:13.571 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:13.572 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.572 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:41:31Z" 2022-09-13T19:12:13.573 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:13.573 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054890.998697_150.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:41:30.998697Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:13.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054891.002339_151.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:13.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054890.998697_150.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:13.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1663054891.002339_151.1 target=1_1663054890.998697_150.1 2022-09-13T19:12:13.575 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:13.576 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:13.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:13.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:13.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:13.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:36.493+0000 7fe5b0d22600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:13.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:36.495+0000 7fe5b0d22600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:13.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "sync_status": { 2022-09-13T19:12:13.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "info": { 2022-09-13T19:12:13.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "status": "sync", 2022-09-13T19:12:13.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:13.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:13.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "markers": [ 2022-09-13T19:12:13.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 0, 2022-09-13T19:12:13.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:13.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:13.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:13.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:13.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:13.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:13.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 1, 2022-09-13T19:12:13.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:13.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:13.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:13.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:13.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:13.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:13.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 2, 2022-09-13T19:12:13.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:13.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:13.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054891.002339_151.1", 2022-09-13T19:12:13.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:13.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "2022-09-13T07:41:31.002339Z", 2022-09-13T19:12:13.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:13.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "key": 3, 2022-09-13T19:12:13.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "val": { 2022-09-13T19:12:13.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "state": 1, 2022-09-13T19:12:13.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:13.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:13.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:13.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:13.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:13.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:13.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "full_sync": { 2022-09-13T19:12:13.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "total": 1, 2022-09-13T19:12:13.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "complete": 1 2022-09-13T19:12:13.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "current_time": "2022-09-13T07:41:36Z" 2022-09-13T19:12:13.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:13.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:13.607 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:13.607 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:13.607 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:13.608 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:13.608 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:13.609 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.609 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.609 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:13.610 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.610 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:13.611 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.612 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:13.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.612 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:13.613 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.613 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:13.614 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.614 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.614 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.615 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.615 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:13.616 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.616 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.616 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:13.617 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.617 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:13.618 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.618 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:13.618 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.619 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.619 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.620 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.620 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:13.621 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.621 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.622 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054891.002339_151.1", 2022-09-13T19:12:13.622 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.623 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:13.623 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.623 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:41:31.002339Z", 2022-09-13T19:12:13.624 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.624 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.625 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.626 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.626 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:13.627 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.627 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.628 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:13.628 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.629 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:13.630 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.630 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:13.631 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.631 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.632 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.632 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:13.633 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.633 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:13.634 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:13.634 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:13.634 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.635 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:41:36Z" 2022-09-13T19:12:13.635 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:13.636 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663054891.002339_151.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:41:31.002339Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:13.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054891.002339_151.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:13.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663054891.002339_151.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:13.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2022-09-13T19:12:13.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:13.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:13.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.642 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.643 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.643 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.644 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:36 GMT 2022-09-13T19:12:13.644 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:13.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.645 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:QPaSPhpU4ACoMJKkRkhSs++rK48= 2022-09-13T19:12:13.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:41:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:QPaSPhpU4ACoMJKkRkhSs++rK48=', 'Content-Length': '0'} 2022-09-13T19:12:13.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b11bf367ffa556eb-0063203430-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:41:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:13.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj 2022-09-13T19:12:13.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-13T19:12:13.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:13.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.652 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:13.652 INFO:tasks.rgw_multisite_tests:1B2M2Y8AsgTpgAmY7PhCfg== 2022-09-13T19:12:13.653 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:13.653 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:41:36 GMT 2022-09-13T19:12:13.654 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj 2022-09-13T19:12:13.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.655 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:76ZbTaNDQozrGw875MGrcRrwrnA= 2022-09-13T19:12:13.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:41:36 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:76ZbTaNDQozrGw875MGrcRrwrnA='} 2022-09-13T19:12:13.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000871903815a085c67-0063203430-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:41:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created null version id=None 2022-09-13T19:12:13.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yxvvuh-20 2022-09-13T19:12:13.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:13.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:13.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:13.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:13.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:13.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:41:36.715+0000 7faffb600600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:13.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T19:12:13.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T19:12:13.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:41:41.717+0000 7faffb600600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:13.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T19:12:13.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T19:12:13.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:41:46.719+0000 7faffb600600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:13.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T19:12:13.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T19:12:13.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:41:51.721+0000 7faffb600600 1 bucket sync caught up with source: 2022-09-13T19:12:13.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T19:12:13.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T19:12:13.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:41:51.721+0000 7faffb600600 0 bucket checkpoint complete 2022-09-13T19:12:13.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=yxvvuh-20 2022-09-13T19:12:13.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:13.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:13.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:13.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:13.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:13.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:51.862+0000 7fce40f14600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:13.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000003.40.12, , ] 2022-09-13T19:12:13.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T19:12:13.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:56.864+0000 7fce40f14600 1 bucket sync caught up with source: 2022-09-13T19:12:13.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T19:12:13.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000005.44.17, , ] 2022-09-13T19:12:13.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:56.864+0000 7fce40f14600 0 bucket checkpoint complete 2022-09-13T19:12:13.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yxvvuh-20 2022-09-13T19:12:13.671 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:13.672 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:13.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:13.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:13.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:13.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:41:56.984+0000 7f7ada256600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:13.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T19:12:13.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000006.70.17, , ] 2022-09-13T19:12:13.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:42:01.986+0000 7f7ada256600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:13.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000004.65.9, , ] 2022-09-13T19:12:13.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000006.70.17, , ] 2022-09-13T19:12:13.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:42:06.987+0000 7f7ada256600 1 bucket sync caught up with source: 2022-09-13T19:12:13.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000007.72.14, , ] 2022-09-13T19:12:13.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000006.70.17, , ] 2022-09-13T19:12:13.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:42:06.987+0000 7f7ada256600 0 bucket checkpoint complete 2022-09-13T19:12:13.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone3 bucket=yxvvuh-20 2022-09-13T19:12:13.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:13.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:13.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:13.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:13.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:13.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:42:07.125+0000 7f3452c03600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:13.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000006.70.17, , ] 2022-09-13T19:12:13.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000007.72.14, , ] 2022-09-13T19:12:13.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:42:12.126+0000 7f3452c03600 1 waiting for incremental sync to catch up: 2022-09-13T19:12:13.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000006.70.17, , ] 2022-09-13T19:12:13.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000007.72.14, , ] 2022-09-13T19:12:13.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:42:17.128+0000 7f3452c03600 1 bucket sync caught up with source: 2022-09-13T19:12:13.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000007.72.14, , ] 2022-09-13T19:12:13.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000007.72.14, , ] 2022-09-13T19:12:13.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:42:17.128+0000 7f3452c03600 0 bucket checkpoint complete 2022-09-13T19:12:13.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone3 target=test-zone1 bucket=yxvvuh-20 2022-09-13T19:12:13.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:13.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:13.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:13.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:13.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:13.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:42:17.253+0000 7fad0ff40600 1 bucket sync caught up with source: 2022-09-13T19:12:13.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: local status: [, , , , , , , , 00000000008.77.14, , ] 2022-09-13T19:12:13.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: remote markers: [, , , , , , , , 00000000008.77.14, , ] 2022-09-13T19:12:13.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: 2022-09-13T07:42:17.253+0000 7fad0ff40600 0 bucket checkpoint complete 2022-09-13T19:12:13.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone3 target=test-zone2 bucket=yxvvuh-20 2022-09-13T19:12:13.691 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:13.692 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yxvvuh-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-13T19:12:13.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yxvvuh-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-13T19:12:13.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:13.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:13.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:42:17.383+0000 7fb55b741600 1 bucket sync caught up with source: 2022-09-13T19:12:13.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: local status: [, , , , , , , , 00000000008.77.14, , ] 2022-09-13T19:12:13.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: remote markers: [, , , , , , , , 00000000008.77.14, , ] 2022-09-13T19:12:13.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:42:17.383+0000 7fb55b741600 0 bucket checkpoint complete 2022-09-13T19:12:13.696 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=yxvvuh-20 zones={test-zone2, test-zone1} 2022-09-13T19:12:13.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:13.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.701 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.702 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.702 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.702 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:42:17 GMT 2022-09-13T19:12:13.703 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:13.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.704 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:urpWmF5z9SLCfk+PfbZ8VYfYAYw= 2022-09-13T19:12:13.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:urpWmF5z9SLCfk+PfbZ8VYfYAYw=', 'Content-Length': '0'} 2022-09-13T19:12:13.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e7a268231060543-0063203459-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/ 2022-09-13T19:12:13.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:13.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.710 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.710 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.711 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.711 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:42:17 GMT 2022-09-13T19:12:13.712 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/ 2022-09-13T19:12:13.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.712 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:urpWmF5z9SLCfk+PfbZ8VYfYAYw= 2022-09-13T19:12:13.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:urpWmF5z9SLCfk+PfbZ8VYfYAYw=', 'Content-Length': '0'} 2022-09-13T19:12:13.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090583e5261689af3-0063203459-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.714 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2022-09-13T19:12:13.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.720 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.720 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.721 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.721 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:42:17 GMT 2022-09-13T19:12:13.722 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:13.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.723 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4= 2022-09-13T19:12:13.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4=', 'Content-Length': '0'} 2022-09-13T19:12:13.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058b5bdf7a4c03347-0063203459-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:41:36.570Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vfalse2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.724 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:13.725 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:13.725 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2022-09-13T19:12:13.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:13.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.732 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.733 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.733 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.734 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:42:17 GMT 2022-09-13T19:12:13.734 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:13.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.735 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4= 2022-09-13T19:12:13.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4=', 'Content-Length': '0'} 2022-09-13T19:12:13.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008db5e8a30ca30504-0063203459-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:41:36.570Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vfalse2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.738 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:13.738 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj 2022-09-13T19:12:13.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.745 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.746 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.747 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.747 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:42:17 GMT 2022-09-13T19:12:13.748 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:13.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.748 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4= 2022-09-13T19:12:13.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4=', 'Content-Length': '0'} 2022-09-13T19:12:13.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ea7eea92babc888-0063203459-10c9-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:41:36.570Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vfalse2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/ 2022-09-13T19:12:13.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/ 2022-09-13T19:12:13.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/?versions&max-keys=999 2022-09-13T19:12:13.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:13.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.756 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.757 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.757 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.758 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:42:17 GMT 2022-09-13T19:12:13.758 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/?versions 2022-09-13T19:12:13.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.759 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4= 2022-09-13T19:12:13.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:kAcpQiAtW4oQcJKpXeh4nyANo+4=', 'Content-Length': '0'} 2022-09-13T19:12:13.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b560d40bfa8236b4-0063203459-10ed-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yxvvuh-20999falseobjnulltrue2022-09-13T07:41:36.570Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz8eAuq6jQubRLW50nSXtjtNqYKt6T4Vfalse2022-09-13T07:40:32.619Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-13T19:12:13.761 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:13.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.767 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.767 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.767 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.768 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:42:17 GMT 2022-09-13T19:12:13.768 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.769 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:9bh+8rjcE7s01JTF3kfeMA32Kys= 2022-09-13T19:12:13.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9bh+8rjcE7s01JTF3kfeMA32Kys=', 'Content-Length': '0'} 2022-09-13T19:12:13.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002bc4b5c94fe3174a-0063203459-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2022-09-13T19:12:13.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:13.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.776 INFO:tasks.rgw_multisite_tests:GET 2022-09-13T19:12:13.777 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.777 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.778 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:42:17 GMT 2022-09-13T19:12:13.778 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.779 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:9bh+8rjcE7s01JTF3kfeMA32Kys= 2022-09-13T19:12:13.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9bh+8rjcE7s01JTF3kfeMA32Kys=', 'Content-Length': '0'} 2022-09-13T19:12:13.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:41:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000019fd4b3ca0b852f1-0063203459-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.787 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.787 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.788 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.788 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:42:17 GMT 2022-09-13T19:12:13.789 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.790 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ycwFijssen/dc7uoVLamXRvq97M= 2022-09-13T19:12:13.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ycwFijssen/dc7uoVLamXRvq97M=', 'Content-Length': '0'} 2022-09-13T19:12:13.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:40:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000087eeaaa6ec53203f-0063203459-10c9-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj 2022-09-13T19:12:13.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj 2022-09-13T19:12:13.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:13.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:13.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.798 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:13.799 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.799 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.799 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:42:17 GMT 2022-09-13T19:12:13.800 INFO:tasks.rgw_multisite_tests:/yxvvuh-20/obj?versionId=null 2022-09-13T19:12:13.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.801 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ycwFijssen/dc7uoVLamXRvq97M= 2022-09-13T19:12:13.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:42:17 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ycwFijssen/dc7uoVLamXRvq97M=', 'Content-Length': '0'} 2022-09-13T19:12:13.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 13 Sep 2022 07:41:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e884b945212f9d95-0063203459-10ed-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 13 Sep 2022 07:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.802 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj 2022-09-13T19:12:13.802 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:13.803 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.803 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:13.804 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_delete_notempty 2022-09-13T19:12:13.804 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:13.807 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:13.808 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:13.809 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:13.809 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 866, in test_bucket_delete_notempty 2022-09-13T19:12:13.810 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2022-09-13T19:12:13.811 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:13.812 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2022-09-13T19:12:13.813 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 196, in zone_meta_checkpoint 2022-09-13T19:12:13.814 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2022-09-13T19:12:13.814 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2022-09-13T19:12:13.815 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:13.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=yxvvuh-36 2022-09-13T19:12:13.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-36/ 2022-09-13T19:12:13.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-36/ 2022-09-13T19:12:13.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:13.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-36/ 2022-09-13T19:12:13.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:13.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.825 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:13.826 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.827 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.827 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:45:01 GMT 2022-09-13T19:12:13.828 INFO:tasks.rgw_multisite_tests:/yxvvuh-36/ 2022-09-13T19:12:13.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.829 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:1PAGcd/B9huMFYnrzo7LV2TerNs= 2022-09-13T19:12:13.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:45:01 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1PAGcd/B9huMFYnrzo7LV2TerNs=', 'Content-Length': '0'} 2022-09-13T19:12:13.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000b1f299f414eb203f-00632034fd-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:45:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=yxvvuh-37 2022-09-13T19:12:13.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-37/ 2022-09-13T19:12:13.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-37/ 2022-09-13T19:12:13.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:13.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-37/ 2022-09-13T19:12:13.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:13.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:13.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.840 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:13.840 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.841 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.842 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:45:01 GMT 2022-09-13T19:12:13.843 INFO:tasks.rgw_multisite_tests:/yxvvuh-37/ 2022-09-13T19:12:13.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.845 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:y8aFfrd7Ow+XZDeXWjkLko2PA7Y= 2022-09-13T19:12:13.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:45:01 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:y8aFfrd7Ow+XZDeXWjkLko2PA7Y=', 'Content-Length': '0'} 2022-09-13T19:12:13.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000ca5d6d20cddc8cf4-00632034fd-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:45:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=yxvvuh-38 2022-09-13T19:12:13.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-38/ 2022-09-13T19:12:13.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-38/ 2022-09-13T19:12:13.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:13.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-38/ 2022-09-13T19:12:13.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:13.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:13.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:13.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:13.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:13.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:13.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:13.855 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:13.856 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.857 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.858 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:45:01 GMT 2022-09-13T19:12:13.859 INFO:tasks.rgw_multisite_tests:/yxvvuh-38/ 2022-09-13T19:12:13.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:13.860 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:lS5NYml0lKg3RVXjB+2L1BP7Ho0= 2022-09-13T19:12:13.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:45:01 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lS5NYml0lKg3RVXjB+2L1BP7Ho0=', 'Content-Length': '0'} 2022-09-13T19:12:13.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000090e745dc3b9c25c4-00632034fd-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:45:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:13.861 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:13.862 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:13.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:13.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:13.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:13.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:13.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:13.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:13.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:13.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:13.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:13.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663055101.786343_268.1", 2022-09-13T19:12:13.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:45:01.786343Z" 2022-09-13T19:12:13.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:13.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:13.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:13.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:13.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:13.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:13.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:13.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:13.874 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:13.875 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:13.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:13.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:13.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:13.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:02.051+0000 7fe090871600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:13.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:02.052+0000 7fe090871600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:13.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:13.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:13.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:13.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:13.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:13.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:13.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:13.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:13.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:13.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:13.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:13.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:13.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:13.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:13.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:13.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:13.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:13.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:13.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:13.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:13.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:13.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:13.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:13.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:13.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:13.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:13.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:13.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:13.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:13.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:13.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:13.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:13.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:13.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:13.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:13.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:13.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:13.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:13.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:13.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:45:02Z" 2022-09-13T19:12:13.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:13.909 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:13.909 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:13.910 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:13.910 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:13.911 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:13.911 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.911 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.912 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:13.912 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.913 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:13.913 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.914 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.914 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:13.915 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.915 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:13.916 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.916 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:13.916 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.917 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.917 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.918 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.918 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:13.919 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.919 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.920 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:13.920 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.920 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:13.921 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.921 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:13.922 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.922 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.923 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.923 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.923 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:13.924 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.924 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.925 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:13.925 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.926 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:13.926 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.926 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:13.927 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.927 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.928 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.929 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.929 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:13.930 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.930 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.930 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:13.931 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.931 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:13.932 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.932 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:13.933 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.933 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.933 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.934 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:13.934 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.935 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:13.935 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:13.936 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:13.936 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.936 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:45:02Z" 2022-09-13T19:12:13.937 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:13.937 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:13.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:13.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:13.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:13.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.858000_209.1 2022-09-13T19:12:13.940 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:13.940 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:13.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:13.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:13.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:13.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:07.242+0000 7f2ca377e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:13.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:07.243+0000 7f2ca377e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:13.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:13.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:13.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:13.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:13.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:13.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:13.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:13.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:13.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:13.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:13.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:13.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:13.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:13.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:13.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:13.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:13.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:13.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:13.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:13.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:13.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:13.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:13.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:13.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:13.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:13.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:13.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:13.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:13.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:13.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:13.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:13.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:13.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:13.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:13.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:13.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:13.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:13.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:13.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:13.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:13.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:13.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:13.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:13.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:45:07Z" 2022-09-13T19:12:13.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:13.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:13.972 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:13.972 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:13.973 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:13.973 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:13.974 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:13.974 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.974 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.975 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:13.975 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.976 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:13.976 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.977 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.977 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:13.977 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.978 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:13.978 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.979 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:13.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.980 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.980 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.981 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.981 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:13.982 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.982 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:13.983 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.983 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:13.984 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.984 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:13.985 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.985 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.985 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.986 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.986 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:13.987 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.987 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.988 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:13.988 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.989 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:13.989 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.990 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:13.990 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.990 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.991 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.991 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:13.992 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:13.992 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:13.993 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:13.993 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:13.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:13.994 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:13.995 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:13.995 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:13.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:13.996 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.996 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:13.997 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:13.997 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.998 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:13.998 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:13.999 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:13.999 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:13.999 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:45:07Z" 2022-09-13T19:12:14.000 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:14.000 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:14.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:14.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.900410_211.1 2022-09-13T19:12:14.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.003 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:14.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:14.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:14.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:12.376+0000 7fee8d2bd600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:12.378+0000 7fee8d2bd600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:14.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:14.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:14.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:14.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:14.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:14.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:14.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:14.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:14.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:14.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:14.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:14.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:14.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:14.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:14.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:14.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:14.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:14.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:45:12Z" 2022-09-13T19:12:14.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:14.034 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:14.034 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:14.035 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:14.035 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:14.036 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.036 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.037 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:14.037 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.038 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:14.038 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.039 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.039 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.040 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.040 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:14.040 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.041 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.041 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.042 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.042 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.043 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.043 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:14.044 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.044 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.044 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:14.045 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.045 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.046 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.046 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:14.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.047 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.048 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.048 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.048 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:14.049 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.049 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.050 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:14.050 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.051 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.052 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:14.052 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.052 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.053 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.053 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.054 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:14.054 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.055 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.055 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.055 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.056 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.056 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.057 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.057 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.058 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.058 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.059 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:14.059 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.059 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:14.060 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:14.060 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:14.061 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.061 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:45:12Z" 2022-09-13T19:12:14.062 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:14.062 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:14.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:14.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:14.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.065 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:14.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:14.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:14.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:17.551+0000 7fe1fcacb600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:17.552+0000 7fe1fcacb600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:14.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:14.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:14.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:14.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:14.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:14.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:14.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:14.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:14.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:14.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:14.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:14.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:14.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:14.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:14.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:14.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:14.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:14.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:45:17Z" 2022-09-13T19:12:14.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:14.097 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:14.097 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:14.098 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:14.098 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:14.099 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.099 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.100 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.100 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:14.100 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.101 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:14.101 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.102 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.102 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.103 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.103 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:14.104 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.104 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.105 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.105 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.105 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.106 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.106 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:14.107 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.107 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.108 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:14.108 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.109 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.109 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.109 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:14.110 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.110 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.111 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.111 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.111 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:14.112 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.112 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.113 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:14.113 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.114 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.114 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.115 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:14.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.116 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.116 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.117 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.117 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:14.118 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.118 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.119 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.119 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.120 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.120 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.120 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.121 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.121 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.122 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.122 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:14.122 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.123 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:14.123 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:14.124 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:14.124 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.125 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:45:17Z" 2022-09-13T19:12:14.125 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:14.125 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:14.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:14.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.861945_210.1 2022-09-13T19:12:14.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:14.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:14.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:14.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:22.683+0000 7fee813dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:22.684+0000 7fee813dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:14.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:14.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:14.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:14.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:14.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:14.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:14.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:14.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:14.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:14.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:14.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.906479_212.1", 2022-09-13T19:12:14.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.906479Z", 2022-09-13T19:12:14.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:14.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:14.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:14.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:14.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:14.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:45:22Z" 2022-09-13T19:12:14.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:14.164 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:14.164 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:14.165 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:14.165 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:14.165 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.166 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.166 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.167 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:14.167 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.168 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:14.168 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.168 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.169 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.170 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.170 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:14.171 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.171 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.172 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.172 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.173 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.173 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.174 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:14.174 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.175 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.175 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:14.176 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.176 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.177 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.177 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:14.177 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.178 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.178 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.179 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.179 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:14.180 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.180 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.180 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.906479_212.1", 2022-09-13T19:12:14.181 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.181 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.182 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.182 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.906479Z", 2022-09-13T19:12:14.183 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.183 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.184 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.184 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.185 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:14.185 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.186 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.186 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.187 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.187 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.188 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.188 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.189 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.189 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.189 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.190 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:14.190 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.191 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:14.191 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:14.192 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:14.192 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.193 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:45:22Z" 2022-09-13T19:12:14.193 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:14.194 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:14.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.906479_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.906479Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:14.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.906479_212.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.906479_212.1 2022-09-13T19:12:14.196 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.197 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:14.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:14.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:14.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:27.796+0000 7ff2c5bac600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:27.797+0000 7ff2c5bac600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:14.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:14.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:14.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:14.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:14.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:14.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:14.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:14.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:14.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:14.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:14.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:14.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:14.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:14.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:14.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:14.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:14.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:14.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:45:27Z" 2022-09-13T19:12:14.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:14.238 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:14.238 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:14.239 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:14.239 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:14.240 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.240 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.241 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.241 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:14.242 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.242 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:14.242 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.243 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.244 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.244 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.245 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:14.245 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.246 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.246 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.247 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.248 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.248 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.248 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:14.249 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.249 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.250 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:14.250 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.251 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.251 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.251 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:14.252 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.252 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.253 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.253 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.254 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:14.254 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.255 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.255 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:14.256 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.256 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.257 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.257 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:14.257 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.258 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.258 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.259 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.259 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:14.260 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.260 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.261 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.262 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.262 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.263 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.263 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.263 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.264 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.264 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.265 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:14.265 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.266 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:14.266 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:14.267 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:14.267 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.267 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:45:27Z" 2022-09-13T19:12:14.268 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:14.268 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:14.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:14.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:14.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:14.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:14.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:14.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:32.913+0000 7f56e2cb0600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:32.914+0000 7f56e2cb0600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:14.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:14.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:14.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:14.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:14.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:14.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:14.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:14.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:14.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:14.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:14.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:14.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:14.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:14.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:14.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:14.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:14.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:14.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:45:32Z" 2022-09-13T19:12:14.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:14.305 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:14.305 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:14.306 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:14.306 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:14.307 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.307 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.308 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.308 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:14.309 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.309 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:14.310 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.310 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.310 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.311 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.311 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:14.312 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.312 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.313 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.314 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.314 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.315 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.315 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:14.316 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.316 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.316 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:14.317 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.317 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.318 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.318 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:14.319 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.319 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.319 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.320 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.320 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:14.321 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.321 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.322 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:14.322 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.323 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.323 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.324 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:14.324 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.324 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.325 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.325 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.326 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:14.326 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.327 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.327 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.327 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.328 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.328 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.329 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.329 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.330 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.331 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.331 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:14.331 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.332 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:14.332 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:14.333 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:14.333 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.334 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:45:32Z" 2022-09-13T19:12:14.334 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:14.334 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:14.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:14.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.854520_207.1 2022-09-13T19:12:14.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:14.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:14.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:14.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:38.037+0000 7fa9fe00d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:38.038+0000 7fa9fe00d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:14.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:14.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:14.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:14.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:14.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:14.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:14.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:14.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:14.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:14.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:14.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:14.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:14.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:14.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:14.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:14.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:14.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:14.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:45:38Z" 2022-09-13T19:12:14.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:14.379 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:14.381 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:14.381 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:14.382 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:14.383 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.384 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.385 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.386 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:14.387 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.387 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:14.388 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.389 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.390 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.391 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.392 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:14.393 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.394 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.395 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.396 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.397 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.397 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.398 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:14.399 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.400 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.401 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:14.402 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.403 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.404 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.404 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:14.405 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.406 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.407 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.408 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.409 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:14.410 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.411 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.412 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:14.413 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.413 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.414 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.415 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:14.416 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.417 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.418 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.419 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.420 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:14.421 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.422 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.422 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.423 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.426 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.427 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.428 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.428 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.429 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.430 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.431 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:14.432 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.433 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:14.434 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:14.435 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:14.435 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.436 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:45:38Z" 2022-09-13T19:12:14.437 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:14.437 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:14.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:14.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.990623_217.1 2022-09-13T19:12:14.440 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.441 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:14.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:14.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:14.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:43.209+0000 7efc2c2f9600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:43.211+0000 7efc2c2f9600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:14.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:14.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:14.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:14.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:14.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:14.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:14.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:14.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:14.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:14.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:14.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:14.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:14.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:14.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:14.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:14.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:14.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:14.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:45:43Z" 2022-09-13T19:12:14.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:14.479 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:14.480 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:14.480 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:14.481 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:14.481 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.482 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.482 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.483 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:14.483 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.484 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:14.484 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.485 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.485 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.486 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.486 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:14.487 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.487 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.488 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.488 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.489 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.489 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.490 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:14.490 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.491 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.491 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:14.492 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.492 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.493 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.493 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:14.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.494 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.495 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.495 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.496 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:14.497 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.497 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.498 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:14.498 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.499 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.499 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.500 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:14.501 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.501 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.502 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.502 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.503 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:14.503 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.503 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.504 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.504 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.505 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.506 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.506 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.507 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.507 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.508 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:14.508 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.508 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:14.509 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:14.509 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:14.510 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.510 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:45:43Z" 2022-09-13T19:12:14.511 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:14.511 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:14.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:14.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.900410_211.1 2022-09-13T19:12:14.513 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:14.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:14.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:14.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:48.371+0000 7f65f2588600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:48.372+0000 7f65f2588600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:14.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:14.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:14.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:14.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:14.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:14.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:14.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:14.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:14.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:14.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:14.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:14.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:14.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:14.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:14.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:14.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:14.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:14.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:45:48Z" 2022-09-13T19:12:14.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:14.548 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:14.549 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:14.549 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:14.549 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:14.550 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.550 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.551 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.551 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:14.552 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.552 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:14.553 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.553 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.553 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.554 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.554 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:14.555 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.555 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.556 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.556 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.557 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.557 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.558 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:14.558 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.559 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.559 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:14.560 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.560 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.561 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.561 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:14.561 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.562 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.562 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.563 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.563 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:14.564 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.564 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.564 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:14.565 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.565 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.566 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.566 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:14.567 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.567 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.568 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.568 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.569 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:14.569 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.570 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.570 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.571 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.572 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.572 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.573 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.573 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.574 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.574 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:14.574 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.575 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:14.575 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:14.576 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:14.576 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.577 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:45:48Z" 2022-09-13T19:12:14.577 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:14.578 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:14.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:14.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:14.580 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.580 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:14.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:14.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:14.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:53.499+0000 7f2b106b1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:53.500+0000 7f2b106b1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:14.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:14.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:14.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:14.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:14.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:14.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:14.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:14.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:14.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:14.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:14.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:14.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:14.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:14.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:14.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:14.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:14.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:14.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:45:53Z" 2022-09-13T19:12:14.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:14.619 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:14.619 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:14.619 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:14.635 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:14.638 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.639 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.640 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.640 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:14.641 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.642 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:14.643 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.644 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.646 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.648 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.649 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:14.651 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.652 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.654 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.655 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.656 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.657 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:14.658 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.659 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.660 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:14.661 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.661 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.662 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.663 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:14.664 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.665 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.666 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.667 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.668 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:14.669 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.670 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.671 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:14.672 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.673 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.674 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.675 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:14.675 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.676 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.677 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.678 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.678 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:14.679 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.679 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.680 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.680 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.681 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.681 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.681 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.682 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.683 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.683 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.684 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:14.684 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.685 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:14.685 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:14.685 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:14.686 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.687 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:45:53Z" 2022-09-13T19:12:14.687 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:14.687 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:14.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:14.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055099.070128_219.1 2022-09-13T19:12:14.690 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.690 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:14.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:14.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:14.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:58.648+0000 7f8b5d5d4600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:45:58.649+0000 7f8b5d5d4600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:14.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:14.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:14.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:14.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:14.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:14.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:14.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:14.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:14.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:14.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:14.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:14.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:14.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:14.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:14.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:14.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:14.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:14.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:45:58Z" 2022-09-13T19:12:14.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:14.723 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:14.723 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:14.724 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:14.724 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:14.725 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.725 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.725 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.726 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:14.726 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.727 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:14.727 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.728 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.728 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.729 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:14.729 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.730 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.730 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.731 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.731 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.732 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.732 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:14.732 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.733 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.733 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:14.735 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.735 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.736 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.736 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:14.736 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.737 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.737 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.738 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.738 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:14.739 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.739 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.740 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:14.741 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.741 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.742 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:14.742 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.743 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.743 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.744 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.744 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:14.745 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.745 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.745 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.746 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.746 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.747 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.747 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.748 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.748 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.748 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.749 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:14.750 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.750 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:14.750 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:14.751 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:14.751 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.752 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:45:58Z" 2022-09-13T19:12:14.752 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:14.752 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:14.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:14.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.900410_211.1 2022-09-13T19:12:14.755 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.755 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:14.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:14.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:14.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:03.793+0000 7fa13cad2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:03.794+0000 7fa13cad2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:14.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:14.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:14.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:14.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:14.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:14.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:14.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:14.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:14.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:14.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:14.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.907374_213.1", 2022-09-13T19:12:14.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.907374Z", 2022-09-13T19:12:14.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:14.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:14.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:14.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:14.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:14.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:46:03Z" 2022-09-13T19:12:14.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:14.786 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:14.786 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:14.787 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:14.787 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:14.788 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.788 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.788 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.789 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:14.789 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.790 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:14.790 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.790 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.791 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.791 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.792 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:14.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.793 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.793 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.793 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.794 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.794 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.795 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:14.795 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.795 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.796 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:14.796 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.797 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.797 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.798 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:14.798 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.799 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.799 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.800 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.800 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:14.801 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.801 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.802 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.907374_213.1", 2022-09-13T19:12:14.802 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.802 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.803 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.803 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.907374Z", 2022-09-13T19:12:14.804 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.804 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.804 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.805 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.805 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:14.806 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.806 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.807 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.807 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.808 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.808 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.809 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.809 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.809 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.810 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.810 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:14.811 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.811 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:14.812 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:14.812 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:14.812 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.813 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:46:03Z" 2022-09-13T19:12:14.813 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:14.814 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:14.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.907374_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.907374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:14.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.907374_213.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.907374_213.1 2022-09-13T19:12:14.816 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.816 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:14.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:14.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:14.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:08.925+0000 7f1b0a447600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:08.926+0000 7f1b0a447600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:14.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:14.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:14.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:14.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:14.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:14.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:14.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:14.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:14.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:14.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:14.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:14.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:14.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:14.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:14.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:14.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:14.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:14.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:46:08Z" 2022-09-13T19:12:14.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:14.847 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:14.847 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:14.848 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:14.848 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:14.849 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.849 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.849 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.850 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:14.850 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.851 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:14.851 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.851 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.852 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.852 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.853 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:14.853 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.854 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.854 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.854 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.855 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.855 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.856 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:14.856 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.857 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.857 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:14.857 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.858 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.858 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.859 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:14.859 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.859 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.860 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.860 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.861 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:14.861 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.862 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.862 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:14.862 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.863 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.863 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.864 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:14.864 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.864 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.865 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.865 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.866 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:14.866 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.867 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.867 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.867 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.868 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.868 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.869 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.869 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.869 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.870 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.870 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:14.871 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.871 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:14.872 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:14.872 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:14.872 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.873 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:46:08Z" 2022-09-13T19:12:14.873 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:14.874 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:14.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:14.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:14.876 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.876 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:14.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:14.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:14.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:14.072+0000 7fb3ab8e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:14.075+0000 7fb3ab8e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:14.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:14.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:14.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:14.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:14.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:14.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:14.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:14.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:14.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:14.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:14.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:14.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:14.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:14.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:14.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:14.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:14.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:14.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:46:14Z" 2022-09-13T19:12:14.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:14.907 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:14.907 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:14.908 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:14.908 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:14.909 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.909 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.909 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.910 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:14.910 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.911 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:14.911 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.912 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.912 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.913 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.913 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:14.914 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.914 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.914 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.915 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.915 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.916 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.916 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:14.916 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.917 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.917 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:14.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.918 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.919 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:14.919 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.920 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.920 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.921 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.921 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:14.921 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.922 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.922 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:14.923 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.923 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.924 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.924 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:14.924 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.925 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.925 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.926 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.926 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:14.926 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.927 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.927 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.928 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.929 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.929 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.930 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.930 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.931 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:14.931 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.931 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:14.932 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:14.932 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:14.933 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.933 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:46:14Z" 2022-09-13T19:12:14.934 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:14.934 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:14.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:14.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:14.936 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:14.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:14.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:14.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:19.262+0000 7f354aa51600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:19.263+0000 7f354aa51600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:14.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:14.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:14.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:14.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:14.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:14.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:14.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:14.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:14.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:14.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:14.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:14.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:14.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:14.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:14.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:14.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:14.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:14.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:14.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:14.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:14.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:14.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:14.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:14.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:14.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:14.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:46:19Z" 2022-09-13T19:12:14.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:14.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:14.967 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:14.967 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:14.968 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:14.968 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:14.968 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:14.969 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.969 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.970 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:14.970 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.971 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:14.971 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.971 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.972 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:14.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.973 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:14.973 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.973 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:14.974 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.975 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.975 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.975 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.976 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:14.976 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.977 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.977 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:14.977 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.978 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.978 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.979 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:14.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.980 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.980 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.980 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.981 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:14.981 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.982 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:14.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.983 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.983 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.984 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:14.984 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.985 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.985 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.985 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:14.986 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:14.986 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:14.987 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:14.987 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:14.987 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:14.988 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:14.988 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:14.989 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:14.989 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:14.990 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.990 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:14.990 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:14.991 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.991 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:14.992 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:14.992 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:14.992 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:14.993 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:46:19Z" 2022-09-13T19:12:14.993 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:14.994 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:14.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:14.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:14.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:14.996 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.996 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:14.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:14.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:14.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:14.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:24.403+0000 7f1666650600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:24.404+0000 7f1666650600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:14.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:15.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:15.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:15.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:15.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:15.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:15.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:15.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:15.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:15.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:15.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:15.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:15.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:15.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:15.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:15.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:15.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:15.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:15.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:46:24Z" 2022-09-13T19:12:15.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:15.028 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:15.028 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:15.029 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:15.029 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:15.030 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.030 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.030 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.031 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:15.031 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.032 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:15.032 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.032 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.033 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.033 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.034 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:15.034 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.035 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.035 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.035 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.036 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.036 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.037 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:15.037 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.037 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.038 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:15.038 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.039 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.039 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.040 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:15.040 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.040 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.041 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.041 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.042 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:15.042 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.043 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.043 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:15.043 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.044 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.044 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.045 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:15.045 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.045 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.046 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.046 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.047 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:15.047 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.048 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.048 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.048 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.049 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.049 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.050 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.050 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.051 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.051 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.051 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:15.052 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.052 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:15.053 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:15.053 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:15.053 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.054 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:46:24Z" 2022-09-13T19:12:15.054 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:15.055 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:15.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:15.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.900410_211.1 2022-09-13T19:12:15.057 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.057 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:15.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:15.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:15.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:29.541+0000 7f7212b3d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:29.542+0000 7f7212b3d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:15.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:15.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:15.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:15.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:15.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:15.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:15.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:15.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:15.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:15.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:15.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:15.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:15.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:15.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:15.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:15.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:15.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:15.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:46:29Z" 2022-09-13T19:12:15.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:15.088 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:15.088 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:15.089 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:15.089 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:15.089 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.090 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.090 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.091 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:15.091 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.092 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:15.092 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.092 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.093 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.093 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.094 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:15.094 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.095 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.095 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.095 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.096 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.096 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.097 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:15.097 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.097 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.098 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:15.099 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.099 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.099 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.103 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:15.104 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.105 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.105 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.106 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.107 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:15.108 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.109 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.110 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:15.111 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.113 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.115 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.115 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:15.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.116 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.116 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.117 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.117 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:15.118 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.118 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.119 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.119 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.119 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.120 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.120 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.121 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.121 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.122 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.122 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:15.122 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.123 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:15.123 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:15.124 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:15.124 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.125 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:46:29Z" 2022-09-13T19:12:15.125 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:15.125 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:15.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:15.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:15.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:15.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:15.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:15.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:34.756+0000 7f87fbf5c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:34.757+0000 7f87fbf5c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:15.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:15.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:15.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:15.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:15.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:15.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:15.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:15.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:15.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:15.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:15.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:15.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:15.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:15.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:15.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:15.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:15.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:15.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:46:34Z" 2022-09-13T19:12:15.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:15.159 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:15.159 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:15.159 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:15.160 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:15.160 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.161 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.161 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.162 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:15.162 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.162 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:15.163 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.163 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.164 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.164 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.164 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:15.165 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.165 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.166 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.166 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.167 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.167 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.168 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:15.168 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.169 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.169 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:15.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.170 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.170 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.171 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:15.171 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.171 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.172 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.172 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.173 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:15.173 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.174 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.174 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:15.174 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.175 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.175 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.176 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:15.176 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.176 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.177 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.177 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.178 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:15.178 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.179 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.179 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.179 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.180 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.180 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.181 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.181 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.181 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.182 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.182 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:15.183 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.183 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:15.184 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:15.184 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:15.184 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.185 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:46:34Z" 2022-09-13T19:12:15.185 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:15.186 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:15.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:15.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.858000_209.1 2022-09-13T19:12:15.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:15.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:15.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:15.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:39.892+0000 7fa58782b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:39.893+0000 7fa58782b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:15.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:15.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:15.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:15.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:15.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:15.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:15.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:15.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:15.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:15.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:15.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:15.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:15.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:15.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:15.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:15.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:15.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:15.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:46:39Z" 2022-09-13T19:12:15.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:15.219 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:15.220 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:15.220 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:15.220 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:15.221 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.221 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.222 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.222 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:15.223 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.223 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:15.224 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.224 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.224 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.225 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.225 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:15.226 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.226 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.227 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.227 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.228 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.228 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.229 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:15.229 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.229 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.230 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:15.230 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.231 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.231 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.232 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:15.232 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.233 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.233 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.234 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.234 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:15.235 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.235 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.235 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:15.236 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.236 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.237 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.237 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:15.238 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.238 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.239 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.239 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.240 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:15.240 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.241 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.241 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.241 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.242 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.242 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.243 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.243 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.244 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.244 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.245 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:15.245 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.246 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:15.246 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:15.247 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:15.247 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.247 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:46:39Z" 2022-09-13T19:12:15.248 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:15.248 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:15.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:15.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:15.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:15.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:15.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:15.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:45.008+0000 7fdaf9d4e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:45.009+0000 7fdaf9d4e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:15.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:15.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:15.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:15.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:15.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:15.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:15.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:15.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:15.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:15.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:15.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:15.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:15.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:15.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:15.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:15.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:15.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:15.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:46:45Z" 2022-09-13T19:12:15.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:15.284 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:15.285 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:15.285 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:15.285 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:15.286 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.286 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.287 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.287 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:15.288 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.288 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:15.288 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.289 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.289 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.290 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.290 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:15.291 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.291 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.291 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.292 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.292 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.293 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.293 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:15.293 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.294 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.294 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:15.295 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.295 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.296 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.296 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:15.296 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.297 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.297 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.298 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.298 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:15.299 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.299 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.300 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:15.300 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.301 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.301 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.301 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:15.302 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.302 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.303 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.303 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.304 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:15.304 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.305 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.305 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.306 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.306 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.306 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.307 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.307 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.308 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.308 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.309 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:15.309 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.309 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:15.310 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:15.310 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:15.311 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.311 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:46:45Z" 2022-09-13T19:12:15.311 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:15.312 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:15.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:15.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.858000_209.1 2022-09-13T19:12:15.314 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.315 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:15.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:15.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:15.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:50.147+0000 7f7ebfe4a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:50.148+0000 7f7ebfe4a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:15.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:15.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:15.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:15.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:15.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:15.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:15.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:15.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:15.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:15.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:15.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:15.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:15.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:15.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:15.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:15.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:15.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:15.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:46:50Z" 2022-09-13T19:12:15.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:15.345 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:15.345 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:15.346 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:15.346 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:15.347 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.347 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.347 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.348 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:15.348 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.349 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:15.349 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.350 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.350 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.350 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.351 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:15.351 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.352 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.352 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.353 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.353 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.353 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.354 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:15.354 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.355 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.355 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:15.356 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.356 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.356 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.357 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:15.357 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.358 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.358 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.359 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.359 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:15.359 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.360 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.360 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:15.361 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.361 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.361 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.362 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:15.362 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.363 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.363 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.364 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.364 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:15.364 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.365 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.365 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.366 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.368 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.368 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.369 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:15.369 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.369 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:15.370 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:15.370 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:15.371 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.371 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:46:50Z" 2022-09-13T19:12:15.372 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:15.372 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:15.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:15.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.911619_214.1 2022-09-13T19:12:15.374 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:15.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:15.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:15.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:55.292+0000 7f8d752dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:46:55.294+0000 7f8d752dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:15.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:15.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:15.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:15.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:15.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:15.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:15.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:15.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:15.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:15.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:15.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:15.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:15.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:15.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:15.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:15.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:15.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:15.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:46:55Z" 2022-09-13T19:12:15.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:15.417 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:15.418 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:15.419 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:15.419 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:15.420 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.421 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.422 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.423 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:15.423 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.424 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:15.425 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.426 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.426 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.429 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:15.429 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.430 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.431 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.432 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.432 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.433 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.434 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:15.435 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.436 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.437 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:15.437 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.438 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.439 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.440 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:15.441 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.442 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.442 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.443 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.444 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:15.445 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.446 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.446 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:15.447 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.448 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.449 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.450 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:15.451 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.452 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.452 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.453 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.454 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:15.455 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.455 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.456 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.457 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.458 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.458 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.459 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.460 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.461 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.462 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.462 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:15.463 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.464 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:15.465 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:15.466 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:15.467 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.467 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:46:55Z" 2022-09-13T19:12:15.468 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:15.468 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:15.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:15.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055099.142561_227.1 2022-09-13T19:12:15.471 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.471 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:15.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:15.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:15.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:00.512+0000 7f7882f8c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:00.513+0000 7f7882f8c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:15.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:15.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:15.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:15.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:15.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:15.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:15.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:15.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:15.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:15.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:15.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:15.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:15.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:15.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:15.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:15.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:15.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:15.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:47:00Z" 2022-09-13T19:12:15.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:15.515 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:15.516 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:15.516 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:15.517 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:15.518 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.518 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.519 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.520 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:15.521 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.522 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:15.523 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.523 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.524 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.525 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.526 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:15.527 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.527 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.528 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.529 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.530 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.531 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.532 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:15.532 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.533 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.534 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:15.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.536 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.537 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.538 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:15.539 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.540 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.540 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.541 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.542 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:15.543 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.544 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.545 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:15.546 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.546 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.547 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.548 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:15.549 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.550 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.551 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.551 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.553 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:15.554 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.555 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.556 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.556 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.557 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.558 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.559 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.560 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.560 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.561 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.562 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:15.562 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.563 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:15.564 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:15.565 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:15.566 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.566 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:47:00Z" 2022-09-13T19:12:15.567 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:15.568 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:15.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:15.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:15.572 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:15.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:15.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:15.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:05.660+0000 7f8a185a7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:05.661+0000 7f8a185a7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:15.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:15.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:15.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:15.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:15.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:15.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:15.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:15.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:15.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:15.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:15.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:15.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:15.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:15.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:15.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:15.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:15.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:15.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:47:05Z" 2022-09-13T19:12:15.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:15.614 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:15.614 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:15.615 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:15.615 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:15.615 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.616 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.616 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.617 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:15.618 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.618 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:15.619 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.619 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.619 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.620 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.620 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:15.621 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.621 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.622 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.622 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.623 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.623 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.623 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:15.624 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.624 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.625 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:15.625 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.626 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.626 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.627 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:15.627 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.627 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.628 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.628 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.629 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:15.629 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.630 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.630 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:15.631 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.631 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.631 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.632 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:15.632 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.633 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.633 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.634 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.634 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:15.634 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.635 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.635 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.636 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.636 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.637 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.637 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.638 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.638 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.638 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.639 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:15.639 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.640 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:15.640 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:15.641 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:15.641 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.641 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:47:05Z" 2022-09-13T19:12:15.642 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:15.642 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:15.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:15.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:15.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:15.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:15.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:15.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:10.822+0000 7f383f8e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:10.823+0000 7f383f8e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:15.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:15.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:15.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:15.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:15.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:15.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:15.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:15.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:15.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:15.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:15.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:15.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:15.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:15.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:15.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:15.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:15.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:15.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:47:10Z" 2022-09-13T19:12:15.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:15.677 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:15.678 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:15.678 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:15.679 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:15.679 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.680 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.680 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.681 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:15.681 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.682 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:15.682 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.683 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.683 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.684 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.684 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:15.685 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.685 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.686 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.687 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.687 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.688 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:15.688 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.689 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.689 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:15.690 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.690 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.691 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.691 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:15.692 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.692 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.693 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.693 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.694 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:15.694 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.695 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.695 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:15.696 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.696 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.697 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.697 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:15.698 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.698 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.699 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.699 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.700 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:15.700 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.701 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.701 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.702 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.702 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.703 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.703 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.704 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.704 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.705 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.705 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:15.706 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.707 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:15.707 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:15.708 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:15.708 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.709 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:47:10Z" 2022-09-13T19:12:15.709 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:15.709 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:15.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:15.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.858000_209.1 2022-09-13T19:12:15.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:15.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:15.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:15.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:15.966+0000 7ff695ffb600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:15.967+0000 7ff695ffb600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:15.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:15.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:15.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:15.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:15.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:15.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:15.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:15.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:15.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:15.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:15.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:15.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:15.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:15.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:15.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:15.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:15.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:15.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:47:15Z" 2022-09-13T19:12:15.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:15.744 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:15.745 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:15.745 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:15.746 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:15.746 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.746 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.747 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.747 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:15.748 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.748 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:15.749 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.749 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.750 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.750 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:15.751 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.752 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.752 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.753 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.753 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.753 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:15.754 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.754 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.755 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:15.755 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.756 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.756 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.757 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:15.757 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.757 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.758 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.758 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.759 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:15.759 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.760 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.760 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:15.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.763 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:15.763 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.764 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.765 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.765 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.766 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:15.767 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.767 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.768 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.769 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.771 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.772 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.773 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.774 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:15.774 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.775 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:15.775 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:15.776 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:15.777 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.777 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:47:15Z" 2022-09-13T19:12:15.778 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:15.778 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:15.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:15.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.900410_211.1 2022-09-13T19:12:15.782 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.782 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:15.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:15.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:15.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:21.122+0000 7f23251aa600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:21.123+0000 7f23251aa600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:15.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:15.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:15.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:15.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:15.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:15.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:15.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:15.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:15.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:15.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:15.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:15.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:15.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:15.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:15.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:15.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:15.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:15.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:47:21Z" 2022-09-13T19:12:15.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:15.813 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:15.814 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:15.814 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:15.815 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:15.815 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.815 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.816 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.816 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:15.817 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.817 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:15.817 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.818 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.818 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.819 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.819 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:15.820 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.820 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.821 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.821 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.821 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.823 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.823 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:15.824 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.824 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.826 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:15.826 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.827 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.827 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.827 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:15.828 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.829 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.829 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.830 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.830 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:15.830 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.831 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.831 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:15.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.832 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:15.834 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.834 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.834 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.835 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.835 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:15.836 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.836 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.837 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.837 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.837 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.838 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.839 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.839 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.839 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.840 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.840 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:15.841 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.841 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:15.841 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:15.842 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:15.842 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.843 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:47:21Z" 2022-09-13T19:12:15.843 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:15.844 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:15.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:15.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:15.846 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.846 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:15.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:15.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:15.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:26.245+0000 7fa3b4a6b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:26.247+0000 7fa3b4a6b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:15.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:15.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:15.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:15.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:15.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:15.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:15.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:15.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:15.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:15.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:15.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:15.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:15.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:15.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:15.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:15.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:15.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:15.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:47:26Z" 2022-09-13T19:12:15.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:15.883 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:15.884 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:15.884 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:15.885 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:15.885 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.887 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.888 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:15.888 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.889 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:15.889 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.890 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.890 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.890 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.891 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:15.891 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.892 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.892 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.893 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.894 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.894 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.895 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:15.895 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.895 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.896 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:15.896 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.897 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.897 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:15.898 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.898 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.899 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.899 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.900 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:15.900 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.901 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.901 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:15.902 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.902 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.903 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.903 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:15.904 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.904 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.905 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.905 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.906 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:15.906 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.907 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.907 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.907 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.908 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.908 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.909 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.909 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.910 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.910 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.910 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:15.911 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.912 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:15.912 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:15.913 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:15.913 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.913 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:47:26Z" 2022-09-13T19:12:15.914 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:15.914 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:15.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:15.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.861945_210.1 2022-09-13T19:12:15.916 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.917 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:15.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:15.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:15.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:31.387+0000 7f1ff068d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:31.388+0000 7f1ff068d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:15.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:15.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:15.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:15.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:15.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:15.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:15.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:15.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:15.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:15.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:15.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:15.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:15.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:15.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:15.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:15.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:15.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:15.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:15.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:15.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:15.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:15.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:15.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:15.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:15.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:15.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:15.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:47:31Z" 2022-09-13T19:12:15.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:15.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:15.951 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:15.951 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:15.952 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:15.952 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:15.952 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:15.953 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.954 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.955 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:15.955 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.956 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:15.956 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.957 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.957 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:15.958 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.958 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:15.959 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.959 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:15.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.960 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.960 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.961 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.962 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:15.963 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.963 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.964 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:15.964 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.965 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.966 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.966 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:15.966 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.967 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.967 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.968 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.968 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:15.969 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.969 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.969 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:15.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.971 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.972 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.972 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:15.973 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.974 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.975 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.976 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:15.976 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:15.977 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:15.978 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:15.979 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:15.980 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:15.980 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:15.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:15.982 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:15.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:15.984 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.985 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:15.986 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:15.987 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.987 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:15.988 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:15.989 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:15.989 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:15.990 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:47:31Z" 2022-09-13T19:12:15.991 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:15.991 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:15.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:15.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:15.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.900410_211.1 2022-09-13T19:12:15.995 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.996 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:15.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:15.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:15.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:15.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:36.522+0000 7f03318e3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:36.523+0000 7f03318e3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:16.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:16.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:16.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:16.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:16.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:16.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:16.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:16.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:16.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:16.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:16.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.186446_230.1", 2022-09-13T19:12:16.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.186446Z", 2022-09-13T19:12:16.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:16.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:16.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:16.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:16.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:16.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:47:36Z" 2022-09-13T19:12:16.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:16.042 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:16.043 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:16.044 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:16.044 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:16.045 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.046 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.047 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.047 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:16.048 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.049 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:16.050 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.051 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.051 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.052 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.053 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:16.054 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.055 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.055 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.056 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.056 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.056 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.057 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:16.057 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.058 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.058 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:16.059 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.059 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.060 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.060 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:16.060 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.061 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.062 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.062 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.063 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:16.063 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.064 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.186446_230.1", 2022-09-13T19:12:16.065 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.066 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.067 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.067 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.186446Z", 2022-09-13T19:12:16.068 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.069 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.070 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.070 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.071 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:16.072 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.072 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.073 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.073 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.074 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.076 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.077 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.078 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.079 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:16.079 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.080 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:16.081 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:16.082 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:16.082 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.083 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:47:36Z" 2022-09-13T19:12:16.084 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:16.084 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:16.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.186446_230.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.186446Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:16.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.186446_230.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055099.186446_230.1 2022-09-13T19:12:16.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:16.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:16.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:16.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:41.660+0000 7f0cc516c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:41.661+0000 7f0cc516c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:16.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:16.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:16.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:16.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:16.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:16.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:16.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:16.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:16.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:16.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:16.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:16.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:16.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:16.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:16.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:16.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:16.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:16.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:47:41Z" 2022-09-13T19:12:16.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:16.146 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:16.146 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:16.148 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:16.149 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:16.149 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.150 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.151 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.151 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:16.152 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.153 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:16.153 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.154 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.155 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.156 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.156 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:16.157 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.157 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.158 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.158 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.159 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.159 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.160 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:16.161 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.162 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.162 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:16.163 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.164 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.165 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.166 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:16.166 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.167 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.167 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.168 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.168 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:16.169 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.170 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.171 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:16.172 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.172 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.173 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.174 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:16.174 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.175 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.176 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.177 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.178 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:16.178 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.178 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.179 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.179 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.180 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.180 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.181 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.181 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.182 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.182 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.183 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:16.183 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.184 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:16.184 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:16.185 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:16.185 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.185 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:47:41Z" 2022-09-13T19:12:16.186 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:16.186 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:16.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:16.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055099.220845_233.1 2022-09-13T19:12:16.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:16.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:16.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:16.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:46.840+0000 7fe618480600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:46.841+0000 7fe618480600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:16.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:16.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:16.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:16.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:16.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:16.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:16.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:16.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:16.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:16.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:16.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.907374_213.1", 2022-09-13T19:12:16.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.907374Z", 2022-09-13T19:12:16.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:16.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:16.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:16.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:16.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:16.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:47:46Z" 2022-09-13T19:12:16.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:16.241 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:16.242 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:16.243 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:16.243 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:16.245 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.245 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.246 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.247 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:16.247 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.248 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:16.249 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.249 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.250 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.252 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:16.252 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.254 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.254 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.255 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.256 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.256 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.257 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:16.257 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.257 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.258 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:16.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.261 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:16.262 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.263 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.264 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.265 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.265 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:16.266 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.266 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.267 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.907374_213.1", 2022-09-13T19:12:16.267 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.267 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.268 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.268 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.907374Z", 2022-09-13T19:12:16.269 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.269 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.270 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.270 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.271 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:16.271 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.272 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.272 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.273 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.273 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.273 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.274 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.274 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.275 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.275 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.276 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:16.276 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.276 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:16.277 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:16.277 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:16.278 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.278 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:47:46Z" 2022-09-13T19:12:16.279 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:16.279 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:16.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.907374_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.907374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:16.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.907374_213.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.907374_213.1 2022-09-13T19:12:16.281 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.282 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:16.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:16.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:16.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:51.986+0000 7fea2d1a4600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:51.987+0000 7fea2d1a4600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:16.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:16.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:16.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:16.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:16.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:16.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:16.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:16.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:16.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:16.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:16.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:16.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:16.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:16.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:16.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:16.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:16.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:16.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:47:51Z" 2022-09-13T19:12:16.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:16.314 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:16.314 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:16.315 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:16.315 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:16.316 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.316 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.316 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.317 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:16.317 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.318 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:16.318 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.319 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.319 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.320 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:16.320 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.321 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.321 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.321 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.322 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.322 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.323 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:16.323 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.324 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.324 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:16.324 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.325 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.325 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.326 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:16.326 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.327 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.327 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.327 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.328 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:16.328 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.329 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.329 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:16.330 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.330 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.331 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.331 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:16.332 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.332 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.333 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.333 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.333 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:16.334 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.334 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.335 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.335 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.336 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.336 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.336 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.337 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.337 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.338 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.338 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:16.339 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.339 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:16.339 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:16.340 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:16.340 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.341 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:47:51Z" 2022-09-13T19:12:16.341 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:16.342 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:16.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:16.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.990623_217.1 2022-09-13T19:12:16.344 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.344 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:16.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:16.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:16.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:57.127+0000 7f0a5a096600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:47:57.128+0000 7f0a5a096600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:16.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:16.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:16.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:16.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:16.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:16.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:16.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:16.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:16.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:16.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:16.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:16.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:16.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:16.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:16.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:16.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:16.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:16.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:47:57Z" 2022-09-13T19:12:16.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:16.375 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:16.376 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:16.376 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:16.377 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:16.377 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.378 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.378 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.379 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:16.379 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.379 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:16.380 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.381 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.381 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.381 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.382 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:16.382 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.383 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.383 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.384 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.384 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.384 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.385 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:16.385 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.386 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.386 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:16.387 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.387 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.387 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.388 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:16.388 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.389 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.389 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.390 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.390 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:16.390 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.391 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.391 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:16.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.392 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.393 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.393 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:16.394 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.394 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.395 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.395 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.396 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:16.396 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.397 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.397 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.397 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.398 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.398 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.399 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.399 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.400 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.400 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.401 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:16.401 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.401 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:16.402 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:16.402 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:16.403 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.403 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:47:57Z" 2022-09-13T19:12:16.404 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:16.404 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:16.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:16.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:16.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:16.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:16.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:16.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:02.261+0000 7f421a9ef600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:02.262+0000 7f421a9ef600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:16.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:16.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:16.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:16.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:16.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:16.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:16.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:16.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:16.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:16.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:16.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:16.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:16.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:16.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:16.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:16.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:16.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:16.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:48:02Z" 2022-09-13T19:12:16.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:16.439 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:16.439 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:16.440 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:16.440 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:16.441 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.441 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.442 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.442 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:16.442 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.443 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:16.443 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.444 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.444 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.445 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.445 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:16.445 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.446 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.446 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.447 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.447 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.448 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.448 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:16.448 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.449 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.449 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:16.450 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.450 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.451 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.451 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:16.451 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.452 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.452 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.453 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.453 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:16.454 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.454 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.454 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:16.455 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.455 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.456 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.456 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:16.457 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.457 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.457 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.458 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.458 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:16.459 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.459 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.460 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.460 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.460 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.461 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.461 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.462 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.462 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.463 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.463 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:16.463 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.464 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:16.464 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:16.465 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:16.465 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.466 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:48:02Z" 2022-09-13T19:12:16.466 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:16.467 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:16.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:16.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.854520_207.1 2022-09-13T19:12:16.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:16.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:16.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:16.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:07.390+0000 7f2eb7923600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:07.391+0000 7f2eb7923600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:16.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:16.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:16.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:16.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:16.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:16.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:16.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:16.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:16.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:16.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:16.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:16.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:16.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:16.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:16.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:16.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:16.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:16.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:48:07Z" 2022-09-13T19:12:16.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:16.519 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:16.519 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:16.520 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:16.521 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:16.522 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.523 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.523 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.524 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:16.525 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.526 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:16.526 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.527 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.528 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.529 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.529 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:16.530 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.531 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.533 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.534 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.534 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.535 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.535 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:16.536 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.536 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.537 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:16.537 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.538 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.538 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.539 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:16.539 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.540 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.541 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.542 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.543 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:16.543 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.544 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.545 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:16.546 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.547 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.547 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.548 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:16.549 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.550 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.550 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.551 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.551 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:16.552 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.552 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.553 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.553 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.554 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.554 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.555 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.556 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.556 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.557 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:16.557 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.557 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:16.558 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:16.558 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:16.559 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.560 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:48:07Z" 2022-09-13T19:12:16.560 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:16.561 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:16.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:16.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.858000_209.1 2022-09-13T19:12:16.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.565 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:16.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:16.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:16.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:12.525+0000 7f7449f5e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:12.526+0000 7f7449f5e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:16.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:16.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:16.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:16.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:16.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:16.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:16.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:16.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:16.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:16.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:16.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:16.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:16.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:16.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:16.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:16.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:16.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:16.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:48:12Z" 2022-09-13T19:12:16.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:16.615 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:16.616 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:16.617 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:16.617 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:16.618 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.619 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.619 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.620 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:16.621 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.621 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:16.622 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.622 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.623 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.623 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.623 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:16.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.624 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.625 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.625 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.626 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.626 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.626 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:16.627 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.627 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.628 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:16.628 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.629 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.629 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.629 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:16.630 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.630 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.631 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.631 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.632 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:16.632 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.632 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.633 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:16.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.634 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.634 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.635 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:16.635 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.635 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.636 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.636 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.637 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:16.637 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.637 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.638 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.638 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.639 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.639 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.640 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.640 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.640 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.641 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.641 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:16.642 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.642 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:16.643 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:16.643 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:16.643 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.644 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:48:12Z" 2022-09-13T19:12:16.644 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:16.645 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:16.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:16.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.854520_207.1 2022-09-13T19:12:16.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:16.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:16.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:16.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:17.675+0000 7f56b402a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:17.676+0000 7f56b402a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:16.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:16.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:16.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:16.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:16.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:16.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:16.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:16.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:16.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:16.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:16.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:16.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:16.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:16.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:16.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:16.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:16.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:16.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:48:17Z" 2022-09-13T19:12:16.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:16.687 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:16.688 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:16.688 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:16.689 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:16.689 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.690 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.690 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.690 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:16.691 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.691 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:16.692 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.692 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.693 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.693 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.693 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:16.694 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.694 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.695 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.695 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.696 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.696 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.696 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:16.697 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.697 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.698 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:16.698 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.698 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.699 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.699 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:16.700 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.700 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.701 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.701 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.701 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:16.702 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.702 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.703 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:16.703 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.704 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.704 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.704 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:16.705 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.705 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.706 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.706 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.707 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:16.707 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.707 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.708 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.708 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.709 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.709 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.709 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.710 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.710 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.711 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.711 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:16.712 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.712 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:16.712 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:16.713 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:16.713 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.714 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:48:17Z" 2022-09-13T19:12:16.714 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:16.715 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:16.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:16.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.854520_207.1 2022-09-13T19:12:16.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:16.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:16.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:16.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:22.821+0000 7f1e58aa0600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:22.822+0000 7f1e58aa0600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:16.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:16.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:16.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:16.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:16.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:16.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:16.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:16.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:16.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:16.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:16.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:16.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:16.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:16.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:16.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:16.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:16.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:16.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:48:22Z" 2022-09-13T19:12:16.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:16.748 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:16.749 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:16.749 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:16.749 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:16.750 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.750 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.751 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.751 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:16.751 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.752 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:16.752 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.753 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.753 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.754 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.754 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:16.754 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.755 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.755 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.756 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.756 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.757 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.757 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:16.757 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.758 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.759 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:16.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.760 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.760 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.760 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:16.761 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.761 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.762 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.762 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.763 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:16.763 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.763 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.764 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:16.764 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.765 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.765 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.765 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:16.766 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.766 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.767 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.767 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.768 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:16.768 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.768 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.769 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.770 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.771 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.771 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.772 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.772 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:16.773 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.773 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:16.774 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:16.774 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:16.775 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.775 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:48:22Z" 2022-09-13T19:12:16.775 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:16.776 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:16.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:16.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:16.778 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.778 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:16.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:16.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:16.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:27.975+0000 7f3055548600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:27.976+0000 7f3055548600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:16.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:16.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:16.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:16.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:16.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:16.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:16.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:16.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:16.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:16.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:16.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:16.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:16.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:16.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:16.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:16.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:16.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:16.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:48:27Z" 2022-09-13T19:12:16.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:16.809 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:16.810 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:16.810 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:16.811 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:16.811 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.811 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.812 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.812 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:16.813 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.813 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:16.814 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.814 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.814 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.815 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.815 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:16.816 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.816 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.817 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.817 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.817 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.818 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.818 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:16.819 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.819 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.819 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:16.820 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.820 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.821 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.821 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:16.822 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.822 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.822 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.823 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.823 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:16.824 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.824 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.825 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:16.825 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.825 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.826 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.826 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:16.827 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.827 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.827 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.828 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.828 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:16.829 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.829 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.830 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.830 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.830 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.831 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.831 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.832 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.832 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.833 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.833 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:16.833 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.834 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:16.834 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:16.835 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:16.835 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.836 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:48:27Z" 2022-09-13T19:12:16.836 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:16.836 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:16.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:16.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.953175_216.1 2022-09-13T19:12:16.838 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:16.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:16.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:16.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:33.117+0000 7f4880825600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:33.119+0000 7f4880825600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:16.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:16.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:16.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:16.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:16.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:16.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:16.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:16.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:16.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:16.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:16.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:16.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:16.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:16.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:16.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:16.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:16.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:16.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:48:33Z" 2022-09-13T19:12:16.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:16.870 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:16.870 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:16.871 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:16.871 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:16.871 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.872 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.872 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.873 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:16.873 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.874 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:16.874 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.874 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.875 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.875 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.876 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:16.876 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.877 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.877 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.877 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.878 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.878 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.879 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:16.879 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.879 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.880 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:16.880 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.881 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.881 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.882 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:16.882 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.882 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.883 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.883 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.884 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:16.884 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.885 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.885 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:16.885 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.886 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.886 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.887 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:16.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.888 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.888 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.888 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.889 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:16.889 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.890 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.890 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.891 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.891 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.891 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.892 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.892 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.893 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.893 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.893 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:16.894 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.894 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:16.895 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:16.895 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:16.896 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.896 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:48:33Z" 2022-09-13T19:12:16.896 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:16.897 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:16.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:16.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:16.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.900 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:16.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:16.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:16.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:38.283+0000 7ffbca1b6600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:38.293+0000 7ffbca1b6600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:16.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:16.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:16.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:16.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:16.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:16.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:16.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:16.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:16.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:16.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:16.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:16.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:16.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:16.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:16.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:16.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:16.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:16.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:48:38Z" 2022-09-13T19:12:16.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:16.930 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:16.930 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:16.931 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:16.931 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:16.932 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.932 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.933 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.933 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:16.933 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.934 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:16.934 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.935 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.935 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.936 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.936 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:16.936 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.937 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.938 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.938 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.939 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.939 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:16.939 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.940 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.940 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:16.941 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.941 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.941 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.942 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:16.942 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.943 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.943 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.944 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.944 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:16.945 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.945 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.946 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:16.946 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.947 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.947 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.948 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:16.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.948 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.949 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.949 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:16.950 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:16.950 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:16.951 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:16.951 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.952 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:16.952 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:16.953 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:16.953 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.954 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.954 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.954 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:16.955 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:16.955 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.956 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:16.956 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:16.957 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:16.957 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:16.958 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:48:38Z" 2022-09-13T19:12:16.958 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:16.959 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:16.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:16.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:16.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055099.220845_233.1 2022-09-13T19:12:16.961 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.961 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:16.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:16.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:16.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:16.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:43.480+0000 7f5f745ed600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:43.481+0000 7f5f745ed600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:16.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:16.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:16.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:16.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:16.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:16.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:16.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:16.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:16.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:16.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:16.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:16.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:16.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:16.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:16.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:16.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:16.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:16.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:16.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:16.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:16.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:16.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:16.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:16.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:16.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:16.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:16.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:16.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:16.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:16.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:16.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:48:43Z" 2022-09-13T19:12:16.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:16.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:16.996 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:16.997 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:16.997 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:16.998 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:16.998 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:16.999 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:16.999 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.000 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.000 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.000 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.001 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.001 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.002 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.002 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.003 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.004 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.004 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.004 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.005 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.005 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.006 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.006 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.007 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.007 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.007 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.008 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.008 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.009 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.009 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.010 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.010 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.011 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.011 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.012 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.012 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:17.012 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.013 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.013 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.014 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:17.014 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.015 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.015 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.015 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.016 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.016 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.017 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.017 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.017 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.018 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.018 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.019 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.019 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.020 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.020 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.020 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.021 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.021 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.022 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.022 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.023 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.023 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:48:43Z" 2022-09-13T19:12:17.023 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.024 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.861945_210.1 2022-09-13T19:12:17.026 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.026 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:48.624+0000 7f690b7d0600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:48.625+0000 7f690b7d0600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:17.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:17.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:48:48Z" 2022-09-13T19:12:17.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.057 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.057 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.058 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.058 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.059 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.059 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.060 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.060 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.060 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.061 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.061 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.062 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.062 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.064 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.066 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.066 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.066 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.067 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.067 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.068 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.068 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.069 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.069 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.069 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.070 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.070 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.071 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.071 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.072 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.072 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.072 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.073 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.073 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:17.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.074 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.075 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:17.075 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.076 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.076 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.077 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.077 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.077 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.078 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.078 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.079 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.079 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.080 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.080 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.080 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.081 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.081 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.082 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.082 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.083 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.083 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.083 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.084 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.084 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:48:48Z" 2022-09-13T19:12:17.085 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.085 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:17.087 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:53.758+0000 7f308a3ce600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:53.760+0000 7f308a3ce600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:17.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:17.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:48:53Z" 2022-09-13T19:12:17.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.119 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.119 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.119 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.120 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.120 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.121 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.121 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.122 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.122 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.122 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.123 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.123 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.124 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.124 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.125 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.125 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.125 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.126 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.126 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.127 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.127 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.128 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.128 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.128 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.129 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.129 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.130 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.130 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.131 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.131 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.131 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.132 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.132 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.133 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.133 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.134 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.134 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:17.135 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.135 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.136 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.136 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:17.137 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.137 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.137 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.138 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.138 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.139 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.139 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.140 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.140 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.141 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.141 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.141 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.142 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.142 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.143 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.143 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.144 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.144 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.145 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.145 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.146 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.146 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:48:53Z" 2022-09-13T19:12:17.147 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.147 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.953175_216.1 2022-09-13T19:12:17.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:58.892+0000 7f09f4516600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:48:58.893+0000 7f09f4516600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:17.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:17.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:48:58Z" 2022-09-13T19:12:17.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.183 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.184 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.184 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.185 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.185 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.185 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.186 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.186 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.187 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.187 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.188 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.188 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.188 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.189 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.189 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.190 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.190 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.191 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.191 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.192 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.192 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.193 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.193 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.193 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.194 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.194 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.195 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.195 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.196 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.196 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.196 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.197 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.197 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.198 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.198 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.199 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.199 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:17.200 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.200 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.200 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.201 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:17.201 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.202 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.202 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.203 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.203 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.204 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.204 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.204 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.205 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.206 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.206 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.206 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.207 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.207 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.208 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.208 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.208 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.209 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.209 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.210 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.210 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.211 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:48:58Z" 2022-09-13T19:12:17.211 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.211 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.858000_209.1 2022-09-13T19:12:17.214 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.214 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:04.035+0000 7ff0f65f3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:04.036+0000 7ff0f65f3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:17.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:17.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:49:04Z" 2022-09-13T19:12:17.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.246 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.246 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.247 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.247 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.248 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.248 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.248 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.249 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.249 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.250 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.250 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.251 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.252 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.252 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.252 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.253 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.253 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.254 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.254 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.255 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.255 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.255 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.256 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.256 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.257 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.257 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.258 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.258 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.258 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.259 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.259 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.260 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.260 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.261 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.261 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.262 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.262 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:17.263 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.263 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.263 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.264 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:17.264 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.265 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.265 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.265 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.266 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.266 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.267 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.267 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.267 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.268 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.269 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.269 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.269 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.270 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.270 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.271 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.271 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.272 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.272 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.272 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.273 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.273 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:49:04Z" 2022-09-13T19:12:17.274 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.274 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:17.276 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:09.206+0000 7fc17fa67600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:09.207+0000 7fc17fa67600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:17.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:17.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:49:09Z" 2022-09-13T19:12:17.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.308 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.308 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.309 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.309 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.310 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.310 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.311 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.311 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.311 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.312 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.312 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.313 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.313 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.314 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.314 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.315 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.315 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.315 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.316 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.316 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.317 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.317 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.317 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.318 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.318 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.319 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.320 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.320 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.321 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.321 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.322 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.322 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.322 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.323 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.323 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:17.324 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.324 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.325 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.325 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:17.325 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.326 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.326 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.327 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.327 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.328 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.328 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.329 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.329 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.330 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.330 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.330 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.331 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.331 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.332 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.332 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.333 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.333 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.333 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.334 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.334 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.335 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:49:09Z" 2022-09-13T19:12:17.335 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.335 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.857076_208.1 2022-09-13T19:12:17.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:14.357+0000 7fc5d2a7a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:14.358+0000 7fc5d2a7a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:17.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:17.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:49:14Z" 2022-09-13T19:12:17.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.370 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.370 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.371 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.371 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.372 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.372 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.372 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.373 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.373 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.374 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.374 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.375 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.375 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.376 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.376 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.376 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.377 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.377 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.378 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.378 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.379 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.379 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.379 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.380 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.380 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.381 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.381 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.381 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.382 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.382 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.383 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.383 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.384 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.384 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.384 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.385 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.385 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:17.386 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.386 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.386 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.387 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:17.387 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.388 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.388 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.388 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.389 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.389 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.390 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.390 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.391 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.391 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.391 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.392 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.392 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.393 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.393 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.393 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.394 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.394 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.395 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.395 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.396 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.396 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:49:14Z" 2022-09-13T19:12:17.396 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.397 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:17.399 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.399 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:19.516+0000 7ffa8a41c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:19.517+0000 7ffa8a41c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:17.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:17.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:49:19Z" 2022-09-13T19:12:17.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.429 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.429 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.429 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.430 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.430 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.431 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.431 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.431 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.432 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.432 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.433 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.433 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.433 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.434 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.434 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.435 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.435 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.435 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.436 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.436 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.437 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.437 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.438 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.438 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.438 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.439 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.439 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.440 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.440 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.441 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.441 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.441 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.442 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.442 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.443 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.443 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.443 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:17.444 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.444 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.445 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.445 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:17.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.446 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.446 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.447 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.447 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.448 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.448 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.449 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.449 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.449 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.450 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.450 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.451 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.451 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.451 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.452 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.452 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.453 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.453 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.453 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.454 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.454 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:49:19Z" 2022-09-13T19:12:17.455 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.455 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.857076_208.1 2022-09-13T19:12:17.457 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.458 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:24.649+0000 7fcbc9600600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:24.651+0000 7fcbc9600600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:17.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:17.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:49:24Z" 2022-09-13T19:12:17.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.490 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.491 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.491 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.492 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.492 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.492 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.493 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.493 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.494 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.494 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.495 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.495 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.495 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.496 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.496 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.497 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.497 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.498 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.498 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.499 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.499 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.500 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.500 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.500 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.501 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.501 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.502 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.502 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.503 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.503 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.503 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.504 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.504 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.505 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.505 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.505 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.506 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:17.506 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.507 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.507 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.508 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:17.508 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.509 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.509 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.510 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.510 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.510 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.511 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.511 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.512 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.512 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.512 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.513 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.513 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.514 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.514 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.515 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.515 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.515 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.516 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.516 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.517 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.518 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:49:24Z" 2022-09-13T19:12:17.518 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.519 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:17.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:29.849+0000 7fbe18ea3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:29.850+0000 7fbe18ea3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:17.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:17.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:49:29Z" 2022-09-13T19:12:17.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.553 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.553 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.554 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.554 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.555 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.556 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.556 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.557 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.557 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.558 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.558 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.558 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.559 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.559 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.560 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.560 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.561 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.561 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.561 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.562 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.562 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.563 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.563 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.564 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.565 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.565 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.566 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.566 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.567 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.567 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.568 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.568 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.569 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.569 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.570 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.570 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:17.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.571 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.572 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.572 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:17.573 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.573 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.573 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.574 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.574 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.575 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.575 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.576 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.577 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.577 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.578 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.578 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.579 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.579 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.580 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.580 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.581 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.581 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.581 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.582 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.582 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.583 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:49:29Z" 2022-09-13T19:12:17.583 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.584 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.911619_214.1 2022-09-13T19:12:17.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:35.025+0000 7fef12810600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:35.026+0000 7fef12810600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:17.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:17.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:49:35Z" 2022-09-13T19:12:17.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.619 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.619 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.620 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.620 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.621 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.621 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.621 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.622 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.622 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.623 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.623 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.624 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.624 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.624 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.625 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.625 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.626 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.626 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.627 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.627 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.627 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.628 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.628 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.629 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.629 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.630 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.630 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.630 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.631 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.631 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.632 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.632 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.633 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.633 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.633 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.634 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.634 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:17.635 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.635 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.636 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.636 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:17.636 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.637 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.637 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.638 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.638 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.639 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.639 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.640 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.640 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.640 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.641 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.641 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.642 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.642 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.642 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.643 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.643 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.644 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.644 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.645 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.645 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.645 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:49:35Z" 2022-09-13T19:12:17.646 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.646 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.858000_209.1 2022-09-13T19:12:17.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:40.160+0000 7f15e9a4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:40.161+0000 7f15e9a4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:17.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:17.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:49:40Z" 2022-09-13T19:12:17.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.683 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.684 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.685 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.685 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.686 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.687 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.687 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.687 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.688 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.688 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.689 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.689 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.689 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.690 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.690 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.691 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.691 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.692 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.692 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.693 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.693 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.693 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.694 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.694 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.695 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.695 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.696 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.696 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.696 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.697 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.697 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.698 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.698 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.698 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.699 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.699 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:17.700 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.700 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.701 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:17.702 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.702 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.703 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.703 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.703 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.704 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.704 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.705 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.705 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.705 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.706 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.706 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.707 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.707 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.708 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.708 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.709 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.709 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.709 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.710 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.710 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.711 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:49:40Z" 2022-09-13T19:12:17.711 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.712 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:17.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:45.297+0000 7f91145ac600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:45.298+0000 7f91145ac600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:17.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:17.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:49:45Z" 2022-09-13T19:12:17.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.745 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.745 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.746 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.746 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.746 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.747 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.747 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.748 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.748 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.749 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.749 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.749 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.750 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.751 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.751 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.752 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.752 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.753 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.753 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.753 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.754 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.754 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.755 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.755 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.756 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.756 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.757 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.757 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.757 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.758 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.758 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.759 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.759 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.759 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.760 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:17.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.761 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:17.762 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.762 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.763 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.763 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.764 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.764 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.765 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.765 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.765 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.766 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.766 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.767 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.767 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.768 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.768 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.768 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.769 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.769 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.770 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.770 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.770 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.771 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:49:45Z" 2022-09-13T19:12:17.771 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.772 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.900410_211.1 2022-09-13T19:12:17.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:50.481+0000 7f5257cfc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:50.482+0000 7f5257cfc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:17.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:17.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:49:50Z" 2022-09-13T19:12:17.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.804 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.805 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.805 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.805 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.806 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.806 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.807 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.807 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.808 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.808 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.809 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.809 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.810 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.810 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.810 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.811 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.811 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.812 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.812 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.812 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.813 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.813 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.814 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.814 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.814 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.815 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.815 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.816 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.816 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.816 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.817 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.817 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.818 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.818 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.819 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.819 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.819 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:17.820 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.820 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.821 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.821 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:17.821 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.822 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.822 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.823 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.823 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.823 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.824 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.824 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.825 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.825 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.825 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.826 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.826 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.827 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.827 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.828 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.828 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.829 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.829 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.830 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.830 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.831 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:49:50Z" 2022-09-13T19:12:17.831 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.832 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:17.834 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.834 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:55.635+0000 7fbf00cfc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:49:55.636+0000 7fbf00cfc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:17.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:17.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:49:55Z" 2022-09-13T19:12:17.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.865 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.865 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.866 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.866 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.867 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.867 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.868 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.868 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.869 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.869 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.869 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.870 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.870 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.871 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.871 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.871 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.872 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.872 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.873 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.873 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.874 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.874 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.874 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.875 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.875 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.876 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.876 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.876 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.877 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.877 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.878 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.878 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.879 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.879 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.879 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.880 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:17.880 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.881 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.881 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.881 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:17.882 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.882 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.883 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.883 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.883 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.884 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.884 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.885 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.885 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.885 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.886 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.886 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.887 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.887 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.888 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.888 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.889 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.889 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.890 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.890 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.890 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:49:55Z" 2022-09-13T19:12:17.891 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.891 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055099.070128_219.1 2022-09-13T19:12:17.893 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:00.863+0000 7f70b782e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:00.866+0000 7f70b782e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.105395_221.1", 2022-09-13T19:12:17.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T19:12:17.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:50:00Z" 2022-09-13T19:12:17.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.924 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.924 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.925 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.925 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.926 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.926 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.927 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.927 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.927 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.928 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.928 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.929 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.929 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.929 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.930 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.930 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.931 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.931 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.931 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.932 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.932 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.933 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.933 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.934 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.934 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.934 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.935 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.935 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.936 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.936 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.936 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.937 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.937 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.938 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.938 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.938 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.939 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.105395_221.1", 2022-09-13T19:12:17.939 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.940 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.940 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.940 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T19:12:17.941 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.941 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.942 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.942 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.943 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:17.943 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.943 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.944 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.944 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.945 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.945 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.945 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.946 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.946 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.947 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.947 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:17.947 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.948 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:17.948 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:17.949 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:17.949 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.949 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:50:00Z" 2022-09-13T19:12:17.950 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:17.950 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:17.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.105395_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.105395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:17.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.105395_221.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:17.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055099.105395_221.1 2022-09-13T19:12:17.952 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.953 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:17.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:17.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:17.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:17.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:06.035+0000 7f13d82cd600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:06.036+0000 7f13d82cd600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:17.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:17.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:17.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:17.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:17.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:17.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:17.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:17.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:17.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:17.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:17.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:17.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:17.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:17.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:17.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:17.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:17.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:17.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:17.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:17.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:17.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:17.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:17.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:17.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:17.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:17.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:17.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:17.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:17.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:50:06Z" 2022-09-13T19:12:17.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:17.982 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:17.983 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:17.983 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:17.984 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:17.984 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:17.985 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:17.985 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.986 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.986 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:17.987 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.987 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:17.987 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.988 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.988 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:17.989 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.989 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:17.989 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.990 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:17.990 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.991 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.991 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.991 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.992 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:17.992 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.993 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.993 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:17.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.994 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:17.995 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:17.995 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:17.996 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:17.996 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:17.996 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:17.997 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:17.997 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:17.998 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:17.998 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:17.998 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:17.999 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:17.999 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.000 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.000 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:18.001 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.001 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.001 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.002 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.002 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:18.003 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.003 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.004 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.004 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.005 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.005 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.006 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.006 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.007 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.007 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.007 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:18.008 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.008 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:18.009 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:18.009 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:18.010 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.010 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:50:06Z" 2022-09-13T19:12:18.010 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:18.011 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:18.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.786343_268.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055101.786343_268.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:18.013 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:18.014 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.014 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:18.015 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_period_incremental_sync 2022-09-13T19:12:18.015 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:18.018 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:18.018 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:18.019 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:18.019 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 899, in test_multi_period_incremental_sync 2022-09-13T19:12:18.020 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2022-09-13T19:12:18.020 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:18.020 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2022-09-13T19:12:18.021 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 196, in zone_meta_checkpoint 2022-09-13T19:12:18.021 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2022-09-13T19:12:18.022 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2022-09-13T19:12:18.022 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:18.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=yxvvuh-39 2022-09-13T19:12:18.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-39/ 2022-09-13T19:12:18.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-39/ 2022-09-13T19:12:18.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:18.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-39/ 2022-09-13T19:12:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:18.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:18.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:18.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:18.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:18.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:18.030 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:18.030 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.031 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.032 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:50:11 GMT 2022-09-13T19:12:18.033 INFO:tasks.rgw_multisite_tests:/yxvvuh-39/ 2022-09-13T19:12:18.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:18.034 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:nGoKqYVaD9ea7mQTyzPxW6JB7OM= 2022-09-13T19:12:18.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:50:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nGoKqYVaD9ea7mQTyzPxW6JB7OM=', 'Content-Length': '0'} 2022-09-13T19:12:18.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000eef94920a47e8c27-0063203633-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:18.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=yxvvuh-40 2022-09-13T19:12:18.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-40/ 2022-09-13T19:12:18.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-40/ 2022-09-13T19:12:18.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:18.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-40/ 2022-09-13T19:12:18.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:18.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:18.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:18.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:18.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:18.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:18.046 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:18.047 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.047 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.048 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:50:11 GMT 2022-09-13T19:12:18.048 INFO:tasks.rgw_multisite_tests:/yxvvuh-40/ 2022-09-13T19:12:18.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:18.049 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:rjiSdm8WIwkoMwA7QiChkxSKqJo= 2022-09-13T19:12:18.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:50:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rjiSdm8WIwkoMwA7QiChkxSKqJo=', 'Content-Length': '0'} 2022-09-13T19:12:18.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000c84b10d0cf50ebb4-0063203633-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:18.050 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=yxvvuh-41 2022-09-13T19:12:18.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-41/ 2022-09-13T19:12:18.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-41/ 2022-09-13T19:12:18.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:18.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-41/ 2022-09-13T19:12:18.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:18.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:18.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:18.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:18.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:18.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-13T19:12:18.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:18.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:18.057 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:18.058 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.058 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.059 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:50:11 GMT 2022-09-13T19:12:18.059 INFO:tasks.rgw_multisite_tests:/yxvvuh-41/ 2022-09-13T19:12:18.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:18.060 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:tdckn/zSbHPEQDwMRZhxQo2O2RU= 2022-09-13T19:12:18.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:50:11 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tdckn/zSbHPEQDwMRZhxQo2O2RU=', 'Content-Length': '0'} 2022-09-13T19:12:18.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f910536f596dfe57-0063203633-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:18.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.062 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:18.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:18.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:18.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:18.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:18.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:18.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:18.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:18.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:18.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:18.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663055411.238167_296.1", 2022-09-13T19:12:18.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:50:11.238167Z" 2022-09-13T19:12:18.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:18.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:18.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:18.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:18.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:18.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:18.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.074 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:11.496+0000 7fe9b3e0c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:11.497+0000 7fe9b3e0c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:18.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:18.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:18.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:18.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:18.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:18.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:18.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:18.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:18.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:18.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:18.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:18.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:18.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:18.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:18.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:18.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:18.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:18.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:50:11Z" 2022-09-13T19:12:18.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:18.108 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:18.109 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:18.109 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:18.110 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:18.110 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.110 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.111 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.111 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:18.112 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.112 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:18.112 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.113 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.113 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.114 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.115 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:18.115 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.116 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.117 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.117 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.117 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.118 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:18.119 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.119 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.120 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:18.120 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.121 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.121 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.121 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:18.122 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.122 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.123 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.123 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.124 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:18.124 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.125 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.125 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:18.126 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.126 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.127 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.127 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:18.127 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.128 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.128 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.129 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.129 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:18.130 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.130 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.130 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.131 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.131 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.132 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.132 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.133 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.133 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.134 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.134 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:18.135 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.135 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:18.135 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:18.136 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:18.136 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.137 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:50:11Z" 2022-09-13T19:12:18.137 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:18.138 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:18.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:18.140 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.140 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:16.650+0000 7f7798e24600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:16.651+0000 7f7798e24600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:18.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:18.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:18.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:18.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:18.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:18.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:18.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:18.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:18.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:18.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:18.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:18.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:18.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:18.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:18.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:18.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:18.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:18.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:50:16Z" 2022-09-13T19:12:18.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:18.173 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:18.173 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:18.174 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:18.174 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:18.175 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.175 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.176 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.176 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:18.177 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.177 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:18.177 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.178 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.178 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.179 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.179 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:18.180 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.180 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.180 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.181 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.181 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.182 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.182 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:18.183 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.183 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.184 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:18.184 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.185 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.185 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.186 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:18.186 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.186 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.187 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.187 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.188 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:18.188 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.189 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.189 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:18.189 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.190 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.190 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.191 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:18.191 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.192 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.192 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.193 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.193 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:18.193 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.194 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.194 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.195 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.196 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.196 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.197 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.197 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.198 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.198 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:18.198 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.199 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:18.199 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:18.200 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:18.200 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.200 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:50:16Z" 2022-09-13T19:12:18.201 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:18.201 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:18.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.854520_207.1 2022-09-13T19:12:18.203 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:21.787+0000 7f1a4aaf3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:21.788+0000 7f1a4aaf3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:18.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:18.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:18.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:18.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:18.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:18.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:18.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:18.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:18.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:18.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:18.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:18.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:18.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:18.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:18.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:18.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:18.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:18.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:50:21Z" 2022-09-13T19:12:18.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:18.234 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:18.235 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:18.235 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:18.236 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:18.236 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.237 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.237 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.237 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:18.238 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.238 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:18.239 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.239 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.240 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.240 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.240 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:18.241 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.241 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.242 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.242 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.243 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.243 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.244 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:18.244 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.245 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.245 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:18.245 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.246 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.246 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.247 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:18.247 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.247 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.248 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.248 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.249 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:18.249 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.250 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.250 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:18.250 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.251 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.251 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.252 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:18.252 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.253 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.253 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.253 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.254 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:18.254 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.255 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.255 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.256 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.256 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.256 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.257 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.257 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.258 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.258 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.259 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:18.259 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.259 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:18.260 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:18.260 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:18.261 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.261 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:50:21Z" 2022-09-13T19:12:18.261 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:18.262 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:18.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.911619_214.1 2022-09-13T19:12:18.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:27.020+0000 7f53ea506600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:27.021+0000 7f53ea506600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:18.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:18.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:18.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:18.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:18.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:18.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:18.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:18.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:18.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:18.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:18.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:18.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:18.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:18.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:18.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:18.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:18.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:18.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:50:27Z" 2022-09-13T19:12:18.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:18.296 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:18.297 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:18.297 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:18.298 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:18.298 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.299 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.299 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.299 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:18.300 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.300 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:18.301 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.301 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.301 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.302 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.302 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:18.303 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.303 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.304 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.304 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.304 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.305 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.305 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:18.306 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.306 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.307 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:18.307 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.307 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.308 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.308 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:18.309 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.309 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.309 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.310 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.310 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:18.311 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.311 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.312 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:18.312 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.312 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.313 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.313 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:18.314 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.314 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.314 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.315 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.315 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:18.316 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.316 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.317 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.317 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.317 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.318 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.318 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.319 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.319 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.320 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.320 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:18.320 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.321 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:18.321 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:18.322 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:18.322 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.323 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:50:27Z" 2022-09-13T19:12:18.323 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:18.323 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:18.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:18.325 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.326 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:32.152+0000 7f4f8ab70600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:32.153+0000 7f4f8ab70600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:18.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:18.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:18.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:18.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:18.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:18.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:18.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:18.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:18.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:18.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:18.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:18.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:18.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:18.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:18.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:18.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:18.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:18.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:50:32Z" 2022-09-13T19:12:18.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:18.358 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:18.358 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:18.359 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:18.359 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:18.360 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.360 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.361 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.361 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:18.362 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.362 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:18.363 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.363 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.364 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.365 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.365 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:18.366 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.366 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.367 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.368 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.368 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.369 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:18.369 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.370 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.370 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:18.371 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.371 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.372 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.373 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:18.373 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.374 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.374 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.375 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.376 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:18.376 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.377 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.378 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:18.378 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.379 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.379 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.380 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:18.380 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.381 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.381 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.382 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.382 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:18.383 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.383 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.384 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.385 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.385 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.386 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.386 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.387 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.387 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.388 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:18.388 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.389 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:18.389 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:18.390 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:18.390 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.391 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:50:32Z" 2022-09-13T19:12:18.391 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:18.392 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:18.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:18.394 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:37.288+0000 7fd9175ad600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:37.289+0000 7fd9175ad600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:18.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:18.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:18.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:18.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:18.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:18.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:18.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:18.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:18.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:18.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:18.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.994620_218.1", 2022-09-13T19:12:18.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.994620Z", 2022-09-13T19:12:18.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:18.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:18.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:18.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:18.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:18.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:50:37Z" 2022-09-13T19:12:18.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:18.431 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:18.432 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:18.432 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:18.433 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:18.433 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.433 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.434 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.434 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:18.435 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.435 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:18.436 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.436 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.436 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.437 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.437 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:18.438 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.438 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.439 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.439 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.439 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.440 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.440 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:18.441 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.441 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.441 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:18.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.442 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.443 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.443 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:18.444 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.444 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.444 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.445 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.445 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:18.446 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.446 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.447 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.994620_218.1", 2022-09-13T19:12:18.447 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.448 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.448 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.448 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.994620Z", 2022-09-13T19:12:18.449 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.449 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.450 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.450 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.451 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:18.451 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.451 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.452 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.452 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.453 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.453 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.454 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.454 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.455 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.455 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:18.456 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.456 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:18.456 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:18.457 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:18.457 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.458 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:50:37Z" 2022-09-13T19:12:18.458 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:18.459 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.994620_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.994620Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:18.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.994620_218.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.994620_218.1 2022-09-13T19:12:18.461 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.461 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:42.483+0000 7f5e3598e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:42.485+0000 7f5e3598e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:18.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:18.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:18.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:18.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:18.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:18.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:18.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:18.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:18.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:18.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:18.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:18.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:18.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:18.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:18.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:18.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:18.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:18.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:50:42Z" 2022-09-13T19:12:18.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:18.495 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:18.495 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:18.496 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:18.496 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:18.496 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.497 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.497 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.498 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:18.498 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.499 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:18.500 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.500 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.501 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.501 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.501 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:18.502 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.502 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.503 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.503 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.504 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.504 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.505 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:18.505 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.505 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.506 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:18.506 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.507 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.507 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.508 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:18.508 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.509 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.509 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.510 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.510 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:18.511 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.511 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.512 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:18.512 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.513 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.513 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.513 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:18.514 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.514 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.515 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.515 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.516 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:18.516 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.517 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.517 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.518 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.518 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.519 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.519 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.520 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.520 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.521 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.521 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:18.522 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.522 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:18.522 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:18.523 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:18.523 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.524 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:50:42Z" 2022-09-13T19:12:18.524 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:18.525 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:18.526 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.526 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.526 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:18.527 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.527 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:47.633+0000 7fcd5684d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:47.634+0000 7fcd5684d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:18.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:18.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:18.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:18.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:18.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:18.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:18.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:18.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:18.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:18.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:18.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:18.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:18.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:18.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:18.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:18.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:18.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:18.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:50:47Z" 2022-09-13T19:12:18.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:18.560 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:18.560 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:18.561 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:18.561 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:18.562 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.562 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.563 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.563 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:18.564 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.564 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:18.565 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.565 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.566 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.566 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.567 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:18.567 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.567 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.568 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.568 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.569 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.569 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.570 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:18.570 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.571 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.571 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:18.572 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.572 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.572 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.573 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:18.573 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.574 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.574 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.575 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.575 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:18.576 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.576 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.576 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:18.577 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.577 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.578 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.578 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:18.579 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.579 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.580 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.581 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.581 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:18.582 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.582 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.583 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.583 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.584 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.584 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.584 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.585 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.585 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.586 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.586 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:18.587 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.587 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:18.588 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:18.588 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:18.589 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.589 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:50:47Z" 2022-09-13T19:12:18.589 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:18.590 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:18.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.070128_219.1 2022-09-13T19:12:18.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:52.793+0000 7f97f5467600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:52.794+0000 7f97f5467600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:18.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:18.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:18.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:18.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:18.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:18.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:18.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:18.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:18.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:18.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:18.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:18.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:18.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:18.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:18.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:18.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:18.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:18.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:50:52Z" 2022-09-13T19:12:18.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:18.628 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:18.628 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:18.628 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:18.629 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:18.629 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.630 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.630 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.631 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:18.631 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.632 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:18.632 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.633 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.634 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.634 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:18.635 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.635 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.635 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.636 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.636 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.637 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.637 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:18.638 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.638 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.638 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:18.639 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.639 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.640 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.640 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:18.641 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.641 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.642 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.642 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.643 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:18.643 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.644 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.644 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:18.645 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.645 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.646 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.646 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:18.647 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.647 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.648 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.648 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.649 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:18.649 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.649 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.650 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.650 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.651 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.652 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.652 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.653 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.653 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.654 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:18.654 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.655 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:18.655 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:18.656 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:18.656 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.657 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:50:52Z" 2022-09-13T19:12:18.657 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:18.658 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:18.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:18.660 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.660 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:57.928+0000 7f1ac1573600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:50:57.929+0000 7f1ac1573600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:18.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:18.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:18.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:18.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:18.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:18.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:18.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:18.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:18.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:18.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:18.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.146590_228.1", 2022-09-13T19:12:18.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.146590Z", 2022-09-13T19:12:18.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:18.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:18.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:18.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:18.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:18.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:50:57Z" 2022-09-13T19:12:18.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:18.704 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:18.705 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:18.705 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:18.706 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:18.706 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.707 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.707 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.708 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:18.708 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.708 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:18.709 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.709 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.710 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.711 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:18.711 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.712 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.712 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.712 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.713 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.713 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.714 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:18.714 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.715 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.715 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:18.716 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.716 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.717 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.717 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:18.718 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.718 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.719 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.719 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.719 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:18.720 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.720 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.721 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.146590_228.1", 2022-09-13T19:12:18.721 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.722 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.722 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.723 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.146590Z", 2022-09-13T19:12:18.723 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.723 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.724 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.724 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.725 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:18.725 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.726 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.726 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.726 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.727 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.727 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.728 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.728 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.729 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.729 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.730 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:18.730 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.731 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:18.731 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:18.732 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:18.732 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.733 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:50:57Z" 2022-09-13T19:12:18.733 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:18.734 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.146590_228.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.146590Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:18.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.146590_228.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.736 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.146590_228.1 2022-09-13T19:12:18.736 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.736 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:03.061+0000 7f55e7eab600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:03.062+0000 7f55e7eab600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:18.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:18.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:18.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:18.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:18.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:18.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:18.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:18.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:18.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:18.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:18.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.105395_221.1", 2022-09-13T19:12:18.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T19:12:18.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:18.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:18.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:18.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:18.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:18.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:51:03Z" 2022-09-13T19:12:18.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:18.771 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:18.771 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:18.772 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:18.772 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:18.773 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.773 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.773 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.774 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:18.774 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.775 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:18.775 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.776 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.776 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.777 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.777 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:18.777 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.778 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.778 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.779 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.779 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.780 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.780 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:18.781 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.782 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.783 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:18.783 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.784 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.784 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.785 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:18.785 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.786 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.786 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.786 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.787 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:18.787 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.788 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.788 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.105395_221.1", 2022-09-13T19:12:18.789 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.789 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.790 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.790 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T19:12:18.791 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.791 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.791 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.792 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.792 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:18.793 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.793 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.794 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.794 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.795 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.795 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.796 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.796 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.796 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.797 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.797 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:18.798 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.798 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:18.799 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:18.799 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:18.800 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.800 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:51:03Z" 2022-09-13T19:12:18.800 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:18.801 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.105395_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.105395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:18.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.105395_221.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.105395_221.1 2022-09-13T19:12:18.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.804 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:08.182+0000 7fb131da7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:08.183+0000 7fb131da7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:18.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:18.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:18.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:18.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:18.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:18.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:18.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:18.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:18.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:18.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:18.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:18.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:18.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:18.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:18.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:18.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:18.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:18.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:51:08Z" 2022-09-13T19:12:18.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:18.836 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:18.837 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:18.837 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:18.838 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:18.838 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.839 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.839 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.839 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:18.840 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.840 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:18.841 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.841 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.842 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.842 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.843 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:18.843 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.844 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.844 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.845 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.845 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.845 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.846 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:18.846 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.847 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.847 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:18.848 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.848 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.849 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.849 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:18.849 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.850 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.850 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.851 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.851 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:18.852 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.852 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.853 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:18.853 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.853 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.854 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.854 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:18.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.855 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.856 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.856 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.857 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:18.857 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.858 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.858 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.859 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.859 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.859 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.860 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.860 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.861 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.861 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.862 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:18.862 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.863 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:18.863 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:18.864 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:18.864 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.864 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:51:08Z" 2022-09-13T19:12:18.865 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:18.865 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:18.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:18.867 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.868 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:13.360+0000 7fc1852c7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:13.362+0000 7fc1852c7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:18.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:18.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:18.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:18.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:18.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:18.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:18.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:18.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:18.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:18.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:18.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:18.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:18.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:18.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:18.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:18.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:18.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:18.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:51:13Z" 2022-09-13T19:12:18.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:18.900 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:18.900 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:18.901 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:18.901 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:18.901 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.902 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.902 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.903 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:18.903 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.904 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:18.904 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.905 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.905 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.906 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:18.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.907 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.908 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.908 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.908 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.909 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:18.909 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.910 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.910 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:18.911 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.911 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.912 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.912 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:18.912 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.913 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.913 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.914 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.914 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:18.915 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.915 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.915 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:18.916 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.916 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.917 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.917 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:18.918 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.918 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.918 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.919 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.919 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:18.920 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.920 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.921 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.921 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.922 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.922 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.922 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.923 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.923 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.924 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.924 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:18.925 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.925 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:18.926 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:18.926 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:18.926 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.927 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:51:13Z" 2022-09-13T19:12:18.927 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:18.928 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:18.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:18.930 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.930 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:18.495+0000 7fb0cc987600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:18.496+0000 7fb0cc987600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:18.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:18.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:18.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:18.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:18.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:18.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:18.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:18.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:18.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:18.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:18.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:18.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:18.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:18.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:18.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:18.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:18.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:18.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:18.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:18.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:18.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:18.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:18.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:51:18Z" 2022-09-13T19:12:18.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:18.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:18.962 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:18.963 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:18.963 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:18.964 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:18.964 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.965 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.965 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.966 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:18.966 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.967 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:18.967 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.967 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.968 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:18.968 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.969 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:18.969 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.970 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:18.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.970 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.971 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.971 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.972 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:18.972 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.973 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.973 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:18.974 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.974 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.974 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.975 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:18.975 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.976 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.976 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.977 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.977 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:18.977 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.978 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.978 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:18.979 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.979 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.980 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.980 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:18.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.981 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.981 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.982 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:18.982 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:18.983 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:18.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:18.984 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:18.984 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:18.985 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:18.985 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:18.985 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:18.986 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:18.986 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.987 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:18.987 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:18.988 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.988 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:18.988 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:18.989 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:18.989 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:18.990 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:51:18Z" 2022-09-13T19:12:18.990 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:18.991 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:18.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:18.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:18.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.861945_210.1 2022-09-13T19:12:18.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:18.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:18.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:18.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:18.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:23.631+0000 7fe03d378600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:23.632+0000 7fe03d378600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:18.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:18.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:18.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:18.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:18.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:18.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:18.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:18.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:18.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:19.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:19.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:19.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:19.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:19.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:19.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:19.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:19.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:19.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:19.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:19.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:51:23Z" 2022-09-13T19:12:19.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:19.025 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:19.026 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:19.026 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:19.027 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:19.027 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.027 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.028 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.028 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:19.029 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.029 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:19.030 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.030 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.031 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.031 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.032 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:19.032 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.032 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.033 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.033 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.034 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.034 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.035 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:19.035 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.035 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.036 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:19.036 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.037 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.037 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.038 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:19.038 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.039 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.039 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.039 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.040 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:19.040 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.041 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.041 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:19.042 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.042 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.043 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.043 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:19.043 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.044 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.044 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.045 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.045 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:19.046 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.046 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.047 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.047 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.047 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.048 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.048 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.049 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.050 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.050 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:19.050 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.051 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:19.051 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:19.052 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:19.052 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.053 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:51:23Z" 2022-09-13T19:12:19.053 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:19.054 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:19.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:19.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.953175_216.1 2022-09-13T19:12:19.056 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.056 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:19.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:19.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:19.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:28.813+0000 7eff283fe600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:28.814+0000 7eff283fe600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:19.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:19.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:19.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:19.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:19.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:19.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:19.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:19.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:19.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:19.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:19.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:19.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:19.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:19.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:19.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:51:28Z" 2022-09-13T19:12:19.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:19.088 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:19.089 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:19.089 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:19.090 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:19.090 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.090 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.091 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.091 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:19.092 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.092 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:19.093 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.093 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.093 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.094 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.094 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:19.095 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.095 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.096 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.096 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.097 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.097 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.097 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:19.098 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.098 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.099 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:19.099 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.100 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.100 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.101 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:19.101 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.101 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.102 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.102 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.103 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:19.103 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.104 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.104 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:19.105 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.105 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.106 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.106 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:19.106 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.107 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.107 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.108 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.108 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:19.109 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.109 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.109 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.110 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.110 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.111 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.111 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.112 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.112 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.113 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.113 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:19.113 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.114 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:19.114 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:19.115 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:19.115 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.116 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:51:28Z" 2022-09-13T19:12:19.116 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:19.117 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:19.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:19.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:19.119 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.119 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:19.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:19.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:19.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:33.961+0000 7f24b092d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:33.962+0000 7f24b092d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:19.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:19.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:19.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:19.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:19.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:19.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:19.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:19.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:19.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:19.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:19.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:19.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:19.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:19.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:19.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:51:33Z" 2022-09-13T19:12:19.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:19.153 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:19.154 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:19.155 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:19.155 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:19.156 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.156 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.157 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.157 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:19.158 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.158 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:19.159 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.159 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.160 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.160 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.160 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:19.161 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.162 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.162 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.163 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.163 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.164 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.164 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:19.165 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.165 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.166 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:19.166 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.167 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.167 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.168 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:19.168 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.169 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.169 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.170 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.171 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:19.171 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.172 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.172 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:19.173 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.173 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.174 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.174 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:19.175 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.175 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.176 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.176 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.177 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:19.177 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.178 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.178 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.179 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.179 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.180 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.181 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.181 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.182 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.182 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.183 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:19.183 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.184 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:19.184 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:19.185 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:19.185 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.186 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:51:33Z" 2022-09-13T19:12:19.187 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:19.187 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:19.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:19.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.858000_209.1 2022-09-13T19:12:19.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.190 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:19.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:19.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:19.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:39.100+0000 7fef0a407600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:39.101+0000 7fef0a407600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:19.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:19.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:19.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:19.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:19.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:19.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:19.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:19.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:19.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:19.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:19.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:19.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:19.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:19.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:19.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:51:39Z" 2022-09-13T19:12:19.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:19.232 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:19.232 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:19.233 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:19.233 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:19.234 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.234 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.235 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.235 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:19.235 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.236 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:19.236 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.237 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.237 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.238 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.238 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:19.239 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.239 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.239 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.240 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.240 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.241 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.241 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:19.242 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.242 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.243 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:19.243 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.244 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.244 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.245 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:19.245 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.245 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.246 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.246 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.247 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:19.247 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.248 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.248 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:19.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.249 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.250 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:19.250 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.251 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.251 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.252 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.252 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:19.253 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.253 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.253 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.254 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.254 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.255 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.255 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.256 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.256 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.257 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.257 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:19.257 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.258 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:19.258 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:19.259 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:19.259 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.260 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:51:39Z" 2022-09-13T19:12:19.260 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:19.261 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:19.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:19.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:19.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:19.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:19.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:19.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:44.239+0000 7f678cf71600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:44.240+0000 7f678cf71600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:19.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:19.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:19.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:19.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:19.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:19.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:19.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:19.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:19.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:19.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:19.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:19.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:19.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:19.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:19.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:51:44Z" 2022-09-13T19:12:19.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:19.296 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:19.296 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:19.296 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:19.297 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:19.297 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.298 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.298 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.299 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:19.299 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.300 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:19.300 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.300 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.301 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.301 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.302 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:19.302 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.303 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.303 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.304 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.304 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.305 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.305 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:19.305 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.306 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.306 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:19.307 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.307 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.308 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.308 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:19.309 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.309 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.310 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.310 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.310 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:19.311 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.311 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.312 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:19.312 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.313 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.313 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.314 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:19.314 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.314 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.315 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.315 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.316 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:19.316 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.317 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.317 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.318 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.318 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.318 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.319 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.319 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.320 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.320 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.321 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:19.321 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.322 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:19.322 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:19.322 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:19.323 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.323 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:51:44Z" 2022-09-13T19:12:19.324 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:19.324 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:19.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:19.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.900410_211.1 2022-09-13T19:12:19.326 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.327 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:19.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:19.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:19.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:49.361+0000 7f68b1cd8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:49.362+0000 7f68b1cd8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:19.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:19.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:19.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:19.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:19.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:19.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:19.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:19.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.111513_224.1", 2022-09-13T19:12:19.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.111513Z", 2022-09-13T19:12:19.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:19.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:19.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:19.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:19.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:19.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:51:49Z" 2022-09-13T19:12:19.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:19.360 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:19.360 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:19.361 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:19.361 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:19.362 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.362 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.363 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.363 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:19.364 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.364 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:19.364 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.365 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.365 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.366 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:19.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.368 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.368 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.368 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.369 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.369 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:19.370 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.370 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.371 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:19.371 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.371 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.372 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.372 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:19.373 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.373 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.374 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.374 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.375 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:19.375 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.376 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.376 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.111513_224.1", 2022-09-13T19:12:19.376 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.377 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.377 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.378 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.111513Z", 2022-09-13T19:12:19.378 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.379 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.379 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.380 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.380 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:19.380 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.381 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.381 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.382 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.382 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.383 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.383 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.383 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.384 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.384 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.385 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:19.385 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.386 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:19.386 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:19.387 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:19.387 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.387 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:51:49Z" 2022-09-13T19:12:19.388 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:19.388 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:19.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.111513_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.111513Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:19.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.111513_224.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.111513_224.1 2022-09-13T19:12:19.391 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.391 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:19.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:19.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:19.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:54.509+0000 7efcc83d6600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:54.510+0000 7efcc83d6600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:19.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:19.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:19.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:19.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:19.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:19.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:19.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:19.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:19.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:19.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:19.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:19.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:19.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:19.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:19.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:51:54Z" 2022-09-13T19:12:19.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:19.425 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:19.425 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:19.426 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:19.426 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:19.427 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.427 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.428 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.428 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:19.429 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.429 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:19.429 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.430 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.430 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.431 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.431 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:19.432 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.432 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.433 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.433 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.433 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.434 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.435 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:19.435 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.436 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.436 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:19.437 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.437 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.438 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.438 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:19.439 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.439 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.439 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.440 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.440 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:19.441 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.441 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.442 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:19.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.443 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.443 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.444 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:19.444 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.445 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.445 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.446 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.446 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:19.447 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.447 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.447 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.448 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.448 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.449 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.449 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.450 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.450 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.450 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.451 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:19.451 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.452 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:19.452 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:19.453 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:19.453 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.454 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:51:54Z" 2022-09-13T19:12:19.454 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:19.455 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:19.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:19.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.858000_209.1 2022-09-13T19:12:19.457 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.457 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:19.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:19.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:19.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:59.734+0000 7ff95ed6a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:51:59.736+0000 7ff95ed6a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:19.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:19.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:19.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:19.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:19.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:19.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:19.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:19.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:19.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:19.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:19.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:19.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:19.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:19.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:19.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:51:59Z" 2022-09-13T19:12:19.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:19.490 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:19.490 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:19.491 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:19.491 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:19.492 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.492 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.493 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.493 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:19.494 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.494 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:19.494 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.495 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.495 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.496 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.496 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:19.497 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.497 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.497 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.498 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.498 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.499 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.499 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:19.500 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.500 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.501 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:19.501 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.501 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.502 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.502 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:19.503 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.503 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.504 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.504 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.504 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:19.505 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.505 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.506 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:19.506 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.507 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.507 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.507 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:19.508 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.508 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.509 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.509 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.510 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:19.510 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.511 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.511 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.511 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.512 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.512 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.513 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.513 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.514 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.514 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.514 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:19.515 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.515 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:19.516 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:19.516 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:19.517 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.517 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:51:59Z" 2022-09-13T19:12:19.517 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:19.518 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:19.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:19.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.900410_211.1 2022-09-13T19:12:19.520 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:19.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:19.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:19.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:04.925+0000 7f07b6211600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:04.926+0000 7f07b6211600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:19.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:19.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:19.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:19.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:19.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:19.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:19.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:19.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:19.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:19.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:19.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:19.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:19.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:19.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:19.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:52:04Z" 2022-09-13T19:12:19.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:19.553 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:19.553 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:19.554 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:19.555 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:19.555 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.556 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.556 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:19.557 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.557 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:19.558 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.558 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.558 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.559 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.559 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:19.560 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.560 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.561 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.561 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.562 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.562 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.562 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:19.563 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.563 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.564 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:19.564 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.565 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.565 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.565 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:19.566 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.566 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.567 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.567 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.568 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:19.568 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.568 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.569 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:19.569 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.570 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.570 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.571 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:19.571 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.572 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.572 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.572 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.573 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:19.573 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.574 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.574 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.575 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.575 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.576 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.576 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.577 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.577 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.577 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.578 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:19.578 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.579 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:19.579 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:19.580 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:19.580 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.580 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:52:04Z" 2022-09-13T19:12:19.581 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:19.581 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:19.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:19.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:19.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:19.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:19.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:19.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:10.061+0000 7f10a0c99600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:10.062+0000 7f10a0c99600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:19.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:19.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:19.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:19.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:19.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:19.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:19.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:19.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:19.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:19.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:19.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:19.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:19.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:19.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:19.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:52:10Z" 2022-09-13T19:12:19.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:19.617 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:19.618 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:19.618 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:19.619 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:19.619 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.619 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.620 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.620 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:19.621 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.621 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:19.622 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.622 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.622 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.623 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.623 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:19.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.624 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.625 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.625 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.625 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.626 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.627 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:19.627 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.627 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.628 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:19.628 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.629 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.629 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.630 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:19.630 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.631 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.631 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.632 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.632 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:19.633 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.633 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:19.634 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.634 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.635 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.635 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:19.636 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.636 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.637 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.637 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.637 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:19.638 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.638 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.639 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.639 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.640 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.640 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.640 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.641 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.641 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.642 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.642 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:19.643 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.643 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:19.644 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:19.644 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:19.644 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.645 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:52:10Z" 2022-09-13T19:12:19.645 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:19.646 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:19.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:19.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:19.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:19.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:19.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:19.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:15.206+0000 7f699eca7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:15.207+0000 7f699eca7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:19.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:19.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:19.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:19.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:19.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:19.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:19.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:19.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:19.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:19.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:19.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:19.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:19.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:19.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:19.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:52:15Z" 2022-09-13T19:12:19.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:19.688 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:19.688 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:19.689 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:19.689 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:19.690 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.690 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.690 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.691 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:19.691 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.692 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:19.692 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.693 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.693 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.694 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.694 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:19.694 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.695 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.695 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.696 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.696 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.697 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.697 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:19.697 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.698 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.698 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:19.699 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.699 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.699 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.700 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:19.700 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.701 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.701 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.702 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.702 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:19.703 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.703 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.703 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:19.704 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.704 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.705 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.705 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:19.705 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.706 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.706 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.707 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.707 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:19.707 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.708 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.708 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.709 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.709 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.709 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.710 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.710 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.711 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.711 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.711 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:19.712 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.712 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:19.713 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:19.713 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:19.714 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.714 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:52:15Z" 2022-09-13T19:12:19.714 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:19.715 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:19.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:19.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.070128_219.1 2022-09-13T19:12:19.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:19.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:19.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:19.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:20.375+0000 7fcb0118f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:20.376+0000 7fcb0118f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:19.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:19.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:19.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:19.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:19.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:19.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:19.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:19.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:19.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:19.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:19.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:19.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:19.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:19.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:19.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:52:20Z" 2022-09-13T19:12:19.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:19.748 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:19.749 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:19.749 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:19.750 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:19.750 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.750 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.751 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.751 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:19.752 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.752 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:19.753 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.753 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.753 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.754 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.754 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:19.755 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.755 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.755 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.756 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.756 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.757 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.757 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:19.758 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.758 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.758 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:19.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.759 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.760 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.760 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:19.760 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.761 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.761 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.762 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.762 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:19.763 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.763 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.763 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:19.764 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.764 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.765 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.765 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:19.765 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.766 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.766 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.767 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.767 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:19.768 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.768 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.768 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.769 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.770 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.770 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.771 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.771 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.772 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:19.772 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.772 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:19.773 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:19.773 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:19.774 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.774 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:52:20Z" 2022-09-13T19:12:19.775 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:19.775 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:19.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:19.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.857076_208.1 2022-09-13T19:12:19.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.778 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:19.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:19.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:19.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:25.529+0000 7fcb442d1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:25.530+0000 7fcb442d1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:19.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:19.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:19.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:19.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:19.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:19.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:19.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:19.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:19.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:19.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:19.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:19.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:19.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:19.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:19.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:52:25Z" 2022-09-13T19:12:19.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:19.809 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:19.810 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:19.810 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:19.811 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:19.811 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.812 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.812 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.813 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:19.813 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.813 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:19.814 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.814 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.815 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.815 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.816 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:19.816 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.816 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.817 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.817 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.818 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.818 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.818 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:19.819 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.819 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.820 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:19.820 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.820 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.821 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.821 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:19.822 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.822 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.823 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.823 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.823 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:19.824 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.824 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.825 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:19.825 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.825 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.826 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.826 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:19.827 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.827 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.828 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.828 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.828 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:19.829 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.829 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.830 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.830 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.831 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.831 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.831 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.832 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.832 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.833 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.834 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:19.835 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.836 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:19.837 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:19.837 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:19.838 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.838 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:52:25Z" 2022-09-13T19:12:19.839 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:19.840 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:19.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:19.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:19.843 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.844 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:19.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:19.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:19.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:30.665+0000 7f89b0222600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:30.667+0000 7f89b0222600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:19.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:19.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:19.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:19.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:19.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:19.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:19.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:19.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:19.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:19.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:19.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:19.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:19.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:19.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:19.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:52:30Z" 2022-09-13T19:12:19.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:19.884 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:19.884 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:19.885 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:19.886 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:19.886 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.888 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.888 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:19.889 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.890 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:19.890 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.891 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.892 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.893 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.893 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:19.894 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.895 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.895 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.896 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.897 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.898 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.898 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:19.899 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.899 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.899 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:19.900 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.900 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.901 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.901 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:19.902 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.902 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.903 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.903 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.903 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:19.904 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.904 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.905 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:19.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.906 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.906 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:19.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.907 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.908 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.908 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.909 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:19.909 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.909 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.910 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.911 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.911 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.912 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.912 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.912 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.913 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.913 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.914 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:19.914 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.915 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:19.915 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:19.915 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:19.916 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.916 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:52:30Z" 2022-09-13T19:12:19.917 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:19.917 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:19.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:19.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:19.920 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.920 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:19.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:19.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:19.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:35.801+0000 7f92c737a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:35.802+0000 7f92c737a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:19.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:19.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:19.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:19.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:19.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:19.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:19.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:19.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:19.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:19.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:19.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:19.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:19.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:19.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:19.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:19.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:52:35Z" 2022-09-13T19:12:19.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:19.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:19.957 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:19.957 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:19.958 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:19.959 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:19.959 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.960 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.961 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:19.961 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.962 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:19.962 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.963 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.963 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.964 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.964 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:19.964 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.965 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.965 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.966 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.966 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.967 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.967 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:19.968 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.968 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.968 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:19.969 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.969 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.970 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.970 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:19.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.971 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.971 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.972 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.972 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:19.973 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.973 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.974 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:19.974 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.974 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.975 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.975 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:19.976 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.976 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.976 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.977 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:19.977 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:19.978 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:19.978 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:19.979 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:19.979 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:19.980 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:19.980 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:19.981 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:19.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:19.982 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.982 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:19.982 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:19.983 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.983 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:19.984 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:19.984 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:19.985 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:19.985 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:52:35Z" 2022-09-13T19:12:19.986 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:19.986 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:19.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:19.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:19.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:19.988 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.989 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:19.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:19.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:19.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:19.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:41.036+0000 7fb79d045600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:41.037+0000 7fb79d045600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:19.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:19.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:19.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:19.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:19.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:19.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:19.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:19.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:19.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:19.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:19.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:19.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:19.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:19.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:19.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:19.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:19.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:19.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:20.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:20.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:20.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:20.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.907374_213.1", 2022-09-13T19:12:20.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.907374Z", 2022-09-13T19:12:20.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:20.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:20.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:20.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:20.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:20.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:52:41Z" 2022-09-13T19:12:20.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:20.019 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:20.019 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:20.020 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:20.020 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:20.021 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.021 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.021 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.022 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:20.022 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.023 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:20.023 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.024 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.024 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.024 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.025 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:20.025 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.026 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.026 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.026 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.027 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.027 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.028 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:20.028 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.028 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.029 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:20.029 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.030 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.030 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.030 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:20.031 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.031 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.032 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.032 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.033 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:20.033 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.033 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.034 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.907374_213.1", 2022-09-13T19:12:20.034 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.035 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.035 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.035 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.907374Z", 2022-09-13T19:12:20.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.036 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.037 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.037 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.037 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:20.038 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.038 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.039 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.039 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.040 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.040 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.040 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.041 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.041 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.042 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.042 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:20.042 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.043 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:20.043 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:20.044 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:20.044 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.044 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:52:41Z" 2022-09-13T19:12:20.045 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:20.045 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:20.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.907374_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.907374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:20.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.907374_213.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.907374_213.1 2022-09-13T19:12:20.047 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.048 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:20.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:20.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:20.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:46.167+0000 7fd51def5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:46.168+0000 7fd51def5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:20.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:20.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:20.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:20.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:20.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:20.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:20.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:20.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:20.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:20.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:20.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:20.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:20.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:20.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:20.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:20.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:20.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:20.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:52:46Z" 2022-09-13T19:12:20.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:20.077 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:20.077 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:20.078 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:20.078 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:20.079 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.079 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.079 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.080 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:20.080 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.081 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:20.081 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.081 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.082 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.082 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.083 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:20.083 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.083 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.084 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.084 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.085 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.085 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.085 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:20.086 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.086 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.087 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:20.087 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.088 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.088 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.088 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:20.089 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.089 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.090 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.090 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.090 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:20.091 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.091 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.092 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:20.092 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.092 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.093 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.093 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:20.094 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.094 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.094 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.095 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.095 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:20.096 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.096 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.096 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.097 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.097 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.098 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.098 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.098 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.099 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.099 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.100 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:20.100 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.101 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:20.101 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:20.101 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:20.102 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.102 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:52:46Z" 2022-09-13T19:12:20.103 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:20.103 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:20.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:20.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:20.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:20.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:20.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:20.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:51.300+0000 7f07d162e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:51.301+0000 7f07d162e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:20.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:20.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:20.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:20.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:20.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:20.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:20.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:20.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:20.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:20.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:20.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:20.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:20.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:20.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:20.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:20.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:20.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:20.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:52:51Z" 2022-09-13T19:12:20.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:20.135 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:20.135 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:20.136 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:20.136 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:20.136 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.137 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.137 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.138 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:20.138 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.138 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:20.139 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.139 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.140 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.140 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.140 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:20.141 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.141 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.142 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.142 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.142 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.143 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.143 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:20.144 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.144 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.144 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:20.145 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.145 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.146 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.146 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:20.147 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.147 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.147 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.148 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.148 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:20.149 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.149 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.149 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:20.150 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.150 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.151 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.151 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:20.151 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.152 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.152 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.153 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.153 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:20.153 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.154 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.154 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.155 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.155 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.156 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.156 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.156 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.157 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.157 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.158 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:20.158 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.158 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:20.159 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:20.159 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:20.160 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.160 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:52:51Z" 2022-09-13T19:12:20.160 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:20.161 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:20.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:20.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:20.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:20.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:20.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:20.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:56.426+0000 7ff1df136600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:52:56.427+0000 7ff1df136600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:20.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:20.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:20.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:20.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:20.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:20.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:20.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:20.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:20.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:20.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:20.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:20.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:20.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:20.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:20.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:20.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:20.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:20.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:52:56Z" 2022-09-13T19:12:20.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:20.193 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:20.194 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:20.194 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:20.195 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:20.195 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.196 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.196 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.196 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:20.197 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.197 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:20.198 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.198 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.198 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.199 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.199 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:20.200 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.200 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.200 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.201 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.201 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.202 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.202 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:20.202 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.203 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.203 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:20.204 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.204 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.204 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.205 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:20.205 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.206 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.206 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.207 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.207 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:20.208 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.208 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.208 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:20.209 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.209 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.210 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.210 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:20.210 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.211 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.211 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.212 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.212 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:20.212 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.213 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.213 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.214 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.214 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.215 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.215 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.215 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.216 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.216 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.217 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:20.217 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.217 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:20.218 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:20.218 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:20.219 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.219 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:52:56Z" 2022-09-13T19:12:20.220 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:20.220 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:20.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:20.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.070128_219.1 2022-09-13T19:12:20.222 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.223 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:20.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:20.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:20.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:01.550+0000 7fb3c4093600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:01.551+0000 7fb3c4093600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:20.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:20.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:20.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:20.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:20.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:20.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:20.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:20.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:20.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:20.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:20.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.072712_220.1", 2022-09-13T19:12:20.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.072712Z", 2022-09-13T19:12:20.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:20.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:20.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:20.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:20.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:20.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:53:01Z" 2022-09-13T19:12:20.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.253 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:20.254 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:20.254 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:20.255 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:20.255 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:20.256 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.256 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.256 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.257 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:20.257 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.258 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:20.258 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.258 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.259 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.260 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:20.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.261 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.261 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.262 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.262 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.263 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:20.263 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.263 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.264 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:20.264 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.265 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.265 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:20.266 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.266 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.267 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.267 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.268 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:20.268 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.269 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.269 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.072712_220.1", 2022-09-13T19:12:20.269 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.270 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.270 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.271 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.072712Z", 2022-09-13T19:12:20.271 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.272 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.272 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.273 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.273 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:20.274 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.274 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.275 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.275 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.275 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.276 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.276 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.277 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.277 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.278 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.278 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:20.279 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.279 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:20.280 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:20.281 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:20.281 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.282 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:53:01Z" 2022-09-13T19:12:20.283 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:20.283 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:20.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.072712_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.072712Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:20.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.072712_220.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.072712_220.1 2022-09-13T19:12:20.287 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:20.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:20.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:20.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:06.758+0000 7ff625af5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:06.759+0000 7ff625af5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:20.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:20.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:20.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:20.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:20.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:20.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:20.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:20.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:20.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:20.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:20.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.906479_212.1", 2022-09-13T19:12:20.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.906479Z", 2022-09-13T19:12:20.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:20.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:20.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:20.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:20.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:20.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:53:06Z" 2022-09-13T19:12:20.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:20.339 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:20.340 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:20.341 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:20.341 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:20.342 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.343 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.344 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.344 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:20.345 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.346 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:20.346 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.347 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.347 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.348 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.348 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:20.349 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.349 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.350 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.350 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.351 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.351 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.352 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:20.353 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.353 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.354 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:20.354 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.355 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.355 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.356 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:20.356 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.357 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.357 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.357 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.358 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:20.359 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.359 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.360 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.906479_212.1", 2022-09-13T19:12:20.360 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.360 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.361 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.361 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.906479Z", 2022-09-13T19:12:20.362 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.362 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.363 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.363 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.364 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:20.364 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.365 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.365 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.366 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.368 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.368 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.369 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.369 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:20.370 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.370 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:20.370 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:20.371 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:20.371 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.372 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:53:06Z" 2022-09-13T19:12:20.372 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:20.373 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:20.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.906479_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.906479Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:20.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.906479_212.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.906479_212.1 2022-09-13T19:12:20.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:20.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:20.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:20.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:11.887+0000 7fc694a25600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:11.888+0000 7fc694a25600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:20.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:20.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:20.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:20.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:20.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:20.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:20.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:20.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:20.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:20.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:20.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:20.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:20.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:20.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:20.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:53:11Z" 2022-09-13T19:12:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:20.407 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:20.408 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:20.408 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:20.408 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:20.409 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.409 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.410 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.410 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:20.410 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.411 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:20.411 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.412 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.412 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.412 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.413 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:20.413 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.414 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.414 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.414 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.415 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.415 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.416 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:20.416 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.417 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.417 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:20.418 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.418 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.419 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.419 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:20.419 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.420 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.420 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.421 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.421 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:20.421 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.422 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.422 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:20.423 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.423 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.424 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:20.424 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.425 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.425 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.425 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.426 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:20.426 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.427 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.427 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.428 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.428 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.429 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.429 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.429 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.430 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.431 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:20.431 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.431 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:20.432 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:20.432 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:20.433 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.433 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:53:11Z" 2022-09-13T19:12:20.433 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:20.434 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:20.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:20.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:20.436 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.436 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:20.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:20.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:20.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:17.004+0000 7fde16b67600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:17.006+0000 7fde16b67600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:20.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:20.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:20.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:20.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:20.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:20.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:20.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:20.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:20.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:20.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:20.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:20.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:20.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:20.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:20.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:20.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:20.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:20.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:53:17Z" 2022-09-13T19:12:20.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:20.466 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:20.466 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:20.467 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:20.467 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:20.468 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.468 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.468 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.469 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:20.469 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.470 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:20.470 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.471 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.471 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.471 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.472 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:20.472 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.473 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.473 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.473 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.474 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.474 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.475 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:20.475 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.475 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.476 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:20.476 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.477 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.477 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.477 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:20.478 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.478 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.479 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.479 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.479 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:20.480 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.480 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.481 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:20.481 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.481 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.482 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.482 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:20.483 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.483 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.483 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.484 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.484 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:20.485 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.485 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.485 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.486 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.486 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.487 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.487 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.487 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.488 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.488 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.489 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:20.489 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.489 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:20.490 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:20.490 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:20.491 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.491 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:53:17Z" 2022-09-13T19:12:20.492 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:20.492 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:20.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:20.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.857076_208.1 2022-09-13T19:12:20.494 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:20.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:20.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:20.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:22.125+0000 7f3d35434600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:22.126+0000 7f3d35434600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:20.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:20.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:20.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:20.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:20.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:20.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:20.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:20.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:20.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:20.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:20.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:20.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:20.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:20.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:20.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:20.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:20.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:20.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:53:22Z" 2022-09-13T19:12:20.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:20.525 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:20.525 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:20.526 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:20.526 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:20.526 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.527 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.527 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.528 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:20.528 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.528 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:20.529 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.529 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.530 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.530 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.530 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:20.531 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.531 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.532 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.532 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.532 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.533 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.533 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:20.534 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.534 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.534 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:20.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.535 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.536 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:20.536 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.537 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.537 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.538 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.538 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:20.539 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.539 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.539 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:20.540 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.540 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.541 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.541 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:20.541 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.542 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.542 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.543 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.543 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:20.543 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.544 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.544 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.545 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.545 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.545 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.546 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.546 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.547 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.547 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.547 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:20.548 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.548 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:20.549 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:20.549 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:20.549 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.550 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:53:22Z" 2022-09-13T19:12:20.550 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:20.551 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:20.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:20.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.857076_208.1 2022-09-13T19:12:20.553 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.553 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:20.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:20.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:20.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:27.272+0000 7f90694a3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:27.273+0000 7f90694a3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:20.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:20.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:20.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:20.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:20.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:20.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:20.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:20.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:20.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:20.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:20.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:20.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:20.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:20.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:20.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:20.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:20.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:20.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:53:27Z" 2022-09-13T19:12:20.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:20.583 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:20.583 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:20.584 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:20.584 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:20.585 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.585 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.586 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.586 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:20.587 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.587 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:20.587 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.588 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.588 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.589 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.589 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:20.589 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.590 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.590 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.591 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.591 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.591 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.592 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:20.592 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.593 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.593 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:20.593 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.594 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.594 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.595 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:20.595 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.596 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.596 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.596 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.597 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:20.597 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.598 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.598 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:20.598 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.599 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.599 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.600 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:20.600 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.600 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.601 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.601 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.602 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:20.602 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.603 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.603 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.603 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.604 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.604 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.605 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.605 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.605 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.606 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.606 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:20.607 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.607 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:20.607 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:20.608 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:20.608 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.609 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:53:27Z" 2022-09-13T19:12:20.609 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:20.609 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:20.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:20.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.953175_216.1 2022-09-13T19:12:20.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.612 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:20.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:20.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:20.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:32.414+0000 7f60832e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:32.415+0000 7f60832e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:20.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:20.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:20.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:20.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:20.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:20.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:20.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:20.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:20.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:20.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:20.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:20.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:20.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:20.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:20.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:20.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:20.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:20.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:53:32Z" 2022-09-13T19:12:20.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:20.645 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:20.645 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:20.645 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:20.646 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:20.646 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.647 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.647 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.648 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:20.648 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.649 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:20.649 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.650 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.650 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.651 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.651 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:20.652 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.652 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.653 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.654 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.654 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.655 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:20.655 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.655 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.656 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:20.656 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.657 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.657 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.658 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:20.658 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.659 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.659 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.660 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.660 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:20.661 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.661 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.661 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:20.662 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.662 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.663 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.663 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:20.664 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.664 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.665 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.665 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.666 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:20.666 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.667 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.667 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.668 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.668 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.669 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.669 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.670 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.670 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.671 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.671 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:20.672 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.672 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:20.672 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:20.673 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:20.673 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.674 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:53:32Z" 2022-09-13T19:12:20.674 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:20.675 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:20.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:20.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.854520_207.1 2022-09-13T19:12:20.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:20.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:20.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:20.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:37.647+0000 7f6ae54dd600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:37.649+0000 7f6ae54dd600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:20.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:20.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:20.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:20.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:20.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:20.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:20.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:20.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:20.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:20.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:20.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:20.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:20.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:20.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:20.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:20.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:20.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:20.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:53:37Z" 2022-09-13T19:12:20.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:20.707 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:20.707 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:20.708 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:20.708 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:20.709 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.709 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.709 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.710 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:20.710 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.711 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:20.711 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.711 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.712 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.712 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.713 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:20.713 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.713 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.714 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.714 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.715 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.715 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.715 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:20.716 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.716 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.717 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:20.717 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.718 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.718 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.718 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:20.719 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.719 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.720 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.720 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.721 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:20.721 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.721 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.722 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:20.722 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.723 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.723 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.724 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:20.724 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.724 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.725 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.725 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.726 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:20.726 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.727 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.727 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.729 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.730 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.731 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.731 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.745 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.745 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.746 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:20.747 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.748 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:20.749 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:20.749 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:20.750 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.751 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:53:37Z" 2022-09-13T19:12:20.751 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:20.752 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:20.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:20.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:20.756 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.757 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:20.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:20.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:20.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:42.808+0000 7fc163459600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:42.809+0000 7fc163459600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:20.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:20.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:20.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:20.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:20.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:20.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:20.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:20.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:20.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:20.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:20.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:20.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:20.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:20.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:20.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:20.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:20.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:20.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:53:42Z" 2022-09-13T19:12:20.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:20.801 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:20.801 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:20.802 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:20.803 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:20.803 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.804 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.805 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.806 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:20.806 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.807 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:20.808 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.808 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.809 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.809 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.810 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:20.811 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.811 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.812 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.812 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.813 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.813 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.814 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:20.814 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.815 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.815 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:20.816 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.816 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.817 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.817 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:20.818 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.818 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.819 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.819 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.820 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:20.821 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.821 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.822 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:20.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.823 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.825 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:20.826 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.827 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.827 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.828 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.829 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:20.830 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.830 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.831 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.833 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.834 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.835 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.836 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.836 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.837 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:20.838 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.839 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:20.839 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:20.840 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:20.841 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.842 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:53:42Z" 2022-09-13T19:12:20.843 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:20.843 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:20.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:20.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:20.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.848 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:20.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:20.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:20.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:47.931+0000 7f098e11a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:47.932+0000 7f098e11a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:20.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:20.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:20.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:20.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:20.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:20.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:20.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:20.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:20.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:20.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:20.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:20.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:20.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:20.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:20.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:20.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:20.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:20.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:53:47Z" 2022-09-13T19:12:20.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:20.888 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:20.889 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:20.890 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:20.890 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:20.891 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.891 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.891 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.892 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:20.892 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.893 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:20.893 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.894 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.894 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.895 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:20.896 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.897 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.898 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.898 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.899 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.900 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.901 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:20.901 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.902 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.903 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:20.904 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.905 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.906 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:20.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.907 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.908 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.909 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.910 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:20.911 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.911 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.912 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:20.912 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.913 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.914 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.914 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:20.915 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.915 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.915 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.916 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.916 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:20.917 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.917 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.918 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.919 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.920 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.920 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.921 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.922 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.922 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:20.923 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.924 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:20.924 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:20.925 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:20.925 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.926 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:53:47Z" 2022-09-13T19:12:20.927 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:20.928 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:20.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:20.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:20.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:20.931 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:20.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:20.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:20.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:20.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:53.073+0000 7ff1589e8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:53.075+0000 7ff1589e8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:20.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:20.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:20.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:20.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:20.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:20.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:20.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:20.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:20.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:20.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:20.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:20.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:20.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:20.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:20.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:20.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:20.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:20.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:20.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:20.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:20.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:20.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:20.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:20.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:20.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:20.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:20.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:53:53Z" 2022-09-13T19:12:20.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:20.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:20.972 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:20.972 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:20.972 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:20.973 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:20.973 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:20.974 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.974 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.975 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:20.975 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.975 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:20.976 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.976 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.977 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:20.977 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.978 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:20.978 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.978 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:20.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.979 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.980 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.980 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.981 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:20.981 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.982 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:20.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.983 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.983 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.984 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:20.984 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.985 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.985 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.986 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.986 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:20.986 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.987 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.987 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:20.988 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.988 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.989 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.989 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:20.990 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.990 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.991 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.991 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:20.992 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:20.992 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:20.992 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:20.993 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:20.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:20.994 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:20.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:20.995 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:20.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:20.995 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.996 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:20.996 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:20.997 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.997 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:20.998 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:20.998 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:20.998 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:20.999 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:53:53Z" 2022-09-13T19:12:20.999 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:21.000 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:21.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:21.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.858000_209.1 2022-09-13T19:12:21.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.003 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:21.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:21.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:21.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:58.313+0000 7efdda06d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:53:58.314+0000 7efdda06d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:21.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:21.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:21.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:21.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:21.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:21.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:21.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:21.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:21.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:21.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:21.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:21.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:21.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:21.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:21.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:21.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:21.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:21.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:53:58Z" 2022-09-13T19:12:21.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:21.035 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:21.035 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:21.035 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:21.036 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:21.036 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.037 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.038 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:21.038 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.039 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:21.039 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.039 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.040 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.040 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.041 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:21.041 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.042 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.042 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.043 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.043 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.043 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.044 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:21.044 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.045 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.045 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:21.046 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.046 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.046 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.047 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:21.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.048 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.048 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.048 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.049 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:21.049 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.050 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.050 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:21.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.051 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.052 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:21.052 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.053 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.053 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.053 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.054 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:21.054 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.055 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.055 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.056 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.056 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.056 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.057 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.057 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.058 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.058 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.059 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:21.059 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.059 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:21.060 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:21.060 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:21.061 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.061 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:53:58Z" 2022-09-13T19:12:21.062 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:21.062 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:21.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:21.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:21.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.065 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:21.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:21.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:21.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:03.484+0000 7f2407785600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:03.485+0000 7f2407785600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:21.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:21.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:21.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:21.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:21.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:21.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:21.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:21.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:21.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:21.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:21.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:21.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:21.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:21.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:21.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:21.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:21.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:21.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:54:03Z" 2022-09-13T19:12:21.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:21.096 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:21.096 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:21.096 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:21.097 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:21.097 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.098 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.098 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.098 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:21.099 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.099 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:21.100 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.100 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.101 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.101 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.101 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:21.102 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.102 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.103 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.103 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.103 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.104 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.104 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:21.105 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.105 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.106 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:21.106 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.107 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.107 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.107 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:21.108 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.108 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.109 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.109 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.109 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:21.110 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.110 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.111 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:21.111 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.112 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.112 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.112 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:21.113 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.113 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.114 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.114 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.115 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:21.115 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.116 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.116 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.117 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.117 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.118 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.118 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.119 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.119 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.119 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.120 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:21.120 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.121 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:21.121 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:21.121 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:21.122 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.122 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:54:03Z" 2022-09-13T19:12:21.123 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:21.123 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:21.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:21.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.900410_211.1 2022-09-13T19:12:21.126 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:21.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:21.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:21.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:08.632+0000 7fe553e20600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:08.633+0000 7fe553e20600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:21.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:21.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:21.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:21.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:21.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:21.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:21.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:21.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:21.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:21.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:21.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:21.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:21.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:21.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:21.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:21.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:21.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:21.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:54:08Z" 2022-09-13T19:12:21.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:21.158 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:21.158 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:21.159 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:21.159 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:21.160 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.160 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.161 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.161 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:21.162 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.162 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:21.162 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.163 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.163 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.164 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.164 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:21.165 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.165 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.165 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.166 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.166 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.167 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.167 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:21.168 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.168 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.169 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:21.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.170 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.170 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.171 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:21.171 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.172 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.172 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.173 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.173 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:21.173 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.174 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.174 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:21.175 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.175 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.176 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.176 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:21.176 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.177 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.177 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.178 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.178 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:21.179 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.179 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.180 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.180 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.181 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.181 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.182 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.182 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.183 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.183 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:21.184 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.184 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:21.185 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:21.185 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:21.185 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.186 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:54:08Z" 2022-09-13T19:12:21.186 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:21.187 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:21.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:21.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.854520_207.1 2022-09-13T19:12:21.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.190 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:21.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:21.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:21.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:13.778+0000 7f1643e64600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:13.779+0000 7f1643e64600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:21.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:21.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:21.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:21.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:21.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:21.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:21.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:21.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:21.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:21.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:21.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:21.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:21.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:21.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:21.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:21.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:21.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:21.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:54:13Z" 2022-09-13T19:12:21.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:21.223 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:21.224 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:21.224 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:21.225 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:21.225 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.226 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.226 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.227 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:21.227 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.227 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:21.228 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.228 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.229 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.229 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.230 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:21.230 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.230 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.231 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.231 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.232 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.232 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.233 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:21.234 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.234 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.235 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:21.235 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.235 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.236 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.236 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:21.237 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.237 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.238 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.238 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.239 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:21.239 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.240 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.240 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:21.241 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.242 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.242 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.243 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:21.243 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.244 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.244 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.244 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.245 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:21.245 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.246 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.246 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.247 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.247 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.248 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.248 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.249 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.249 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.250 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.250 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:21.251 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.251 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:21.252 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:21.252 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:21.253 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.253 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:54:13Z" 2022-09-13T19:12:21.254 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:21.254 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:21.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:21.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:21.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:21.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:21.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:21.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:18.922+0000 7f30f33be600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:18.923+0000 7f30f33be600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:21.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:21.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:21.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:21.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:21.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:21.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:21.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:21.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:21.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:21.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:21.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:21.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:21.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:21.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:21.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:21.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:21.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:21.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:54:18Z" 2022-09-13T19:12:21.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:21.292 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:21.292 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:21.293 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:21.293 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:21.293 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.294 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.294 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.295 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:21.295 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.296 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:21.296 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.297 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.297 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.298 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.298 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:21.299 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.299 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.300 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.300 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.301 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.301 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.302 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:21.302 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.303 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.303 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:21.304 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.304 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.304 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.305 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:21.305 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.306 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.307 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.307 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.307 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:21.308 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.308 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.309 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:21.309 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.310 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.310 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.311 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:21.311 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.312 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.312 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.313 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.313 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:21.313 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.314 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.315 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.315 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.315 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.316 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.316 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.317 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.317 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.318 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.318 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:21.318 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.319 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:21.319 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:21.320 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:21.320 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.321 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:54:18Z" 2022-09-13T19:12:21.321 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:21.322 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:21.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:21.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:21.324 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.325 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:21.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:21.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:21.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:24.116+0000 7f07898e8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:24.117+0000 7f07898e8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:21.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:21.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:21.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:21.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:21.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:21.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:21.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:21.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:21.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:21.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:21.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:21.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:21.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:21.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:21.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:21.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:21.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:21.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:54:24Z" 2022-09-13T19:12:21.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:21.378 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:21.378 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:21.379 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:21.379 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:21.380 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.380 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.381 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.381 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:21.382 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.382 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:21.382 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.383 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.383 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.384 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:21.385 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.385 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.385 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.386 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.386 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.387 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.387 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:21.387 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.388 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.388 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:21.389 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.389 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.390 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.390 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:21.390 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.391 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.391 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.392 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.392 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:21.393 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.393 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.393 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:21.394 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.394 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.395 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.395 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:21.396 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.396 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.396 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.397 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.397 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:21.398 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.398 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.398 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.399 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.399 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.400 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.400 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.401 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.401 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.401 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.402 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:21.402 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.403 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:21.403 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:21.404 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:21.404 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.404 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:54:24Z" 2022-09-13T19:12:21.405 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:21.405 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:21.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:21.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.857076_208.1 2022-09-13T19:12:21.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:21.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:21.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:21.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:29.234+0000 7f4292868600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:29.235+0000 7f4292868600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:21.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:21.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:21.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:21.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:21.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:21.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:21.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:21.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:21.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:21.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:21.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:21.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:21.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:21.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:21.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:21.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:21.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:21.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:54:29Z" 2022-09-13T19:12:21.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:21.440 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:21.441 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:21.441 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:21.445 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:21.446 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.446 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.447 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.447 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:21.448 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.448 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:21.449 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.449 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.450 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.450 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.451 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:21.451 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.452 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.452 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.453 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.453 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.454 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.454 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:21.455 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.455 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.455 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:21.456 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.456 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.457 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.457 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:21.458 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.459 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.459 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.460 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.460 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:21.461 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.461 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.462 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:21.462 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.463 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.463 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:21.464 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.465 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.465 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.466 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.466 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:21.467 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.467 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.468 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.468 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.469 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.469 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.469 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.470 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.470 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.471 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.471 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:21.472 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.472 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:21.472 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:21.473 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:21.473 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.474 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:54:29Z" 2022-09-13T19:12:21.474 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:21.475 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:21.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:21.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:21.477 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.477 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:21.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:21.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:21.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:34.361+0000 7f415d91f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:34.362+0000 7f415d91f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:21.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:21.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:21.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:21.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:21.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:21.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:21.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:21.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:21.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:21.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:21.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:21.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:21.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:21.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:21.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:21.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:21.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:21.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:54:34Z" 2022-09-13T19:12:21.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:21.508 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:21.508 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:21.509 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:21.509 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:21.510 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.510 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.510 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.511 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:21.511 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.512 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:21.512 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.513 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.513 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.513 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.514 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:21.514 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.515 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.515 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.516 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.516 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.516 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.517 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:21.517 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.518 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.518 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:21.518 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.519 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.519 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.520 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:21.520 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.521 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.521 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.521 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.522 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:21.522 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.523 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.523 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:21.523 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.524 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.524 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.525 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:21.525 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.526 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.526 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.527 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.527 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:21.527 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.528 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.528 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.529 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.529 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.530 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.530 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.530 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.531 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.531 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.532 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:21.532 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.533 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:21.533 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:21.534 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:21.534 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.534 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:54:34Z" 2022-09-13T19:12:21.535 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:21.535 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:21.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:21.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.854520_207.1 2022-09-13T19:12:21.537 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:21.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:21.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:21.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:39.509+0000 7f4fa5eca600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:39.511+0000 7f4fa5eca600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:21.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:21.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:21.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:21.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:21.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:21.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:21.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:21.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:21.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:21.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:21.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:21.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:21.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:21.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:21.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:21.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:21.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:21.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:54:39Z" 2022-09-13T19:12:21.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:21.570 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:21.571 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:21.571 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:21.571 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:21.572 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.572 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.573 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.573 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:21.574 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.574 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:21.575 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.575 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.576 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.576 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.576 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:21.577 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.577 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.578 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.578 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.579 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.579 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.580 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:21.580 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.581 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.581 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:21.581 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.582 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.582 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.583 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:21.583 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.584 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.584 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.584 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.585 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:21.585 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.586 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.586 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:21.587 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.587 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.587 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.588 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:21.588 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.589 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.589 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.589 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.590 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:21.590 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.591 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.591 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.592 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.592 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.592 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.593 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.593 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.594 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.594 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.595 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:21.595 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.595 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:21.596 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:21.596 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:21.597 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.597 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:54:39Z" 2022-09-13T19:12:21.597 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:21.598 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:21.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:21.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:21.600 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.600 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:21.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:21.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:21.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:44.678+0000 7f87f004f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:44.687+0000 7f87f004f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:21.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:21.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:21.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:21.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:21.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:21.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:21.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:21.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:21.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:21.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:21.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:21.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:21.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:21.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:21.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:21.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:21.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:21.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:54:44Z" 2022-09-13T19:12:21.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:21.634 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:21.635 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:21.635 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:21.636 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:21.637 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.637 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.638 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.639 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:21.640 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.641 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:21.641 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.642 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.642 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.643 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:21.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.644 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.644 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.645 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.645 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.645 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.646 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:21.646 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.647 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.647 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:21.648 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.649 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.649 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.650 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:21.650 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.651 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.651 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.652 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.652 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:21.653 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.654 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:21.654 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.654 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.655 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.656 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:21.656 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.657 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.657 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.658 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.658 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:21.659 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.659 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.660 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.660 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.661 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.662 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.662 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.663 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.663 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.664 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.664 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:21.665 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.666 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:21.666 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:21.667 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:21.668 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.668 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:54:44Z" 2022-09-13T19:12:21.669 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:21.670 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:21.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:21.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.861945_210.1 2022-09-13T19:12:21.674 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.674 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:21.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:21.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:21.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:49.869+0000 7f0553f87600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:49.870+0000 7f0553f87600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:21.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:21.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:21.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:21.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:21.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:21.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:21.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:21.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:21.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:21.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:21.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.906479_212.1", 2022-09-13T19:12:21.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.906479Z", 2022-09-13T19:12:21.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:21.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:21.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:21.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:21.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:21.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:54:49Z" 2022-09-13T19:12:21.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:21.722 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:21.722 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:21.723 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:21.723 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:21.724 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.724 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.725 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.725 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:21.725 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.726 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:21.727 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.727 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.728 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.728 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:21.729 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.730 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.730 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.731 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.731 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.731 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.732 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:21.732 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.733 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.733 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:21.734 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.734 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.735 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.735 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:21.736 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.736 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.736 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.737 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.738 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:21.738 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.739 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.739 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.906479_212.1", 2022-09-13T19:12:21.739 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.740 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.740 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.741 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.906479Z", 2022-09-13T19:12:21.741 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.742 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.742 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.742 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.743 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:21.743 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.744 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.744 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.744 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.745 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.745 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.746 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.746 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.747 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.748 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.749 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:21.749 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.749 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:21.750 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:21.750 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:21.751 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.751 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:54:49Z" 2022-09-13T19:12:21.751 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:21.752 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:21.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.906479_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.906479Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:21.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.906479_212.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.906479_212.1 2022-09-13T19:12:21.756 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.756 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:21.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:21.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:21.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:55.004+0000 7fc2288af600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:54:55.005+0000 7fc2288af600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:21.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:21.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:21.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:21.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:21.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:21.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:21.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:21.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:21.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:21.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:21.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:21.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:21.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:21.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:21.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:21.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:21.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:21.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:54:55Z" 2022-09-13T19:12:21.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:21.799 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:21.799 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:21.800 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:21.801 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:21.801 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.802 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.803 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.803 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:21.804 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.804 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:21.805 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.805 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.805 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.806 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.807 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:21.807 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.808 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.809 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.809 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.810 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.811 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.812 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:21.812 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.813 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.814 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:21.815 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.816 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.816 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.817 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:21.818 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.819 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.819 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.820 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.821 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:21.821 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.822 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.822 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:21.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.824 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.824 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:21.825 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.826 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.826 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.827 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.828 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:21.829 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.829 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.830 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.831 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.832 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.834 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.834 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.835 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.836 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.837 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:21.837 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.838 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:21.839 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:21.840 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:21.841 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.842 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:54:55Z" 2022-09-13T19:12:21.842 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:21.843 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:21.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:21.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.953175_216.1 2022-09-13T19:12:21.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.848 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:21.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:21.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:21.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:00.148+0000 7fdba8fc5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:00.149+0000 7fdba8fc5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:21.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:21.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:21.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:21.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:21.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:21.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:21.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:21.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:21.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:21.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:21.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:21.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:21.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:21.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:21.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:21.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:21.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:21.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:55:00Z" 2022-09-13T19:12:21.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:21.893 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:21.893 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:21.893 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:21.894 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:21.895 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.895 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.895 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.896 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:21.896 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.897 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:21.897 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.898 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.898 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.899 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.899 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:21.900 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.900 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.901 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.901 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.901 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.902 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.902 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:21.903 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.903 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.904 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:21.904 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.904 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.905 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:21.906 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.906 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.907 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.907 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.908 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:21.908 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.909 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.909 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:21.910 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.910 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.911 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.911 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:21.912 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.912 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.913 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.913 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.914 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:21.914 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.914 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.915 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.915 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.916 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.916 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.917 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.918 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.919 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.919 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.920 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:21.920 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.921 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:21.921 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:21.922 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:21.922 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.923 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:55:00Z" 2022-09-13T19:12:21.923 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:21.923 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:21.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:21.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:21.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.990623_217.1 2022-09-13T19:12:21.926 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.927 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:21.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:21.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:21.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:21.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:05.290+0000 7f20c6ca0600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:05.291+0000 7f20c6ca0600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:21.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:21.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:21.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:21.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:21.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:21.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:21.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:21.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:21.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:21.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:21.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:21.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:21.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:21.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:21.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:21.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:21.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:21.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:21.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:21.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:21.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:21.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:21.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:21.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:21.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:21.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:21.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:55:05Z" 2022-09-13T19:12:21.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:21.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:21.969 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:21.969 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:21.970 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:21.971 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:21.971 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:21.972 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.973 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.973 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:21.974 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.974 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:21.974 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.975 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.975 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:21.976 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.976 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:21.977 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.977 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:21.978 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.978 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.978 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.979 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.979 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:21.980 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.980 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.981 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:21.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.982 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.982 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:21.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.983 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.984 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.984 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.985 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:21.985 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.986 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.986 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:21.987 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.987 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.988 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.988 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:21.989 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.989 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.990 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.990 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:21.991 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:21.991 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:21.991 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:21.992 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:21.992 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:21.993 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:21.993 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:21.994 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:21.994 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:21.995 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.995 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:21.995 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:21.996 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.996 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:21.997 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:21.997 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:21.998 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:21.998 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:55:05Z" 2022-09-13T19:12:21.998 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:21.999 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:21.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:22.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.861945_210.1 2022-09-13T19:12:22.001 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:22.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:22.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:22.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:10.474+0000 7f6c8e182600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:10.476+0000 7f6c8e182600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:22.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:22.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:22.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:22.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:22.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:22.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:22.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:22.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:22.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:22.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:22.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:22.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:22.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:22.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:22.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:22.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:22.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:22.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:55:10Z" 2022-09-13T19:12:22.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:22.034 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:22.035 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:22.035 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:22.036 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:22.036 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.037 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.038 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:22.038 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.039 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:22.039 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.040 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.040 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.041 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.041 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:22.041 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.042 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.042 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.043 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.043 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.044 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.044 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:22.045 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.045 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.045 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:22.046 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.046 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.047 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.047 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:22.048 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.048 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.048 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.049 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.049 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:22.050 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.050 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.051 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:22.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.052 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.052 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.052 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:22.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.053 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.054 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.054 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.055 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:22.055 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.056 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.056 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.056 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.057 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.057 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.058 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.058 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.059 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.059 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.059 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:22.060 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.060 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:22.061 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:22.061 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:22.062 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.062 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:55:10Z" 2022-09-13T19:12:22.062 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:22.063 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:22.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.953175_216.1 2022-09-13T19:12:22.065 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.065 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:22.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:15.617+0000 7fa46b071600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:15.618+0000 7fa46b071600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:22.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:22.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:22.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:22.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:22.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:22.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:22.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:22.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:22.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.994620_218.1", 2022-09-13T19:12:22.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.994620Z", 2022-09-13T19:12:22.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:22.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:22.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:22.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:22.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:22.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:55:15Z" 2022-09-13T19:12:22.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:22.103 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:22.103 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:22.104 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:22.104 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:22.105 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.106 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.107 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.107 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:22.108 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.109 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:22.110 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.111 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.111 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.112 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.113 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:22.114 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.114 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.115 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.115 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.115 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.116 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.116 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:22.117 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.117 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.118 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:22.118 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.119 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.119 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.119 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:22.120 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.120 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.121 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.121 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.122 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:22.122 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.123 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.123 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.994620_218.1", 2022-09-13T19:12:22.124 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.124 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.125 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.126 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.994620Z", 2022-09-13T19:12:22.126 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.127 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.127 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.128 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.128 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:22.128 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.129 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.129 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.130 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.130 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.131 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.131 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.132 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.132 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.133 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.133 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:22.134 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.134 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:22.135 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:22.135 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:22.135 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.136 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:55:15Z" 2022-09-13T19:12:22.136 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:22.137 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.994620_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.994620Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:22.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055411.238167_296.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.994620_218.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055411.238167_296.1 target=1_1663055098.994620_218.1 2022-09-13T19:12:22.139 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:22.140 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.140 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:22.141 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_datalog_autotrim 2022-09-13T19:12:22.141 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:22.144 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:22.144 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:22.145 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:22.145 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 984, in test_datalog_autotrim 2022-09-13T19:12:22.146 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2022-09-13T19:12:22.146 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:22.146 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2022-09-13T19:12:22.147 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 196, in zone_meta_checkpoint 2022-09-13T19:12:22.147 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2022-09-13T19:12:22.148 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2022-09-13T19:12:22.148 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:22.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=yxvvuh-42 2022-09-13T19:12:22.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-42/ 2022-09-13T19:12:22.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-42/ 2022-09-13T19:12:22.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:22.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-42/ 2022-09-13T19:12:22.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:22.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:22.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:22.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:22.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:22.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:22.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:22.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:22.155 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:22.155 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.156 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.156 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:55:20 GMT 2022-09-13T19:12:22.157 INFO:tasks.rgw_multisite_tests:/yxvvuh-42/ 2022-09-13T19:12:22.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:22.157 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:lqxlXU5oIW3YMtlnnT1V5sULMUM= 2022-09-13T19:12:22.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lqxlXU5oIW3YMtlnnT1V5sULMUM=', 'Content-Length': '0'} 2022-09-13T19:12:22.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000008edf101371b6fe3a-0063203768-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:22.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=yxvvuh-43 2022-09-13T19:12:22.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-43/ 2022-09-13T19:12:22.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-43/ 2022-09-13T19:12:22.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:22.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-43/ 2022-09-13T19:12:22.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:22.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:22.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:22.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:22.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:22.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:22.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:22.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:22.165 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:22.166 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.166 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.167 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:55:20 GMT 2022-09-13T19:12:22.167 INFO:tasks.rgw_multisite_tests:/yxvvuh-43/ 2022-09-13T19:12:22.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:22.168 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:s7aclqHmfZ5cHyWE3dTtBRwnUIg= 2022-09-13T19:12:22.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:s7aclqHmfZ5cHyWE3dTtBRwnUIg=', 'Content-Length': '0'} 2022-09-13T19:12:22.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000076f9250361f73fca-0063203768-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:22.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=yxvvuh-44 2022-09-13T19:12:22.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-44/ 2022-09-13T19:12:22.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-44/ 2022-09-13T19:12:22.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:22.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-44/ 2022-09-13T19:12:22.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:22.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:22.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:22.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:22.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:22.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-13T19:12:22.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:22.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:22.175 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:22.175 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.176 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.176 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:55:20 GMT 2022-09-13T19:12:22.176 INFO:tasks.rgw_multisite_tests:/yxvvuh-44/ 2022-09-13T19:12:22.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:22.177 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:Pi2VJBjexyAetMoqh/6mk/6wgXM= 2022-09-13T19:12:22.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Pi2VJBjexyAetMoqh/6mk/6wgXM=', 'Content-Length': '0'} 2022-09-13T19:12:22.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000d0c0f03f92b95778-0063203768-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:22.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-42/ 2022-09-13T19:12:22.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-42/ 2022-09-13T19:12:22.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:22.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-42/ 2022-09-13T19:12:22.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:22.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:22.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:22.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:22.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:22.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:22.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:22.183 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:22.184 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.184 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.184 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:55:20 GMT 2022-09-13T19:12:22.185 INFO:tasks.rgw_multisite_tests:/yxvvuh-42/ 2022-09-13T19:12:22.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:22.186 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:CpfA5K/OCkrRH5QXw3NUlvpZKO4= 2022-09-13T19:12:22.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:CpfA5K/OCkrRH5QXw3NUlvpZKO4=', 'Content-Length': '0'} 2022-09-13T19:12:22.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000effb8abb96b7d4fe-0063203768-10ed-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:22.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-42/key 2022-09-13T19:12:22.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-42/key 2022-09-13T19:12:22.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:22.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-42/key 2022-09-13T19:12:22.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:22.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-09-13T19:12:22.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:22.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:22.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:22.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:22.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:22.192 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:22.192 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2022-09-13T19:12:22.193 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:22.193 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:55:20 GMT 2022-09-13T19:12:22.194 INFO:tasks.rgw_multisite_tests:/yxvvuh-42/key 2022-09-13T19:12:22.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:22.195 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:FQ4oWajkhlINkDdn7fZltEQD35M= 2022-09-13T19:12:22.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:FQ4oWajkhlINkDdn7fZltEQD35M='} 2022-09-13T19:12:22.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca8f3101537b0d8c-0063203768-10ed-test-zone1'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:22.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-43/ 2022-09-13T19:12:22.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-43/ 2022-09-13T19:12:22.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:22.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-43/ 2022-09-13T19:12:22.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:22.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:22.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:22.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:22.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:22.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:22.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:22.205 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:22.206 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.207 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.207 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:55:20 GMT 2022-09-13T19:12:22.208 INFO:tasks.rgw_multisite_tests:/yxvvuh-43/ 2022-09-13T19:12:22.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:22.209 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:lCIw+DQBs08CZAg9rRu2OdkrqK4= 2022-09-13T19:12:22.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:lCIw+DQBs08CZAg9rRu2OdkrqK4=', 'Content-Length': '0'} 2022-09-13T19:12:22.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f4f45dcfdaa9a3c-0063203768-10c9-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:22.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-43/key 2022-09-13T19:12:22.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-43/key 2022-09-13T19:12:22.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:22.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-43/key 2022-09-13T19:12:22.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:22.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-09-13T19:12:22.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:22.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:22.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:22.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:22.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:22.216 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:22.216 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2022-09-13T19:12:22.217 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:22.217 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:55:20 GMT 2022-09-13T19:12:22.218 INFO:tasks.rgw_multisite_tests:/yxvvuh-43/key 2022-09-13T19:12:22.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:22.219 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:K/qNJQIRJ0eIl1+1mk8EcT4AiEg= 2022-09-13T19:12:22.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:K/qNJQIRJ0eIl1+1mk8EcT4AiEg='} 2022-09-13T19:12:22.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a98329e91e8276e-0063203768-10c9-test-zone2'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:22.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-44/ 2022-09-13T19:12:22.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-44/ 2022-09-13T19:12:22.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:22.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-44/ 2022-09-13T19:12:22.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:22.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:22.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:22.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:22.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:22.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:22.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:22.228 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:22.228 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.229 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.230 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:55:20 GMT 2022-09-13T19:12:22.231 INFO:tasks.rgw_multisite_tests:/yxvvuh-44/ 2022-09-13T19:12:22.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:22.232 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:JGvc3iKZ7YsLL0yItktM90Guz7k= 2022-09-13T19:12:22.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:JGvc3iKZ7YsLL0yItktM90Guz7k=', 'Content-Length': '0'} 2022-09-13T19:12:22.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000644c6fe21b5aac73-0063203768-10f3-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:22.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-44/key 2022-09-13T19:12:22.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-44/key 2022-09-13T19:12:22.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:22.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-44/key 2022-09-13T19:12:22.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:22.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-09-13T19:12:22.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:22.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:22.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:22.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:22.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:22.243 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:22.244 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2022-09-13T19:12:22.244 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:22.245 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 07:55:20 GMT 2022-09-13T19:12:22.245 INFO:tasks.rgw_multisite_tests:/yxvvuh-44/key 2022-09-13T19:12:22.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:22.246 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:YdVWUqTMqRV5JCaPP/hfq28sqTs= 2022-09-13T19:12:22.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 07:55:20 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:YdVWUqTMqRV5JCaPP/hfq28sqTs='} 2022-09-13T19:12:22.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e9b49658f732d00-0063203768-10f3-test-zone3'), ('Date', 'Tue, 13 Sep 2022 07:55:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:22.248 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.248 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:22.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:22.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:22.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:22.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:22.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663055720.804779_308.1", 2022-09-13T19:12:22.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:55:20.804779Z" 2022-09-13T19:12:22.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:22.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:22.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:22.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:22.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:22.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:22.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:22.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:22.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:21.092+0000 7f7ff4dcd600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:21.093+0000 7f7ff4dcd600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:22.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:22.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:22.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:22.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:22.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:22.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:22.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:22.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:22.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:22.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:22.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:55:21Z" 2022-09-13T19:12:22.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:22.300 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:22.301 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:22.301 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:22.302 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:22.302 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.303 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.303 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.303 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:22.304 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.304 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:22.305 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.305 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.305 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.306 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.306 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:22.307 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.307 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.308 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.308 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.308 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.309 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.309 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:22.310 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.310 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.311 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:22.311 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.311 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.312 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.312 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:22.313 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.313 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.313 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.314 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.314 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:22.315 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.315 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.316 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:22.316 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.316 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.317 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.317 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:22.318 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.318 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.318 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.319 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.319 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:22.320 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.320 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.321 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.321 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.321 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.322 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.322 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.323 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.323 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.324 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.324 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:22.324 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.325 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:22.325 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:22.326 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:22.326 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.327 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:55:21Z" 2022-09-13T19:12:22.327 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:22.328 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:22.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.854520_207.1 2022-09-13T19:12:22.330 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.330 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:22.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:26.218+0000 7fc28550d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:26.219+0000 7fc28550d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:22.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:22.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:22.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:22.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:22.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:22.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:22.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:22.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:22.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:22.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:22.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:22.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:55:26Z" 2022-09-13T19:12:22.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:22.362 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:22.363 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:22.364 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:22.365 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:22.366 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.366 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.366 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.367 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:22.367 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.368 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:22.369 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.369 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.370 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.371 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.372 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:22.372 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.373 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.375 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.376 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.376 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.377 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.378 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:22.379 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.379 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.380 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:22.381 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.382 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.382 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.383 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:22.384 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.385 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.385 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.386 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.387 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:22.388 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.389 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.389 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:22.390 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.391 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.392 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:22.393 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.394 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.395 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.395 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.396 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:22.397 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.398 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.398 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.399 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.400 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.401 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.401 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.402 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.403 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.404 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.404 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:22.405 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.406 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:22.407 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:22.408 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:22.409 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.409 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:55:26Z" 2022-09-13T19:12:22.410 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:22.411 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:22.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.854520_207.1 2022-09-13T19:12:22.414 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.414 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:22.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:22.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:22.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:31.355+0000 7f880f69d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:31.357+0000 7f880f69d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:22.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:22.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:22.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:22.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:22.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:22.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:22.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:22.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:22.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:22.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:55:31Z" 2022-09-13T19:12:22.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:22.445 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:22.445 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:22.446 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:22.446 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:22.446 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.447 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.447 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.448 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:22.448 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.449 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:22.449 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.449 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.450 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.450 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.451 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:22.451 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.452 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.452 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.452 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.453 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.453 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.454 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:22.454 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.454 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.455 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:22.455 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.456 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.457 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.457 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:22.458 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.458 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.458 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.459 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.459 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:22.460 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.460 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.461 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:22.461 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.461 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.462 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.462 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:22.463 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.463 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.464 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.464 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.464 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:22.465 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.465 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.466 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.466 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.466 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.467 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.467 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.468 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.468 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.468 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.469 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:22.469 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.470 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:22.470 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:22.470 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:22.471 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.471 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:55:31Z" 2022-09-13T19:12:22.472 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:22.472 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:22.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:22.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:22.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:22.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:22.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:36.592+0000 7fba184b9600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:36.593+0000 7fba184b9600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:22.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:22.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:22.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:22.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:22.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:22.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:22.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:22.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:55:36Z" 2022-09-13T19:12:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:22.504 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:22.505 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:22.505 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:22.505 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:22.506 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.506 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.507 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.507 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:22.507 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.508 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:22.508 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.509 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.509 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.509 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.510 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:22.510 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.511 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.511 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.511 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.512 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.512 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.513 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:22.513 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.513 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.514 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:22.514 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.515 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.515 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.516 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:22.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.516 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.517 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.517 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.518 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:22.518 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.518 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.519 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:22.519 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.520 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.520 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.520 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:22.521 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.521 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.522 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.522 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.523 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:22.523 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.523 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.524 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.524 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.525 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.525 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.526 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.526 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.526 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.527 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.527 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:22.528 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.528 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:22.529 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:22.529 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:22.530 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.530 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:55:36Z" 2022-09-13T19:12:22.530 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:22.531 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:22.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.857076_208.1 2022-09-13T19:12:22.533 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.533 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:22.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:41.728+0000 7fbc7190b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:41.729+0000 7fbc7190b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:22.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:22.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:22.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:22.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:22.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:22.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:22.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:22.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:22.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:22.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:22.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:22.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:55:41Z" 2022-09-13T19:12:22.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:22.563 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:22.564 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:22.564 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:22.565 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:22.565 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.565 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.566 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.566 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:22.567 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.567 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:22.568 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.568 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.568 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.569 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.569 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:22.570 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.570 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.570 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.571 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.571 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.572 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.572 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:22.572 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.573 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.573 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:22.574 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.574 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.575 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.575 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:22.575 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.576 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.576 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.577 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.577 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:22.577 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.578 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.578 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:22.579 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.579 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.579 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.580 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:22.580 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.581 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.581 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.581 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.582 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:22.582 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.583 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.583 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.584 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.584 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.584 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.585 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.585 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.586 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.586 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.586 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:22.587 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.587 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:22.588 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:22.588 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:22.588 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.589 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:55:41Z" 2022-09-13T19:12:22.589 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:22.590 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:22.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.858000_209.1 2022-09-13T19:12:22.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:22.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:22.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:22.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:46.908+0000 7f5a66b04600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:46.909+0000 7f5a66b04600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:22.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:22.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:22.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:22.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:22.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.994620_218.1", 2022-09-13T19:12:22.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.994620Z", 2022-09-13T19:12:22.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:22.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:22.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:22.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:22.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:22.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:55:46Z" 2022-09-13T19:12:22.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:22.622 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:22.622 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:22.623 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:22.623 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:22.623 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.624 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.624 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.625 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:22.625 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.625 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:22.626 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.626 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.627 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.627 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.628 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:22.628 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.628 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.629 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.629 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.630 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.630 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.630 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:22.631 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.631 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.632 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:22.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.632 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.633 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:22.634 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.634 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.635 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.635 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.635 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:22.636 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.637 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.637 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.994620_218.1", 2022-09-13T19:12:22.637 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.638 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.638 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.639 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.994620Z", 2022-09-13T19:12:22.639 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.639 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.640 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.640 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.641 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:22.641 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.641 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.642 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.642 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.643 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.643 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.644 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.644 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.645 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.645 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:22.646 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.646 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:22.646 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:22.647 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:22.647 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.648 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:55:46Z" 2022-09-13T19:12:22.648 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:22.648 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.994620_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.994620Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:22.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.994620_218.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.994620_218.1 2022-09-13T19:12:22.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.651 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:22.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:22.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:22.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:52.062+0000 7fbd5d94b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:52.063+0000 7fbd5d94b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:22.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:22.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:22.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:22.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:22.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:22.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:22.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:22.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:55:52Z" 2022-09-13T19:12:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:22.681 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:22.681 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:22.681 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:22.682 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:22.682 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.683 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.683 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.683 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:22.684 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.684 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:22.685 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.685 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.685 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.686 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.686 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:22.687 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.687 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.687 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.688 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.688 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.689 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.689 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:22.690 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.690 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.690 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:22.691 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.691 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.692 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.692 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:22.692 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.693 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.693 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.694 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.694 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:22.694 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.695 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.695 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:22.696 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.696 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.696 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.697 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:22.697 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.698 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.698 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.698 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.699 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:22.699 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.700 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.700 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.700 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.701 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.701 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.702 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.702 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.702 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.703 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.703 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:22.704 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.704 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:22.705 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:22.705 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:22.705 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.706 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:55:52Z" 2022-09-13T19:12:22.706 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:22.707 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:22.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.861945_210.1 2022-09-13T19:12:22.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:22.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:22.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:22.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:57.199+0000 7f3843b0a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:55:57.200+0000 7f3843b0a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:22.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:22.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:22.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:22.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:22.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:22.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:22.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:22.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:22.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:22.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:22.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:22.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:22.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:55:57Z" 2022-09-13T19:12:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:22.750 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:22.750 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:22.751 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:22.752 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:22.753 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.754 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.755 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.755 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:22.756 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.756 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:22.757 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.757 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.758 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.768 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:22.769 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.770 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.770 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.771 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.772 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.773 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.774 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:22.775 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.775 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.776 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:22.777 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.778 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.778 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.779 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:22.780 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.781 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.782 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.782 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.783 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:22.784 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.785 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.785 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:22.786 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.787 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.787 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.788 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:22.789 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.790 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.790 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.791 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.792 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:22.793 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.793 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.794 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.795 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.796 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.796 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.797 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.798 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.799 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.799 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.800 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:22.801 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.802 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:22.803 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:22.803 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:22.804 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.804 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:55:57Z" 2022-09-13T19:12:22.805 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:22.806 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:22.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:22.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:02.440+0000 7fa335e6f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:02.449+0000 7fa335e6f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:22.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:22.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:22.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:22.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:22.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:22.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:22.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:22.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:22.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:22.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:22.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:22.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:22.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:22.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:56:02Z" 2022-09-13T19:12:22.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:22.842 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:22.843 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:22.843 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:22.844 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:22.844 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.845 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.845 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.846 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:22.846 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.847 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:22.847 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.848 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.848 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.848 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.849 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:22.849 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.850 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.850 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.851 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.851 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.852 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.852 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:22.853 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.853 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.853 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:22.854 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.855 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.855 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.855 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:22.856 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.856 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.857 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.857 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.858 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:22.858 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.858 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.859 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:22.859 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.860 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.861 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.861 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:22.861 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.862 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.862 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.863 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.863 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:22.864 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.864 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.864 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.865 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.865 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.866 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.866 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.867 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.868 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.868 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:22.868 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.869 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:22.869 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:22.870 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:22.870 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.870 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:56:02Z" 2022-09-13T19:12:22.871 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:22.871 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:22.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:22.873 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.874 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:22.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:07.605+0000 7f6483a87600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:07.606+0000 7f6483a87600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:22.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:22.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:22.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:22.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:22.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:22.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:22.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:22.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:22.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:22.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:22.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:22.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:22.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:56:07Z" 2022-09-13T19:12:22.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:22.910 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:22.911 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:22.912 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:22.912 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:22.913 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.914 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.915 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.915 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:22.916 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.917 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:22.918 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.919 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.919 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.920 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.921 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:22.922 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.923 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.923 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.924 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.925 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.926 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.926 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:22.927 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.927 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.928 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:22.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.929 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.929 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.930 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:22.930 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.930 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.931 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.931 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.932 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:22.932 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.933 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.933 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:22.933 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.934 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.935 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.936 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:22.936 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.937 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.938 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.939 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:22.939 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:22.940 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:22.941 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:22.942 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.942 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:22.943 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:22.944 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:22.945 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.945 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:22.946 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.947 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:22.948 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:22.948 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.949 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:22.949 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:22.950 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:22.950 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:22.951 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:56:07Z" 2022-09-13T19:12:22.951 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:22.952 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:22.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:22.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:22.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:22.955 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.955 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:22.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:22.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:12.760+0000 7fede5cb8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:12.761+0000 7fede5cb8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:22.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:22.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:22.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:22.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:22.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:22.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:22.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:22.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:22.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:22.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:22.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:22.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:22.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:22.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:22.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:22.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:22.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:22.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:22.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:22.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:22.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:22.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:22.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:22.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:22.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:22.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:22.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:22.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:56:12Z" 2022-09-13T19:12:23.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.002 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.003 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.004 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.005 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.005 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.006 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.007 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.007 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.008 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.009 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.009 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.010 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.011 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.012 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.012 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.013 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.013 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.014 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.014 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.015 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.015 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.016 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.016 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.016 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.017 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.017 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.018 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.018 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.019 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.019 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.020 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.020 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.021 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.021 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.021 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.022 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.022 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:23.023 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.023 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.024 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.024 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:23.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.025 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.025 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.026 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.026 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.026 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.027 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.027 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.028 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.028 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.029 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.029 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.030 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.030 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.030 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.031 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:23.031 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.032 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:23.032 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:23.033 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:23.033 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.033 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:56:12Z" 2022-09-13T19:12:23.034 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:23.034 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:23.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:23.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.854520_207.1 2022-09-13T19:12:23.036 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.037 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:23.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:23.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:17.928+0000 7f124924d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:17.929+0000 7f124924d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:23.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:23.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:23.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:23.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:23.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:23.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:23.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:23.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:23.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:23.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:23.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:23.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:56:17Z" 2022-09-13T19:12:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.069 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.070 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.070 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.070 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.071 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.071 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.072 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.072 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.073 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.073 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.073 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.075 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.076 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.077 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.077 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.078 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.078 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.079 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.079 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.079 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.080 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.080 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.081 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.081 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.082 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.082 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.082 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.083 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.083 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.084 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.084 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.085 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:23.085 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.085 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.086 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.086 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:23.087 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.087 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.088 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.088 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.089 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.089 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.090 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.090 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.091 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.091 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.092 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.092 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.092 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.093 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.093 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.094 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:23.095 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.095 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:23.096 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:23.097 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:23.098 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.099 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:56:17Z" 2022-09-13T19:12:23.099 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:23.100 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:23.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:23.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.990623_217.1 2022-09-13T19:12:23.104 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:23.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:23.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:23.090+0000 7f915b73d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:23.091+0000 7f915b73d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:23.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:23.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:23.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:23.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:23.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:23.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:23.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:23.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:56:23Z" 2022-09-13T19:12:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.142 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.142 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.143 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.143 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.144 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.144 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.144 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.145 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.145 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.146 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.146 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.146 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.147 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.147 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.148 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.148 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.149 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.149 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.149 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.150 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.150 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.151 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.151 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.152 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.152 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.152 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.153 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.153 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.154 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.154 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.155 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.155 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.155 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.156 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.156 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.157 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.157 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.900410_211.1", 2022-09-13T19:12:23.158 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.158 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.158 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.159 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.900410Z", 2022-09-13T19:12:23.159 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.160 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.160 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.160 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.161 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.161 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.162 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.162 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.163 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.163 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.163 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.164 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.164 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.165 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.165 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.165 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:23.166 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.166 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:23.167 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:23.167 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:23.168 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.168 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:56:23Z" 2022-09-13T19:12:23.168 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:23.169 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:23.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.900410_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.900410Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:23.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.900410_211.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.900410_211.1 2022-09-13T19:12:23.171 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.172 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:23.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:28.231+0000 7fa7e515a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:28.232+0000 7fa7e515a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:23.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:23.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:23.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:23.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:23.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:23.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:23.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:23.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:23.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:56:28Z" 2022-09-13T19:12:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.204 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.205 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.205 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.206 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.206 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.206 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.207 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.207 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.208 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.208 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.209 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.209 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.209 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.210 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.210 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.211 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.211 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.212 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.212 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.212 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.213 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.213 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.214 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.214 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.214 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.215 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.215 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.216 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.217 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.217 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.218 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.218 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.219 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.219 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.220 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:23.220 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.220 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.221 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.221 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:23.222 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.222 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.222 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.223 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.223 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.224 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.224 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.224 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.225 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.225 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.226 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.226 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.227 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.227 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.227 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.228 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:23.228 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.229 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:23.229 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:23.229 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:23.230 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.230 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:56:28Z" 2022-09-13T19:12:23.231 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:23.231 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:23.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:23.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.953175_216.1 2022-09-13T19:12:23.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:33.427+0000 7f2def545600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:33.429+0000 7f2def545600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:23.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:23.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:23.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:23.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:23.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:23.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:23.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:23.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:23.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:23.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:23.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:23.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:56:33Z" 2022-09-13T19:12:23.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.265 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.265 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.265 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.266 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.266 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.267 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.267 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.268 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.268 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.268 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.269 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.269 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.270 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.270 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.271 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.271 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.271 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.272 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.272 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.273 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.273 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.273 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.274 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.274 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.275 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.275 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.276 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.276 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.276 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.277 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.277 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.278 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.278 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.279 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.279 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.279 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.280 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:23.280 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.281 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.281 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:23.282 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.282 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.283 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.283 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.284 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.284 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.284 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.285 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.285 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.286 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.286 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.287 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.287 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.287 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.288 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.288 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:23.289 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.289 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:23.289 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:23.290 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:23.290 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.291 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:56:33Z" 2022-09-13T19:12:23.291 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:23.292 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:23.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:23.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.861945_210.1 2022-09-13T19:12:23.294 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.294 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:23.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:23.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:38.574+0000 7f5cf5137600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:38.575+0000 7f5cf5137600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:23.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:23.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:23.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:23.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:23.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:23.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:23.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:23.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:23.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:23.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:23.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:23.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:56:38Z" 2022-09-13T19:12:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.325 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.326 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.326 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.327 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.327 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.327 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.328 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.328 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.329 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.329 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.330 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.330 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.330 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.331 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.331 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.332 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.332 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.332 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.333 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.333 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.334 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.334 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.335 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.335 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.335 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.336 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.336 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.337 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.337 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.337 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.338 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.338 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.339 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.339 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.340 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.340 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.340 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:23.341 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.341 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.342 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.342 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:23.343 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.343 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.343 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.344 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.344 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.345 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.345 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.345 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.346 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.346 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.347 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.347 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.348 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.348 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.349 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.349 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:23.349 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.350 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:23.350 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:23.351 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:23.351 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.351 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:56:38Z" 2022-09-13T19:12:23.352 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:23.352 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:23.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:23.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:23.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:23.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:23.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:23.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:43.733+0000 7f6fff467600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:43.734+0000 7f6fff467600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:23.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:23.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:23.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:23.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:23.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:23.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:23.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:23.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:56:43Z" 2022-09-13T19:12:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.385 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.385 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.386 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.386 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.387 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.387 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.388 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.388 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.389 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.389 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.389 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.390 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.390 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.391 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.391 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.392 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.392 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.393 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.393 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.394 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.394 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.395 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.395 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.396 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.396 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.397 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.397 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.398 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.398 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.398 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.399 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.399 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.400 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.400 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.401 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.401 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:23.401 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.402 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.402 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.403 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:23.403 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.403 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.404 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.404 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.405 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.405 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.406 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.406 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.407 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.407 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.408 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.408 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.408 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.409 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.409 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.410 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:23.410 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.411 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:23.411 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:23.411 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:23.412 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.412 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:56:43Z" 2022-09-13T19:12:23.413 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:23.413 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:23.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:23.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:23.415 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:48.884+0000 7fe11ca14600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:48.885+0000 7fe11ca14600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:23.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:23.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:23.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:23.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:23.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:23.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:23.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:23.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:23.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:23.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:23.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:23.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:23.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:56:48Z" 2022-09-13T19:12:23.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.447 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.448 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.448 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.449 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.449 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.450 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.450 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.451 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.451 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.451 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.452 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.452 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.453 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.454 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.454 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.455 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.455 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.456 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.456 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.456 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.457 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.457 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.458 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.458 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.459 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.459 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.460 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.460 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.461 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.461 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.462 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.462 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.462 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.463 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.463 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:23.464 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.464 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.465 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.465 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:23.465 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.466 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.466 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.467 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.467 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.468 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.468 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.468 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.469 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.469 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.470 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.470 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.471 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.471 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.472 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.472 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:23.472 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.473 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:23.473 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:23.474 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:23.474 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.475 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:56:48Z" 2022-09-13T19:12:23.475 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:23.476 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:23.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:23.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:23.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:54.027+0000 7f81abae3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:54.028+0000 7f81abae3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:23.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:23.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:23.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:23.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:23.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:23.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:23.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:23.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:23.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.107994_222.1", 2022-09-13T19:12:23.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.107994Z", 2022-09-13T19:12:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:23.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:23.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:23.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:23.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:56:54Z" 2022-09-13T19:12:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.509 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.510 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.510 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.510 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.511 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.511 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.512 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.512 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.512 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.513 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.513 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.514 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.514 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.515 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.515 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.515 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.516 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.517 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.517 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.518 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.519 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.519 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.519 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.520 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.520 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.521 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.521 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.521 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.522 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.522 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.523 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.523 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.524 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.524 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.524 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.525 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.107994_222.1", 2022-09-13T19:12:23.525 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.526 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.526 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.527 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.107994Z", 2022-09-13T19:12:23.527 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.527 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.528 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.528 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.529 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.529 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.530 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.530 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.530 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.531 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.531 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.532 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.532 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.532 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.533 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.533 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:23.534 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.534 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:23.534 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:23.535 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:23.535 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.536 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:56:54Z" 2022-09-13T19:12:23.536 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:23.537 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:23.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.107994_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.107994Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:23.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.107994_222.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055099.107994_222.1 2022-09-13T19:12:23.539 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.539 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:59.179+0000 7f0fed8c5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:56:59.180+0000 7f0fed8c5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:23.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:23.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:23.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:23.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:23.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:23.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:23.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:23.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:23.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.994620_218.1", 2022-09-13T19:12:23.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.994620Z", 2022-09-13T19:12:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:23.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:23.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:23.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:23.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:56:59Z" 2022-09-13T19:12:23.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.570 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.570 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.570 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.571 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.571 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.572 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.572 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.572 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.573 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.573 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.574 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.574 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.575 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.575 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.575 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.576 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.576 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.577 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.577 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.577 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.578 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.578 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.579 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.579 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.580 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.580 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.580 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.581 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.581 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.582 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.582 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.583 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.583 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.583 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.584 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.584 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.585 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.994620_218.1", 2022-09-13T19:12:23.585 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.585 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.586 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.586 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.994620Z", 2022-09-13T19:12:23.587 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.587 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.587 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.588 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.588 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.589 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.589 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.590 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.590 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.590 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.591 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.591 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.592 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.592 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.592 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.593 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:23.593 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.594 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:23.594 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:23.595 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:23.595 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.595 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:56:59Z" 2022-09-13T19:12:23.596 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:23.596 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:23.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.994620_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.994620Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:23.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.994620_218.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.994620_218.1 2022-09-13T19:12:23.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.600 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:23.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:23.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:04.385+0000 7f0ce3252600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:04.386+0000 7f0ce3252600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:23.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:23.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:23.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:23.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:23.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:23.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:23.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:23.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:23.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:23.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:23.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:23.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:23.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:23.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:57:04Z" 2022-09-13T19:12:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.633 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.633 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.634 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.634 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.635 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.635 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.635 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.636 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.636 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.637 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.637 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.637 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.638 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.638 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.639 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.639 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.639 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.640 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.640 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.641 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.641 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.642 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.642 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.642 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.643 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.644 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.644 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.644 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.645 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.645 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.646 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.646 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.647 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.647 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.647 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.648 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:23.648 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.649 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.649 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.649 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:23.650 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.650 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.651 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.651 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.651 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.652 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.652 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.653 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.654 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.654 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.654 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.655 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.655 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.656 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.656 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:23.657 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.657 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:23.657 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:23.658 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:23.658 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.659 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:57:04Z" 2022-09-13T19:12:23.659 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:23.660 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:23.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:23.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:23.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:09.773+0000 7f7225c01600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:09.774+0000 7f7225c01600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:23.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:23.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:23.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:23.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:23.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.111513_224.1", 2022-09-13T19:12:23.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.111513Z", 2022-09-13T19:12:23.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:23.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:23.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:23.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:23.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:57:09Z" 2022-09-13T19:12:23.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.692 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.693 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.693 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.694 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.694 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.694 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.695 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.695 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.696 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.696 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.696 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.697 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.697 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.698 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.698 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.699 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.699 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.700 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.700 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.700 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.701 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.701 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.702 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.702 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.702 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.703 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.703 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.704 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.704 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.704 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.705 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.705 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.706 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.706 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.707 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.707 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.707 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.111513_224.1", 2022-09-13T19:12:23.708 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.708 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.709 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.709 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.111513Z", 2022-09-13T19:12:23.709 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.710 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.710 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.711 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.711 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.712 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.712 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.712 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.713 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.713 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.714 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.714 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.715 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.715 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.715 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.716 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:23.716 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.717 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:23.717 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:23.717 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:23.718 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.718 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:57:09Z" 2022-09-13T19:12:23.719 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:23.719 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:23.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.111513_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.111513Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:23.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.111513_224.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055099.111513_224.1 2022-09-13T19:12:23.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.722 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:23.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:23.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:23.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:14.930+0000 7fed400f0600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:14.931+0000 7fed400f0600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:23.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:23.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:23.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:23.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:23.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:57:14Z" 2022-09-13T19:12:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.760 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.761 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.762 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.762 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.763 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.764 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.765 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.765 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.766 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.767 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.768 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.769 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.769 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.770 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.771 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.772 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.772 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.773 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.774 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.775 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.776 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.777 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.777 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.778 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.779 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.779 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.779 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.780 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.781 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.782 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.782 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.783 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.784 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.785 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.786 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.786 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.787 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:23.788 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.789 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.789 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.790 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:23.791 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.792 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.793 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.793 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.794 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.795 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.796 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.796 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.798 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.799 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.800 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.800 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.801 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.802 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.802 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:23.803 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.804 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:23.804 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:23.805 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:23.806 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.807 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:57:14Z" 2022-09-13T19:12:23.807 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:23.808 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:23.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:23.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:23.811 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.812 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:23.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:23.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:23.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:20.167+0000 7f7bb6a73600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:20.168+0000 7f7bb6a73600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:23.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:23.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:23.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:23.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:23.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:23.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:23.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:23.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:23.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:23.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:23.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:57:20Z" 2022-09-13T19:12:23.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.844 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.845 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.845 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.845 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.846 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.846 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.847 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.847 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.847 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.848 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.848 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.849 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.849 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.849 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.850 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.850 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.851 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.851 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.852 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.852 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.852 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.853 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.853 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.854 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.854 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.854 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.855 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.855 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.856 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.856 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.856 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.857 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.857 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.858 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.858 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.858 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.859 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:23.859 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.860 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.860 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.860 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:23.861 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.861 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.862 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.862 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.862 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.863 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.863 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.864 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.864 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.865 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.866 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.867 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.867 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:23.867 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.868 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:23.868 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:23.869 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:23.869 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.870 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:57:20Z" 2022-09-13T19:12:23.870 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:23.871 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:23.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:23.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:23.873 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.873 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:23.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:23.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:23.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:25.324+0000 7f15766fc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:25.325+0000 7f15766fc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:23.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:23.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:23.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:23.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:23.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:23.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:23.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:23.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:57:25Z" 2022-09-13T19:12:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.902 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.903 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.903 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.903 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.904 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.904 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.905 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.905 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.905 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.906 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.906 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.907 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.907 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.907 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.908 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.908 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.909 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.909 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.909 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.910 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.910 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.911 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.911 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.912 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.912 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.912 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.913 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.913 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.914 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.914 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.914 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.915 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.915 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.916 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.916 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.916 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.917 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:23.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.918 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.918 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:23.919 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.919 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.920 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.920 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.921 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.923 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.924 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.925 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.925 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.926 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.926 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.927 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.928 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.928 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.929 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.929 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:23.930 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.930 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:23.931 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:23.931 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:23.932 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.932 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:57:25Z" 2022-09-13T19:12:23.933 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:23.933 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:23.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:23.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:23.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.857076_208.1 2022-09-13T19:12:23.936 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:23.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:23.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:30.478+0000 7fc23cfbc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:30.479+0000 7fc23cfbc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:23.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:23.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:23.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:23.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.907374_213.1", 2022-09-13T19:12:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.907374Z", 2022-09-13T19:12:23.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:23.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:23.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:23.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:23.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:23.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:23.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:23.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:23.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:23.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:23.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:23.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:57:30Z" 2022-09-13T19:12:23.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:23.977 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:23.977 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:23.978 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:23.979 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:23.979 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:23.979 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:23.980 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.980 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.981 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:23.981 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.982 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:23.982 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.983 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:23.984 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.984 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:23.985 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.985 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:23.986 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.986 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.987 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.987 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.987 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:23.988 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.988 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.989 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:23.989 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.990 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.990 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.991 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:23.991 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.992 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.992 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.992 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.993 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:23.994 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:23.994 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:23.995 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.907374_213.1", 2022-09-13T19:12:23.995 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:23.996 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:23.996 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:23.996 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.907374Z", 2022-09-13T19:12:23.997 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:23.997 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:23.998 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:23.998 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:23.999 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:23.999 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.000 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.000 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.001 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.001 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.002 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.002 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.003 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.003 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.004 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.004 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:24.005 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.005 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:24.005 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:24.006 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:24.006 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.007 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:57:30Z" 2022-09-13T19:12:24.007 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:24.008 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:24.008 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.907374_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.907374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:24.008 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.907374_213.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.907374_213.1 2022-09-13T19:12:24.010 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.011 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:24.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:24.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:24.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:35.632+0000 7fd710660600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:35.633+0000 7fd710660600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:24.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:24.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:24.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:24.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:24.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.187352_231.1", 2022-09-13T19:12:24.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T19:12:24.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:24.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:24.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:24.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:24.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:57:35Z" 2022-09-13T19:12:24.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:24.045 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:24.045 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:24.046 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:24.046 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:24.047 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.047 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.048 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:24.048 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.049 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:24.049 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.050 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.050 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.050 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.051 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:24.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.052 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.052 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.053 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.053 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.054 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.054 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:24.055 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.055 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.056 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:24.056 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.057 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.057 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.058 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:24.058 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.058 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.059 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.059 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.060 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:24.060 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.061 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.061 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.187352_231.1", 2022-09-13T19:12:24.061 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.062 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.062 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.063 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T19:12:24.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.064 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.064 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.064 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.065 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:24.065 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.066 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.066 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.067 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.067 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.068 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.068 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.069 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.069 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.069 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.070 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:24.070 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.071 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:24.071 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:24.071 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:24.072 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.072 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:57:35Z" 2022-09-13T19:12:24.073 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:24.073 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:24.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.187352_231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.187352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:24.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.187352_231.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055099.187352_231.1 2022-09-13T19:12:24.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:40.774+0000 7f5ad9c50600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:40.775+0000 7f5ad9c50600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:24.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:24.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:24.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:24.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:24.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:24.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:24.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:24.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:24.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:24.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:24.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:24.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:57:40Z" 2022-09-13T19:12:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:24.109 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:24.110 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:24.110 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:24.111 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:24.111 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.112 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.112 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.113 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:24.113 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.114 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:24.114 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.114 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.115 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.115 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.116 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:24.116 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.116 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.117 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.117 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.118 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.118 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.119 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:24.119 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.119 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.120 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:24.120 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.121 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.121 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.122 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:24.122 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.122 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.123 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.123 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.124 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:24.125 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.125 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.125 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:24.126 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.126 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.127 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.127 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:24.127 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.128 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.128 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.129 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.129 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:24.130 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.130 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.131 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.131 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.132 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.132 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.132 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.133 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.133 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.134 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.134 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:24.135 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.135 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:24.135 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:24.136 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:24.136 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.137 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:57:40Z" 2022-09-13T19:12:24.137 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:24.138 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:24.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:24.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:24.140 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:24.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:24.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:45.940+0000 7f1ebec82600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:45.942+0000 7f1ebec82600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:24.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:24.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:24.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:24.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:24.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:24.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:24.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:24.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:24.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:24.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:24.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:24.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:57:45Z" 2022-09-13T19:12:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:24.172 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:24.172 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:24.173 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:24.173 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:24.174 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.174 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.175 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.175 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:24.175 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.176 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:24.176 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.177 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.177 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.178 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.178 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:24.178 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.179 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.179 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.180 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.180 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.181 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.181 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:24.181 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.182 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.182 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:24.183 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.183 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.183 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.184 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:24.184 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.185 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.185 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.186 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.186 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:24.186 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.187 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.187 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:24.188 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.188 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.189 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.189 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:24.189 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.190 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.190 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.191 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.191 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:24.192 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.192 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.193 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.193 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.193 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.194 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.194 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.195 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.195 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.196 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.196 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:24.197 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.197 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:24.197 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:24.198 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:24.198 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.199 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:57:45Z" 2022-09-13T19:12:24.199 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:24.200 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:24.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:24.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.854520_207.1 2022-09-13T19:12:24.203 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.203 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:24.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:24.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:24.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:51.157+0000 7fe6749b5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:51.158+0000 7fe6749b5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:24.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:24.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:24.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:24.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:24.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:24.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:24.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:24.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:24.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.906479_212.1", 2022-09-13T19:12:24.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.906479Z", 2022-09-13T19:12:24.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:24.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:24.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:24.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:24.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:24.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:57:51Z" 2022-09-13T19:12:24.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:24.242 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:24.243 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:24.243 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:24.244 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:24.244 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.244 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.245 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.245 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:24.246 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.246 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:24.247 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.247 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.248 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.249 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:24.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.250 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.250 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.251 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.251 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.252 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.252 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:24.253 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.253 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.253 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:24.254 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.254 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.255 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.255 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:24.256 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.256 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.256 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.257 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.257 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:24.258 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.258 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.259 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.906479_212.1", 2022-09-13T19:12:24.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.261 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.906479Z", 2022-09-13T19:12:24.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.261 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.262 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.262 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.263 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:24.263 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.264 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.264 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.265 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.265 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.266 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.266 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.267 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.267 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.268 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:24.268 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.269 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:24.269 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:24.270 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:24.270 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.270 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:57:51Z" 2022-09-13T19:12:24.271 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:24.271 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:24.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.906479_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.906479Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:24.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.906479_212.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.906479_212.1 2022-09-13T19:12:24.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:24.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:24.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:56.284+0000 7f2b2632c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:57:56.285+0000 7f2b2632c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:24.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:24.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:24.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:24.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:24.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:24.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:24.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:24.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:24.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:24.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:24.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:24.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:24.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:57:56Z" 2022-09-13T19:12:24.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:24.323 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:24.324 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:24.325 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:24.326 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:24.327 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.327 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.328 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.329 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:24.330 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.330 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:24.331 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.332 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.333 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.334 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.335 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:24.336 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.336 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.337 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.338 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.339 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.340 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.340 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:24.340 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.341 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.342 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:24.343 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.344 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.345 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.345 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:24.346 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.347 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.348 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.349 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.350 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:24.350 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.351 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.351 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:24.351 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.352 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.352 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.353 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:24.353 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.354 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.354 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.355 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.355 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:24.355 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.356 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.356 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.357 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.357 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.358 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.358 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.359 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.359 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.360 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.361 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:24.362 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.362 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:24.363 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:24.363 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:24.364 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.364 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:57:56Z" 2022-09-13T19:12:24.365 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:24.365 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:24.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:24.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055099.220845_233.1 2022-09-13T19:12:24.367 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.368 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:01.419+0000 7f4462cea600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:01.420+0000 7f4462cea600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:24.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:24.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:24.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:24.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:24.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:24.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:24.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:24.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:24.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:24.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:24.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:58:01Z" 2022-09-13T19:12:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:24.400 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:24.400 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:24.401 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:24.401 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:24.402 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.402 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.403 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.403 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:24.404 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.404 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:24.405 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.405 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.406 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.407 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:24.407 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.408 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.408 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.408 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.409 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.409 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.410 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:24.410 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.411 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.411 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:24.411 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.412 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.412 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.413 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:24.413 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.414 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.415 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.415 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.415 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:24.416 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.416 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.417 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:24.417 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.417 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.418 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.418 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:24.419 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.419 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.419 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.420 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.420 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:24.421 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.421 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.422 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.422 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.423 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.424 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.424 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.425 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.425 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.426 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:24.426 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.426 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:24.427 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:24.427 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:24.428 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.428 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:58:01Z" 2022-09-13T19:12:24.428 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:24.429 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:24.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:24.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.854520_207.1 2022-09-13T19:12:24.431 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.431 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:06.566+0000 7f4ec501a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:06.567+0000 7f4ec501a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:24.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:24.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:24.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:24.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:24.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:24.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:24.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:24.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:24.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:24.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:24.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:24.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:24.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:24.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:24.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:58:06Z" 2022-09-13T19:12:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:24.464 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:24.465 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:24.465 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:24.466 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:24.466 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.466 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.467 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.467 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:24.468 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.468 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:24.469 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.469 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.469 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.470 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.470 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:24.471 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.471 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.471 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.472 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.472 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.473 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.473 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:24.474 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.474 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.475 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:24.475 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.476 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.476 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.476 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:24.477 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.477 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.478 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.478 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.479 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:24.479 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.480 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.480 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:24.480 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.481 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.481 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.482 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:24.482 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.483 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.483 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.484 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.484 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:24.484 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.485 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.485 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.486 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.486 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.487 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.487 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.487 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.488 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.488 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.489 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:24.489 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.490 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:24.490 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:24.490 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:24.491 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.491 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:58:06Z" 2022-09-13T19:12:24.492 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:24.492 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:24.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:24.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.861945_210.1 2022-09-13T19:12:24.494 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.494 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:11.709+0000 7f7a135a6600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:11.710+0000 7f7a135a6600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:24.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:24.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:24.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:24.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:24.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:24.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:24.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:24.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:24.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:24.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.907374_213.1", 2022-09-13T19:12:24.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.907374Z", 2022-09-13T19:12:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:24.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:24.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:24.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:24.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:58:11Z" 2022-09-13T19:12:24.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:24.524 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:24.525 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:24.525 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:24.525 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:24.526 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.526 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.527 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.527 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:24.527 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.528 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:24.528 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.529 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.529 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.529 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.530 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:24.530 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.531 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.531 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.532 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.532 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.532 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.533 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:24.533 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.534 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.534 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:24.534 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.535 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.535 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.536 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:24.536 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.536 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.537 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.537 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.538 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:24.538 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.538 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.539 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.907374_213.1", 2022-09-13T19:12:24.539 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.540 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.540 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.541 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.907374Z", 2022-09-13T19:12:24.541 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.541 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.542 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.542 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.543 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:24.543 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.543 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.544 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.544 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.545 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.545 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.545 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.546 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.546 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.547 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.547 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:24.548 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.548 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:24.548 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:24.549 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:24.549 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.550 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:58:11Z" 2022-09-13T19:12:24.550 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:24.550 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:24.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.907374_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.907374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:24.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.907374_213.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.907374_213.1 2022-09-13T19:12:24.552 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.553 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:16.859+0000 7f5d78caa600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:16.860+0000 7f5d78caa600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:24.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:24.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:24.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:24.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:24.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:24.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:24.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:24.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:24.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:24.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:58:16Z" 2022-09-13T19:12:24.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:24.585 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:24.586 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:24.586 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:24.587 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:24.587 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.588 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.588 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.589 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:24.589 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.590 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:24.590 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.591 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.591 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.592 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.592 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:24.592 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.593 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.593 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.594 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.594 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.594 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.595 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:24.595 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.596 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.596 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:24.597 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.597 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.597 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.598 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:24.598 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.599 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.599 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.599 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.600 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:24.600 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.601 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.601 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:24.601 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.602 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.602 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.603 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:24.603 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.603 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.604 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.604 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.605 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:24.605 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.605 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.606 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.606 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.607 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.607 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.608 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.608 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.608 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.609 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.609 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:24.610 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.610 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:24.610 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:24.611 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:24.611 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.612 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:58:16Z" 2022-09-13T19:12:24.612 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:24.612 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:24.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:24.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:24.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:24.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:24.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:24.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:22.002+0000 7f80c0920600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:22.003+0000 7f80c0920600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:24.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:24.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:24.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:24.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:24.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:24.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:24.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:24.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:24.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:24.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:24.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:24.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:24.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:24.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:24.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:24.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:58:22Z" 2022-09-13T19:12:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:24.651 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:24.652 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:24.653 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:24.653 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:24.654 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.655 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.656 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.656 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:24.657 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.658 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:24.658 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.659 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.660 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.661 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.661 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:24.662 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.663 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.663 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.664 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.665 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.666 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.666 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:24.667 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.668 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.669 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:24.669 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.670 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.671 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.671 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:24.672 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.673 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.674 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.674 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.675 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:24.675 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.675 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.676 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:24.676 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.677 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.677 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.677 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:24.678 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.678 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.679 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.679 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.680 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:24.681 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.681 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.682 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.682 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.683 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.684 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.684 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.685 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.685 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.685 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.686 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:24.686 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.687 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:24.687 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:24.688 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:24.689 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.689 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:58:22Z" 2022-09-13T19:12:24.690 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:24.690 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:24.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:24.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.861945_210.1 2022-09-13T19:12:24.694 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.695 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:24.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:24.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:24.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:27.246+0000 7fb4b924c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:27.247+0000 7fb4b924c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:24.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:24.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:24.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:24.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:24.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:24.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:24.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:24.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:24.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:24.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:24.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:24.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:24.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:24.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:58:27Z" 2022-09-13T19:12:24.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:24.740 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:24.741 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:24.742 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:24.742 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:24.743 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.744 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.745 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.746 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:24.746 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.747 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:24.748 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.749 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.749 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.751 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:24.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.752 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.753 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.754 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.755 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.755 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.756 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:24.757 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.757 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.758 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:24.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.760 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.762 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:24.762 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.763 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.763 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.763 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.764 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:24.764 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.765 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.765 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:24.765 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.766 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.766 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.767 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:24.767 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.767 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.768 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.768 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.769 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:24.769 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.769 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.770 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.770 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.771 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.772 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.772 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.773 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.773 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:24.773 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.774 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:24.774 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:24.775 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:24.775 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.775 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:58:27Z" 2022-09-13T19:12:24.776 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:24.776 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:24.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:24.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.858000_209.1 2022-09-13T19:12:24.778 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.779 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:24.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:24.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:24.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:32.384+0000 7fc5b3afd600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:32.385+0000 7fc5b3afd600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:24.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:24.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:24.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:24.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:24.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:24.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:24.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:24.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:24.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:24.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:24.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:58:32Z" 2022-09-13T19:12:24.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:24.833 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:24.834 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:24.834 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:24.834 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:24.835 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.835 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.836 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.836 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:24.836 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.837 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:24.837 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.838 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.838 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.839 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.839 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:24.839 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.840 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.841 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.842 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.843 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.843 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.844 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:24.845 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.846 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.847 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:24.847 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.848 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.849 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.850 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:24.851 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.852 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.853 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.853 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.854 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:24.854 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.854 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.855 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:24.855 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.856 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.856 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.857 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:24.857 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.857 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.858 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.858 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.859 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:24.859 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.860 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.860 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.860 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.861 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.861 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.862 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.862 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.862 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.863 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.863 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:24.864 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.864 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:24.865 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:24.865 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:24.865 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.866 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:58:32Z" 2022-09-13T19:12:24.866 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:24.867 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:24.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:24.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:24.869 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.870 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:37.541+0000 7ffa7a586600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:37.542+0000 7ffa7a586600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:24.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:24.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:24.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:24.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:24.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:24.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:24.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:24.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:24.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:24.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:24.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:24.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:24.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:58:37Z" 2022-09-13T19:12:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:24.902 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:24.902 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:24.903 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:24.903 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:24.904 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.904 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.905 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.905 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:24.905 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.906 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:24.906 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.907 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.907 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.908 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.908 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:24.908 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.909 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.909 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.910 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.910 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.911 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.911 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:24.912 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.913 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.913 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:24.913 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.914 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.914 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.915 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:24.915 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.915 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.916 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.916 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.917 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:24.917 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.918 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.918 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:24.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.919 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.920 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:24.920 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.921 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.921 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.921 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.922 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:24.922 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.923 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.923 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.923 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.924 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.924 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.925 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.925 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.926 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.926 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.926 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:24.927 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.927 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:24.928 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:24.928 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:24.929 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.929 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:58:37Z" 2022-09-13T19:12:24.929 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:24.930 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:24.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:24.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.854520_207.1 2022-09-13T19:12:24.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:24.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:24.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:24.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:42.671+0000 7f77e532c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:42.672+0000 7f77e532c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:24.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:24.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:24.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:24.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:24.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:24.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:24.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:24.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:24.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:24.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:24.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:24.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:24.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:24.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:24.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:24.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:24.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:58:42Z" 2022-09-13T19:12:24.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:24.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:24.963 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:24.964 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:24.964 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:24.964 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:24.965 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.965 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.966 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.966 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:24.967 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.967 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:24.968 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.968 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.968 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:24.969 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.969 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:24.970 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.970 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:24.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.971 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.971 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.972 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.972 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:24.973 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.973 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.974 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:24.974 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.975 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.975 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.975 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:24.976 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.976 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.977 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.977 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.977 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:24.978 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.978 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.979 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:24.979 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.980 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.980 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.980 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:24.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.981 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.982 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.982 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:24.983 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:24.983 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:24.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:24.984 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:24.984 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:24.985 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:24.985 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:24.986 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:24.986 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:24.986 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.987 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:24.987 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:24.988 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.988 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:24.989 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:24.989 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:24.989 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:24.990 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:58:42Z" 2022-09-13T19:12:24.990 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:24.991 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:24.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:24.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:24.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.858000_209.1 2022-09-13T19:12:24.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:24.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:24.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:24.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:24.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:47.908+0000 7fc4bb400600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:47.909+0000 7fc4bb400600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:24.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:24.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:24.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:24.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:24.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:24.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:25.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:25.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:25.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:25.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:25.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:58:47Z" 2022-09-13T19:12:25.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:25.025 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:25.026 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:25.026 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:25.026 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:25.027 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.027 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.028 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.028 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:25.029 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.029 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:25.029 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.030 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.030 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.031 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.031 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:25.032 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.032 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.032 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.033 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.033 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.034 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.034 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:25.035 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.035 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.035 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:25.036 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.036 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.037 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.037 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:25.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.038 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.038 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.039 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.039 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:25.040 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.040 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.041 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:25.041 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.041 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.042 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.042 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:25.043 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.043 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.044 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.044 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.044 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:25.045 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.045 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.046 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.046 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.047 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.047 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.047 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.048 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.048 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.049 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.049 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:25.050 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.050 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:25.050 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:25.051 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:25.051 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.052 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:58:47Z" 2022-09-13T19:12:25.052 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:25.052 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:25.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:25.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:25.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:53.045+0000 7fdccb159600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:53.046+0000 7fdccb159600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:25.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:25.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:25.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:25.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:25.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:25.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:25.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:25.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:25.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:25.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:58:53Z" 2022-09-13T19:12:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:25.086 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:25.086 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:25.086 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:25.087 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:25.087 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.088 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.088 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.088 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:25.089 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.089 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:25.090 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.090 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.091 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.091 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.091 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:25.092 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.092 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.093 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.093 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.094 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.094 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.094 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:25.095 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.095 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.096 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:25.096 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.096 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.097 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.097 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:25.098 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.098 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.099 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.099 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.099 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:25.100 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.100 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.101 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:25.101 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.102 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.102 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.102 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:25.103 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.103 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.104 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.104 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.105 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:25.105 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.105 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.106 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.106 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.107 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.107 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.108 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.108 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.108 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.109 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.109 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:25.110 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.110 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:25.111 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:25.111 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:25.111 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.112 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:58:53Z" 2022-09-13T19:12:25.112 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:25.113 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:25.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:25.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:25.115 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.116 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:58.185+0000 7fbb40c1c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:58:58.186+0000 7fbb40c1c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:25.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:25.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:25.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:25.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:58:58Z" 2022-09-13T19:12:25.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:25.147 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:25.147 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:25.148 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:25.148 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:25.148 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.149 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.149 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.150 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:25.150 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.151 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:25.151 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.151 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.152 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.152 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.153 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:25.153 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.154 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.154 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.154 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.155 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.155 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.156 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:25.156 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.157 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.157 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:25.157 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.158 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.158 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.159 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:25.159 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.159 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.160 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.160 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.161 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:25.161 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.162 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.162 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:25.162 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.163 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.163 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.164 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:25.164 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.165 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.165 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.165 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.166 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:25.166 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.167 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.167 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.168 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.168 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.168 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.169 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.169 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.170 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.170 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.171 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:25.171 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.172 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:25.172 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:25.173 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:25.173 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.173 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:58:58Z" 2022-09-13T19:12:25.174 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:25.174 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:25.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:25.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.990623_217.1 2022-09-13T19:12:25.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:25.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:25.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:25.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:03.341+0000 7f4215bb2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:03.342+0000 7f4215bb2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:25.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:25.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:25.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:25.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:25.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:59:03Z" 2022-09-13T19:12:25.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:25.212 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:25.212 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:25.213 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:25.213 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:25.214 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.215 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.215 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:25.215 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.216 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:25.216 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.217 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.217 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.218 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.218 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:25.218 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.219 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.219 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.220 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.220 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.220 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.221 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:25.221 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.222 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.222 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:25.223 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.223 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.224 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.224 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:25.224 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.225 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.225 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.226 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.226 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:25.226 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.227 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.227 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:25.228 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.228 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.229 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.229 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:25.229 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.230 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.230 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.231 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.231 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:25.232 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.232 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.232 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.233 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.233 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.234 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.234 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.235 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.235 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.236 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.236 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:25.236 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.237 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:25.237 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:25.238 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:25.238 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.239 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:59:03Z" 2022-09-13T19:12:25.239 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:25.240 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:25.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:25.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.857076_208.1 2022-09-13T19:12:25.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:08.494+0000 7f163ca39600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:08.495+0000 7f163ca39600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:25.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:25.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:25.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:25.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:25.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:59:08Z" 2022-09-13T19:12:25.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:25.280 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:25.280 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:25.281 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:25.282 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:25.283 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.283 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.284 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.284 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:25.285 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.285 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:25.286 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.286 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.287 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.287 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.288 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:25.288 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.289 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.289 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.290 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.290 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.291 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.291 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:25.292 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.293 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.293 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:25.294 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.294 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.295 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.295 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:25.296 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.296 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.296 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.297 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.297 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:25.298 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.298 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.299 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:25.299 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.299 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.300 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.300 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:25.304 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.305 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.305 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.306 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.306 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:25.307 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.307 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.307 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.308 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.308 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.309 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.309 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.310 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.310 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.310 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.311 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:25.311 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.312 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:25.312 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:25.313 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:25.313 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.314 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:59:08Z" 2022-09-13T19:12:25.314 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:25.315 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:25.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:25.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:25.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:13.642+0000 7f0d30239600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:13.643+0000 7f0d30239600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:25.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:25.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:25.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:25.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:25.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:25.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:25.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:25.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:25.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:25.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:25.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:59:13Z" 2022-09-13T19:12:25.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:25.349 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:25.350 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:25.351 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:25.353 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:25.354 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.355 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.356 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.356 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:25.357 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.358 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:25.358 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.359 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.360 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.360 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.360 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:25.361 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.361 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.362 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.362 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.363 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.363 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.364 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:25.364 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.365 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.366 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:25.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.367 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.368 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:25.369 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.369 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.369 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.370 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.370 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:25.371 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.371 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.372 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.857076_208.1", 2022-09-13T19:12:25.372 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.373 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.373 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.374 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.857076Z", 2022-09-13T19:12:25.374 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.374 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.375 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.375 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.376 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:25.377 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.378 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.378 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.379 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.379 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.380 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.380 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.381 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.381 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.381 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.382 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:25.382 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.383 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:25.383 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:25.384 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:25.384 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.384 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:59:13Z" 2022-09-13T19:12:25.385 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:25.385 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:25.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.857076_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.857076Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:25.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.857076_208.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.857076_208.1 2022-09-13T19:12:25.388 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:25.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:25.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:25.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:18.876+0000 7f3867dbe600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:18.877+0000 7f3867dbe600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:25.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:25.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:25.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:25.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:25.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:25.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:25.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:25.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:25.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:25.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:25.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:25.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:25.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:25.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:25.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:59:18Z" 2022-09-13T19:12:25.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:25.430 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:25.430 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:25.431 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:25.431 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:25.432 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.432 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.433 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.434 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:25.434 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.435 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:25.435 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.436 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.436 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.437 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.437 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:25.438 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.438 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.438 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.439 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.439 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.440 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.440 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:25.441 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.441 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.441 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:25.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.442 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.443 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.443 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:25.443 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.444 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.444 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.445 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.445 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:25.445 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.446 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.446 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:25.447 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.447 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.448 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.448 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:25.448 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.449 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.449 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.450 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.450 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:25.450 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.451 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.451 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.452 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.452 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.452 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.453 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.453 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.454 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.454 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.455 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:25.455 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.455 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:25.456 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:25.456 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:25.457 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.457 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:59:18Z" 2022-09-13T19:12:25.457 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:25.458 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:25.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:25.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:25.460 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.460 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:25.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:25.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:25.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:24.012+0000 7f5eb2365600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:24.013+0000 7f5eb2365600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:25.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:25.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:25.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:25.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:25.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:25.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:25.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:25.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:25.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:25.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.340682_237.1", 2022-09-13T19:12:25.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T19:12:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:25.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:25.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:25.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:25.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:59:24Z" 2022-09-13T19:12:25.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:25.491 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:25.491 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:25.492 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:25.492 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:25.493 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.493 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.493 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.494 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:25.494 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.495 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:25.495 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.496 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.496 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.497 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.497 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:25.498 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.498 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.499 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.499 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.500 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.500 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.500 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:25.501 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.501 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.502 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:25.502 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.503 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.503 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.504 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:25.504 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.504 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.505 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.505 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.506 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:25.506 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.507 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.507 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.340682_237.1", 2022-09-13T19:12:25.507 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.508 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.508 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.509 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T19:12:25.509 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.509 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.510 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.510 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.511 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:25.511 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.512 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.512 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.512 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.513 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.513 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.514 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.514 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.515 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.515 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.515 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:25.516 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.516 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:25.517 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:25.517 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:25.518 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.518 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:59:24Z" 2022-09-13T19:12:25.519 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:25.519 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:25.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.340682_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.340682Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:25.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.340682_237.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055099.340682_237.1 2022-09-13T19:12:25.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:25.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:25.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:25.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:29.164+0000 7f64e5c33600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:29.165+0000 7f64e5c33600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:25.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:25.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:25.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:25.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:25.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:25.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:25.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:25.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:25.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:25.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:25.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:25.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:25.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:25.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:25.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:59:29Z" 2022-09-13T19:12:25.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:25.552 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:25.553 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:25.554 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:25.554 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:25.555 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.555 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.556 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:25.556 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.557 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:25.557 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.558 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.558 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.558 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.559 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:25.559 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.560 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.560 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.561 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.561 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.561 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.562 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:25.562 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.563 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.563 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:25.563 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.564 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.564 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.565 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:25.565 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.566 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.566 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.567 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.567 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:25.567 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.568 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.568 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:25.569 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.569 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.570 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.570 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:25.570 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.571 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.571 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.572 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.572 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:25.573 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.574 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.575 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.575 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.576 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.577 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.577 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.578 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.578 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.579 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.579 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:25.580 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.580 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:25.581 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:25.582 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:25.583 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.583 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:59:29Z" 2022-09-13T19:12:25.584 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:25.585 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:25.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:25.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.911619_214.1 2022-09-13T19:12:25.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:25.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:25.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:25.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:34.329+0000 7fef8be6a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:34.330+0000 7fef8be6a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:25.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:25.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:25.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:25.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:25.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:25.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:25.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:25.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:25.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:25.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:25.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:25.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:25.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:25.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:59:34Z" 2022-09-13T19:12:25.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:25.625 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:25.626 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:25.626 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:25.627 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:25.627 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.628 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.629 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.630 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:25.631 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.631 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:25.632 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.632 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.632 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.634 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:25.634 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.636 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.637 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.639 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.641 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.641 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.642 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:25.643 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.644 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.645 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:25.645 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.646 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.646 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.647 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:25.647 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.648 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.648 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.649 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.649 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:25.650 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.651 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.651 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:25.651 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.652 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.652 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.653 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:25.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.654 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.654 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.654 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.655 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:25.656 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.656 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.657 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.657 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.658 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.658 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.659 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.659 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.660 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.660 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.660 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:25.661 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.661 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:25.662 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:25.662 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:25.662 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.663 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:59:34Z" 2022-09-13T19:12:25.663 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:25.664 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:25.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:25.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:25.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.667 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:39.471+0000 7f45e6d0c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:39.472+0000 7f45e6d0c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:25.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:25.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:25.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:25.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:25.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:25.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:25.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:25.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:25.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:25.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:25.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:25.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:25.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:59:39Z" 2022-09-13T19:12:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:25.705 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:25.705 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:25.706 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:25.707 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:25.708 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.708 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.708 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.709 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:25.709 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.710 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:25.710 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.710 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.718 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.718 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.719 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:25.719 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.720 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.720 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.721 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.721 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.722 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.722 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:25.723 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.723 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.725 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:25.726 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.727 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.727 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.728 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:25.729 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.731 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.731 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.732 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.733 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:25.734 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.735 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.736 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.831134_204.1", 2022-09-13T19:12:25.737 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.737 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.738 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.739 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.831134Z", 2022-09-13T19:12:25.740 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.740 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.741 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.742 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.743 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:25.743 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.743 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.744 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.744 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.745 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.746 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.746 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.747 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.748 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.749 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.749 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:25.750 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.751 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:25.751 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:25.752 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:25.753 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.754 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:59:39Z" 2022-09-13T19:12:25.754 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:25.755 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:25.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.831134_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.831134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:25.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.831134_204.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.831134_204.1 2022-09-13T19:12:25.758 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.759 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:44.639+0000 7fcc897e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:44.640+0000 7fcc897e1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:25.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:25.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:25.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:25.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:25.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:25.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:25.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.105395_221.1", 2022-09-13T19:12:25.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T19:12:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:25.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:25.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:59:44Z" 2022-09-13T19:12:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:25.801 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:25.801 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:25.801 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:25.802 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:25.802 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.803 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.803 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.804 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:25.804 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.805 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:25.805 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.805 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.806 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.806 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.807 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:25.807 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.808 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.808 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.808 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.809 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.809 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.810 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:25.810 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.811 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.811 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:25.811 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.812 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.812 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.813 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:25.813 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.814 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.814 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.815 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.815 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:25.816 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.816 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.817 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.105395_221.1", 2022-09-13T19:12:25.817 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.817 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.818 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.818 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T19:12:25.819 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.819 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.820 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.820 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.821 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:25.821 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.822 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.822 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.823 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.824 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.825 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.825 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.826 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.826 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:25.826 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.827 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:25.827 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:25.828 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:25.828 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.829 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:59:44Z" 2022-09-13T19:12:25.829 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:25.829 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:25.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.105395_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.105395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:25.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.105395_221.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055099.105395_221.1 2022-09-13T19:12:25.832 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.832 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:25.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:25.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:25.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:49.846+0000 7f900d702600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:49.847+0000 7f900d702600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:25.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:25.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:25.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:25.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:25.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:25.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:25.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:25.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:25.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:25.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:25.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:59:49Z" 2022-09-13T19:12:25.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:25.863 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:25.864 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:25.864 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:25.865 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:25.865 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.865 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.866 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.866 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:25.867 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.867 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:25.868 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.868 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.869 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.869 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.869 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:25.870 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.870 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.871 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.871 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.872 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.872 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.873 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:25.873 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.874 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.874 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:25.875 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.875 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.876 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.876 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:25.877 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.877 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.878 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.878 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.879 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:25.879 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.880 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.880 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:25.881 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.881 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.882 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.882 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:25.883 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.883 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.884 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.884 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.884 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:25.885 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.885 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.886 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.886 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.887 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.887 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.888 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.889 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.889 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.890 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:25.890 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.891 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:25.891 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:25.891 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:25.892 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.892 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:59:49Z" 2022-09-13T19:12:25.893 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:25.893 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:25.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:25.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.895 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.858000_209.1 2022-09-13T19:12:25.896 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.896 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:54.987+0000 7f3fa4741600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T07:59:54.988+0000 7f3fa4741600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:25.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:25.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:25.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:25.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:25.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:25.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:25.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:25.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:25.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:25.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:25.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:25.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:25.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T07:59:54Z" 2022-09-13T19:12:25.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:25.929 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:25.929 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:25.930 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:25.931 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:25.931 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.932 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.932 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.933 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:25.933 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.934 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:25.934 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.935 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.935 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.935 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.936 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:25.936 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.937 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.938 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.938 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.938 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.939 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:25.939 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.940 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.940 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:25.941 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.941 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.941 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.942 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:25.942 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.943 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.943 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.944 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.944 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:25.945 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.945 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.945 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.861945_210.1", 2022-09-13T19:12:25.946 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.946 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.947 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.947 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.861945Z", 2022-09-13T19:12:25.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.948 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.948 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.949 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.949 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:25.950 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.950 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.950 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.951 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.951 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:25.952 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.952 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.953 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.953 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.954 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:25.954 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:25.955 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.955 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:25.955 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:25.956 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:25.956 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.957 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T07:59:54Z" 2022-09-13T19:12:25.957 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:25.958 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:25.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.861945_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.861945Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:25.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.861945_210.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:25.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.861945_210.1 2022-09-13T19:12:25.960 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.960 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:25.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:25.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:25.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:25.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:00.141+0000 7f2382c0b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:00.142+0000 7f2382c0b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:25.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:25.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:25.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:25.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:25.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:25.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:25.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:25.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:25.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:25.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:25.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:25.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:25.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:25.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:25.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:25.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:25.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:25.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:25.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:25.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:25.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:25.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:25.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:25.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:25.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:00:00Z" 2022-09-13T19:12:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:25.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:25.993 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:25.993 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:25.993 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:25.994 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:25.994 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:25.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:25.995 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:25.996 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:25.996 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:25.996 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:25.997 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:25.997 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:25.998 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:25.998 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:25.999 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:25.999 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:25.999 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.000 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.000 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.001 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.001 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.002 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:26.002 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.002 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.003 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:26.003 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.004 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.004 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.005 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:26.005 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.006 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.006 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.007 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.007 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:26.007 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.008 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.008 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:26.009 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.009 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.010 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.010 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:26.010 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.011 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.011 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.012 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.012 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:26.013 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.013 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.013 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.014 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.014 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.015 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.015 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.016 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.016 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.016 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.017 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:26.017 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.018 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:26.018 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:26.019 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:26.019 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.019 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:00:00Z" 2022-09-13T19:12:26.020 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:26.020 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:26.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:26.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055099.070128_219.1 2022-09-13T19:12:26.023 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.023 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:26.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:26.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:26.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:05.290+0000 7f8f6a35b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:05.291+0000 7f8f6a35b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:26.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:26.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:26.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:26.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:26.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:26.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:26.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:26.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:26.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:26.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:26.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:26.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:26.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:26.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:26.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:26.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:26.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:00:05Z" 2022-09-13T19:12:26.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:26.056 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:26.057 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:26.058 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:26.058 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:26.059 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.059 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.059 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.060 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:26.060 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.061 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:26.061 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.062 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.062 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.063 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:26.063 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.064 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.064 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.065 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.065 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.066 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.066 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:26.067 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.067 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.068 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:26.068 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.068 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.069 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.069 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:26.070 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.070 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.071 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.071 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.072 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:26.072 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.072 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.073 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.858000_209.1", 2022-09-13T19:12:26.073 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.074 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.074 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.075 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.858000Z", 2022-09-13T19:12:26.075 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.076 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.076 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.077 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.077 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:26.078 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.078 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.079 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.079 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.080 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.080 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.081 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.081 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.082 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.082 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.083 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:26.083 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.084 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:26.084 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:26.085 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:26.085 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.085 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:00:05Z" 2022-09-13T19:12:26.086 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:26.086 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:26.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.858000_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.858000Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:26.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.858000_209.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.858000_209.1 2022-09-13T19:12:26.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:26.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:26.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:10.444+0000 7f4676abd600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:10.445+0000 7f4676abd600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:26.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:26.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:26.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:26.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:26.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:26.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:26.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:26.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:26.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:26.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:26.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:26.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:26.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:26.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:00:10Z" 2022-09-13T19:12:26.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.135 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:26.135 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:26.136 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:26.137 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:26.138 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:26.138 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.139 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.140 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.141 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:26.141 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.143 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:26.143 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.145 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.145 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.146 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.147 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:26.148 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.148 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.149 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.149 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.150 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.151 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.151 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:26.151 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.152 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.152 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:26.153 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.153 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.153 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.154 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:26.154 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.155 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.155 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.155 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.156 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:26.156 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.157 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.157 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:26.158 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.158 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.158 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.159 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:26.160 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.160 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.160 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.161 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.161 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:26.162 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.162 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.162 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.163 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.163 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.164 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.164 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.164 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.165 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.165 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.166 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:26.166 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.166 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:26.167 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:26.167 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:26.168 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.168 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:00:10Z" 2022-09-13T19:12:26.168 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:26.169 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:26.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:26.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:26.171 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.171 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:26.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:15.581+0000 7f9f24394600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:15.582+0000 7f9f24394600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:26.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:26.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:26.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:26.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:26.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:26.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:26.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:26.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:26.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:26.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:26.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:26.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:26.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:26.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:00:15Z" 2022-09-13T19:12:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:26.211 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:26.211 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:26.212 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:26.212 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:26.213 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.213 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.214 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.214 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:26.214 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.215 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:26.215 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.216 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.216 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.217 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.217 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:26.218 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.218 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.218 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.219 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.219 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.220 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.220 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:26.221 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.221 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.222 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:26.222 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.223 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.223 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.224 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:26.224 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.225 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.225 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.226 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.226 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:26.227 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.227 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.228 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.834293_206.1", 2022-09-13T19:12:26.228 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.229 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.229 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.230 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.834293Z", 2022-09-13T19:12:26.230 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.231 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.231 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.232 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.233 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:26.234 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.234 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.235 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.235 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.236 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.236 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.238 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.239 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.240 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.241 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.241 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:26.242 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.243 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:26.244 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:26.245 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:26.246 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.246 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:00:15Z" 2022-09-13T19:12:26.247 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:26.248 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:26.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.834293_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.834293Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:26.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.834293_206.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.834293_206.1 2022-09-13T19:12:26.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:26.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:26.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:26.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:20.711+0000 7f9f42b43600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:20.712+0000 7f9f42b43600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:26.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:26.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:26.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:26.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:26.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:26.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:26.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:26.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:26.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:26.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:26.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:26.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:26.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:00:20Z" 2022-09-13T19:12:26.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:26.282 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:26.283 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:26.283 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:26.284 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:26.284 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.284 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.285 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.285 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:26.286 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.287 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:26.287 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.288 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.288 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.289 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.289 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:26.289 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.290 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.290 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.291 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.291 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.292 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.292 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:26.293 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.294 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.295 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:26.296 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.297 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.298 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.298 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:26.299 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.300 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.301 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.302 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.302 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:26.303 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.303 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.304 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.854520_207.1", 2022-09-13T19:12:26.304 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.304 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.305 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.305 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.854520Z", 2022-09-13T19:12:26.306 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.306 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.307 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.307 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.307 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:26.308 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.308 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.309 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.309 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.309 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.310 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.310 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.311 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.311 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.311 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.312 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:26.312 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.313 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:26.313 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:26.314 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:26.314 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.314 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:00:20Z" 2022-09-13T19:12:26.315 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:26.315 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:26.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.854520_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.854520Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:26.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.854520_207.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.854520_207.1 2022-09-13T19:12:26.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:26.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:26.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:26.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:25.944+0000 7fa6e1ca7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:25.946+0000 7fa6e1ca7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:26.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:26.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:26.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:26.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:26.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:26.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:26.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:26.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:26.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:26.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:26.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:26.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:26.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:26.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:26.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:26.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:00:25Z" 2022-09-13T19:12:26.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:26.349 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:26.349 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:26.349 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:26.350 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:26.350 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.351 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.351 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.351 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:26.352 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.352 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:26.353 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.353 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.354 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.354 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.354 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:26.355 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.355 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.356 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.356 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.356 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.357 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.357 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:26.358 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.358 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.359 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:26.359 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.359 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.360 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.360 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:26.361 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.361 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.361 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.362 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.362 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:26.363 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.363 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.364 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.832045_205.1", 2022-09-13T19:12:26.364 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.364 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.365 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.365 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.832045Z", 2022-09-13T19:12:26.366 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.366 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.367 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.367 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.367 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:26.368 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.368 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.369 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.369 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.369 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.370 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.370 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.371 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.371 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.372 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.372 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:26.372 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.373 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:26.373 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:26.374 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:26.374 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.374 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:00:25Z" 2022-09-13T19:12:26.375 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:26.375 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:26.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.832045_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.832045Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:26.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055720.804779_308.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.832045_205.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663055720.804779_308.1 target=1_1663055098.832045_205.1 2022-09-13T19:12:26.377 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:26.378 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:26.379 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:26.379 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_zone_redirect 2022-09-13T19:12:26.380 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:26.382 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:26.383 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:26.383 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:26.383 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1027, in test_multi_zone_redirect 2022-09-13T19:12:26.384 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2022-09-13T19:12:26.385 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:26.385 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2022-09-13T19:12:26.385 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 196, in zone_meta_checkpoint 2022-09-13T19:12:26.386 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2022-09-13T19:12:26.386 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2022-09-13T19:12:26.387 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:26.387 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T19:12:26.388 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T19:12:26.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-13T19:12:26.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:26.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:26.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:31.086+0000 7f13e6059600 0 failed reading obj info from .rgw.root:zone_info.4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9: (2) No such file or directory 2022-09-13T19:12:26.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:31.086+0000 7f13e6059600 0 WARNING: could not read zone params for zone id=4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9 name=test-zone3 2022-09-13T19:12:26.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:31.086+0000 7f13e6059600 0 failed reading obj info from .rgw.root:zone_info.7103daa4-1b45-47a1-bb49-5d9877f6d137: (2) No such file or directory 2022-09-13T19:12:26.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:00:31.086+0000 7f13e6059600 0 WARNING: could not read zone params for zone id=7103daa4-1b45-47a1-bb49-5d9877f6d137 name=test-zone1 2022-09-13T19:12:26.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "id": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T19:12:26.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "name": "test-zone2", 2022-09-13T19:12:26.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "domain_root": "test-zone2.rgw.meta:root", 2022-09-13T19:12:26.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "control_pool": "test-zone2.rgw.control", 2022-09-13T19:12:26.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-13T19:12:26.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-13T19:12:26.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "log_pool": "test-zone2.rgw.log", 2022-09-13T19:12:26.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-13T19:12:26.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-13T19:12:26.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-13T19:12:26.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-13T19:12:26.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-13T19:12:26.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-13T19:12:26.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-13T19:12:26.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-13T19:12:26.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "otp_pool": "test-zone2.rgw.otp", 2022-09-13T19:12:26.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "system_key": { 2022-09-13T19:12:26.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "access_key": "G693TB5XEJ3206QY", 2022-09-13T19:12:26.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "secret_key": "QS5zddodwd26OlSDjoxox3W0BzgFaGzV" 2022-09-13T19:12:26.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "placement_pools": [ 2022-09-13T19:12:26.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": "default-placement", 2022-09-13T19:12:26.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-13T19:12:26.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "storage_classes": { 2022-09-13T19:12:26.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "STANDARD": { 2022-09-13T19:12:26.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-13T19:12:26.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "compression_type": "random" 2022-09-13T19:12:26.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-13T19:12:26.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "index_type": 0 2022-09-13T19:12:26.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ], 2022-09-13T19:12:26.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T19:12:26.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-13T19:12:26.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T19:12:26.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-13T19:12:26.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-13T19:12:26.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:26.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:26.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: Sending period to new master zone 7103daa4-1b45-47a1-bb49-5d9877f6d137 2022-09-13T19:12:26.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "id": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "epoch": 4, 2022-09-13T19:12:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "predecessor_uuid": "0f1e9404-34a5-47ad-8196-42a4e6e2ff9c", 2022-09-13T19:12:26.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": [], 2022-09-13T19:12:26.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period_map": { 2022-09-13T19:12:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "id": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "zonegroups": [ 2022-09-13T19:12:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "id": "test-zonegroup", 2022-09-13T19:12:26.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "name": "test-zonegroup", 2022-09-13T19:12:26.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "api_name": "", 2022-09-13T19:12:26.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "is_master": true, 2022-09-13T19:12:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "endpoints": [ 2022-09-13T19:12:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "http://smithi036.front.sepia.ceph.com:8000" 2022-09-13T19:12:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ], 2022-09-13T19:12:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "hostnames": [], 2022-09-13T19:12:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "hostnames_s3website": [], 2022-09-13T19:12:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "master_zone": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T19:12:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "zones": [ 2022-09-13T19:12:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "id": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T19:12:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "name": "test-zone2", 2022-09-13T19:12:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "endpoints": [ 2022-09-13T19:12:26.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "http://smithi083.front.sepia.ceph.com:8000" 2022-09-13T19:12:26.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ], 2022-09-13T19:12:26.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "log_meta": false, 2022-09-13T19:12:26.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "log_data": true, 2022-09-13T19:12:26.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "bucket_index_max_shards": 11, 2022-09-13T19:12:26.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "read_only": false, 2022-09-13T19:12:26.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "tier_type": "", 2022-09-13T19:12:26.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_from_all": false, 2022-09-13T19:12:26.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_from": [], 2022-09-13T19:12:26.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "redirect_zone": "", 2022-09-13T19:12:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "supported_features": [ 2022-09-13T19:12:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "resharding" 2022-09-13T19:12:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "id": "4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9", 2022-09-13T19:12:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "name": "test-zone3", 2022-09-13T19:12:26.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "endpoints": [ 2022-09-13T19:12:26.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "http://smithi036.front.sepia.ceph.com:8001" 2022-09-13T19:12:26.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ], 2022-09-13T19:12:26.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "log_meta": false, 2022-09-13T19:12:26.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "log_data": true, 2022-09-13T19:12:26.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "bucket_index_max_shards": 11, 2022-09-13T19:12:26.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "read_only": false, 2022-09-13T19:12:26.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "tier_type": "", 2022-09-13T19:12:26.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_from_all": true, 2022-09-13T19:12:26.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_from": [], 2022-09-13T19:12:26.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "redirect_zone": "", 2022-09-13T19:12:26.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "supported_features": [ 2022-09-13T19:12:26.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "resharding" 2022-09-13T19:12:26.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "id": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T19:12:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "name": "test-zone1", 2022-09-13T19:12:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "endpoints": [ 2022-09-13T19:12:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "http://smithi036.front.sepia.ceph.com:8000" 2022-09-13T19:12:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ], 2022-09-13T19:12:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "log_meta": false, 2022-09-13T19:12:26.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "log_data": true, 2022-09-13T19:12:26.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "bucket_index_max_shards": 11, 2022-09-13T19:12:26.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "read_only": false, 2022-09-13T19:12:26.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "tier_type": "", 2022-09-13T19:12:26.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_from_all": true, 2022-09-13T19:12:26.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_from": [], 2022-09-13T19:12:26.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "redirect_zone": "", 2022-09-13T19:12:26.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "supported_features": [ 2022-09-13T19:12:26.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "resharding" 2022-09-13T19:12:26.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ], 2022-09-13T19:12:26.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "placement_targets": [ 2022-09-13T19:12:26.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "name": "default-placement", 2022-09-13T19:12:26.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "tags": [], 2022-09-13T19:12:26.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "storage_classes": [ 2022-09-13T19:12:26.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "STANDARD" 2022-09-13T19:12:26.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ], 2022-09-13T19:12:26.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "default_placement": "default-placement", 2022-09-13T19:12:26.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T19:12:26.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_policy": { 2022-09-13T19:12:26.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "groups": [] 2022-09-13T19:12:26.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "enabled_features": [ 2022-09-13T19:12:26.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "resharding" 2022-09-13T19:12:26.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ], 2022-09-13T19:12:26.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "short_zone_ids": [ 2022-09-13T19:12:26.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": "30ddd076-3f98-4411-9a5a-48d7fd04ad75", 2022-09-13T19:12:26.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": 2827250327 2022-09-13T19:12:26.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": "4a27707f-ed7f-410e-9a35-1ab5ea4aa5f9", 2022-09-13T19:12:26.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": 1723930414 2022-09-13T19:12:26.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T19:12:26.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": 2837352077 2022-09-13T19:12:26.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "master_zonegroup": "test-zonegroup", 2022-09-13T19:12:26.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "master_zone": "7103daa4-1b45-47a1-bb49-5d9877f6d137", 2022-09-13T19:12:26.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period_config": { 2022-09-13T19:12:26.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "bucket_quota": { 2022-09-13T19:12:26.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "enabled": false, 2022-09-13T19:12:26.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "check_on_raw": false, 2022-09-13T19:12:26.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_size": -1, 2022-09-13T19:12:26.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_size_kb": 0, 2022-09-13T19:12:26.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_objects": -1 2022-09-13T19:12:26.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "user_quota": { 2022-09-13T19:12:26.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "enabled": false, 2022-09-13T19:12:26.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "check_on_raw": false, 2022-09-13T19:12:26.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_size": -1, 2022-09-13T19:12:26.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_size_kb": 0, 2022-09-13T19:12:26.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_objects": -1 2022-09-13T19:12:26.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "user_ratelimit": { 2022-09-13T19:12:26.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_read_ops": 0, 2022-09-13T19:12:26.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_write_ops": 0, 2022-09-13T19:12:26.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_read_bytes": 0, 2022-09-13T19:12:26.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_write_bytes": 0, 2022-09-13T19:12:26.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "enabled": false 2022-09-13T19:12:26.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "bucket_ratelimit": { 2022-09-13T19:12:26.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_read_ops": 0, 2022-09-13T19:12:26.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_write_ops": 0, 2022-09-13T19:12:26.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_read_bytes": 0, 2022-09-13T19:12:26.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_write_bytes": 0, 2022-09-13T19:12:26.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "enabled": false 2022-09-13T19:12:26.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "anonymous_ratelimit": { 2022-09-13T19:12:26.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_read_ops": 0, 2022-09-13T19:12:26.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_write_ops": 0, 2022-09-13T19:12:26.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_read_bytes": 0, 2022-09-13T19:12:26.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "max_write_bytes": 0, 2022-09-13T19:12:26.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "enabled": false 2022-09-13T19:12:26.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_id": "eb5d388e-413c-4d2e-adbc-d8ac71480455", 2022-09-13T19:12:26.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_name": "test-realm", 2022-09-13T19:12:26.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: Set sync_from_all flag on zone test-zone2 to false 2022-09-13T19:12:26.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=yxvvuh-45 2022-09-13T19:12:26.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-45/ 2022-09-13T19:12:26.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-45/ 2022-09-13T19:12:26.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:26.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-45/ 2022-09-13T19:12:26.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:26.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:26.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:26.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:26.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:26.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:26.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:26.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:26.520 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:26.521 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:26.522 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:26.522 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:01:31 GMT 2022-09-13T19:12:26.523 INFO:tasks.rgw_multisite_tests:/yxvvuh-45/ 2022-09-13T19:12:26.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:26.525 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:rGQ51Cjt3r7RJwjnDr1kkXsulRw= 2022-09-13T19:12:26.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:01:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rGQ51Cjt3r7RJwjnDr1kkXsulRw=', 'Content-Length': '0'} 2022-09-13T19:12:26.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000000c407a3c89309934-00632038db-140b-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:01:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:26.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-45/testredirect 2022-09-13T19:12:26.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-45/testredirect 2022-09-13T19:12:26.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:26.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-45/testredirect 2022-09-13T19:12:26.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:26.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-13T19:12:26.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:26.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:26.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:26.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:26.536 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:26.537 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2022-09-13T19:12:26.538 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:26.539 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:01:31 GMT 2022-09-13T19:12:26.540 INFO:tasks.rgw_multisite_tests:/yxvvuh-45/testredirect 2022-09-13T19:12:26.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:26.542 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:woq36g5w5nUY99CwthhyuzS4zyo= 2022-09-13T19:12:26.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 08:01:31 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:woq36g5w5nUY99CwthhyuzS4zyo='} 2022-09-13T19:12:26.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b9606f3beee4312-00632038db-140b-test-zone1'), ('Date', 'Tue, 13 Sep 2022 08:01:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:26.544 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.545 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:26.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:26.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:26.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:26.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:26.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:26.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:26.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:26.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:26.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663056091.338062_357.1", 2022-09-13T19:12:26.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T08:01:31.338062Z" 2022-09-13T19:12:26.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:26.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:26.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:26.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:26.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:26.566 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.567 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:26.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:26.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:01:31.636+0000 7f2591f61600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:01:31.637+0000 7f2591f61600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:26.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:26.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:26.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:26.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:26.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:26.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:26.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:26.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:26.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:26.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:26.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:26.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:26.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:26.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:26.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:26.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:01:31Z" 2022-09-13T19:12:26.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:26.613 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:26.613 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:26.614 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:26.614 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:26.615 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.615 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.616 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.616 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:26.616 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.617 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:26.617 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.618 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.618 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.619 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.619 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:26.619 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.620 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.620 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.621 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.621 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.622 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.622 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:26.622 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.623 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.623 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:26.624 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.624 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.625 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:26.625 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.626 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.626 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.626 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.627 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:26.627 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.628 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.628 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:26.628 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.629 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.629 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.630 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:26.630 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.630 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.631 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.631 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.632 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:26.632 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.633 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.634 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.634 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.634 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.635 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.635 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.636 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.636 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.637 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:26.637 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.638 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:26.638 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:26.639 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:26.639 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.640 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:01:31Z" 2022-09-13T19:12:26.640 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:26.640 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:26.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:26.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T19:12:26.643 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.643 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:26.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:26.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:26.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:01:36.861+0000 7efff9ef5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:01:36.862+0000 7efff9ef5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:26.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:26.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:26.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:26.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:26.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.105395_221.1", 2022-09-13T19:12:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T19:12:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:26.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:26.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:26.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:26.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:01:36Z" 2022-09-13T19:12:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:26.674 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:26.674 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:26.675 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:26.675 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:26.676 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.676 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.677 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.677 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:26.677 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.678 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:26.678 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.679 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.679 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.679 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.680 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:26.680 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.681 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.681 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.681 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.682 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.682 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.683 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:26.683 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.683 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.684 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:26.684 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.685 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.685 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.685 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:26.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.686 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.687 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.687 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.687 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:26.688 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.688 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.689 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.105395_221.1", 2022-09-13T19:12:26.689 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.689 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.690 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.690 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T19:12:26.691 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.691 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.692 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.693 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.695 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:26.696 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.696 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.697 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.697 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.698 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.698 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.699 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.699 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.700 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.700 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.701 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:26.701 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.702 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:26.702 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:26.703 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:26.703 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.704 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:01:36Z" 2022-09-13T19:12:26.704 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:26.705 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:26.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.105395_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.105395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:26.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.105395_221.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.105395_221.1 2022-09-13T19:12:26.707 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.708 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:26.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:26.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:26.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:01:42.001+0000 7fe74b018600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:01:42.002+0000 7fe74b018600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:26.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:26.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:26.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:26.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:26.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:26.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:26.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:26.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:26.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:26.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:26.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.340682_237.1", 2022-09-13T19:12:26.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T19:12:26.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:26.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:26.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:26.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:26.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:01:42Z" 2022-09-13T19:12:26.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:26.741 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:26.741 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:26.742 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:26.742 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:26.743 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.743 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.743 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.744 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:26.744 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.745 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:26.745 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.745 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.746 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.746 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.747 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:26.747 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.747 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.748 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.748 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.749 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.749 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.749 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:26.750 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.750 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.751 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:26.751 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.751 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.752 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:26.753 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.753 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.753 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.754 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.754 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:26.755 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.755 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.756 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.340682_237.1", 2022-09-13T19:12:26.756 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.757 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.757 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.758 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T19:12:26.758 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.758 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.759 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.759 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.760 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:26.760 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.760 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.761 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.762 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.763 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.763 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.763 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.764 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.764 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:26.765 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.765 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:26.765 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:26.766 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:26.766 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.767 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:01:42Z" 2022-09-13T19:12:26.767 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:26.768 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:26.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.340682_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.340682Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:26.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.340682_237.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.340682_237.1 2022-09-13T19:12:26.770 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.770 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:26.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:26.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:26.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:01:47.138+0000 7ff907561600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:01:47.142+0000 7ff907561600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:26.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:26.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:26.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:26.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:26.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:26.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:26.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:26.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:26.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:26.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:26.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:26.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:26.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:26.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:26.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:26.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:01:47Z" 2022-09-13T19:12:26.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:26.801 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:26.802 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:26.802 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:26.802 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:26.803 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.803 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.804 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.804 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:26.805 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.805 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:26.806 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.806 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.807 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.807 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.808 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:26.808 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.809 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.809 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.809 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.810 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.810 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.811 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:26.811 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.812 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.812 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:26.813 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.813 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.814 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.814 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:26.814 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.826 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.826 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.827 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.827 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:26.828 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.828 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.829 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:26.829 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.830 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.830 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.831 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:26.831 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.832 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.832 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.833 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.833 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:26.833 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.834 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.834 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.835 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.835 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.836 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.836 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.837 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.837 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.838 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.838 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:26.838 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.839 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:26.839 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:26.840 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:26.840 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.840 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:01:47Z" 2022-09-13T19:12:26.841 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:26.841 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:26.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:26.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:26.844 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.844 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:26.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:26.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:01:52.465+0000 7f5b1f3d5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:01:52.466+0000 7f5b1f3d5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:26.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:26.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:26.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:26.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:26.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:26.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:26.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:26.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:26.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.146590_228.1", 2022-09-13T19:12:26.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.146590Z", 2022-09-13T19:12:26.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:26.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:26.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:26.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:26.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:01:52Z" 2022-09-13T19:12:26.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:26.875 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:26.875 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:26.876 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:26.876 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:26.876 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.877 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.877 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.878 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:26.878 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.879 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:26.879 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.879 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.880 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.880 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.881 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:26.881 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.881 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.882 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.882 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.883 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.883 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.883 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:26.884 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.884 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.885 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:26.885 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.885 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.886 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.886 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:26.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.887 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.888 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.888 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.888 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:26.889 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.889 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.890 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.146590_228.1", 2022-09-13T19:12:26.890 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.890 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.891 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.891 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.146590Z", 2022-09-13T19:12:26.892 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.892 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.892 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.893 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.894 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:26.894 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.895 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.895 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.896 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.897 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.898 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.899 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.900 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.900 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.901 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.902 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:26.903 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.903 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:26.904 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:26.905 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:26.906 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.906 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:01:52Z" 2022-09-13T19:12:26.907 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:26.908 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:26.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.146590_228.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.146590Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:26.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.146590_228.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.146590_228.1 2022-09-13T19:12:26.912 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.912 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:26.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:26.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:26.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:01:57.611+0000 7fe899853600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:01:57.612+0000 7fe899853600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:26.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:26.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:26.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:26.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:26.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:26.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:26.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:26.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:26.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:26.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:26.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:26.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:26.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:26.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:26.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:26.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:26.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:26.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:26.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:26.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:26.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:26.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:26.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:26.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:26.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:01:57Z" 2022-09-13T19:12:26.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:26.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:26.955 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:26.956 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:26.956 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:26.957 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:26.958 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:26.958 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.959 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.960 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:26.961 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.962 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:26.962 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.963 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.964 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:26.965 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.965 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:26.966 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.967 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:26.968 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.968 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.969 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.970 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.971 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:26.971 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.972 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.973 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:26.974 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.974 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.975 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.975 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:26.976 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.977 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.977 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.978 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.978 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:26.979 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.979 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.980 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:26.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.981 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.982 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:26.982 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.983 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.983 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.984 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:26.984 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:26.985 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:26.985 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:26.985 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:26.986 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:26.986 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:26.987 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:26.987 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:26.988 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:26.988 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.988 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:26.989 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:26.989 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.989 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:26.990 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:26.990 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:26.991 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:26.991 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:01:57Z" 2022-09-13T19:12:26.992 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:26.992 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:26.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:26.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:26.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:26.994 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.995 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:26.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:26.996 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi036.stdout: INFO: 2022-09-13T08:02:02.696+0000 7f8f5e7a1700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-09-13T19:12:26.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:26.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:26.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:02.733+0000 7fb2b9ba8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:02.734+0000 7fb2b9ba8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:26.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:26.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:26.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:27.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:27.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:27.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:27.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:27.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:27.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:27.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:27.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:27.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.107994_222.1", 2022-09-13T19:12:27.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.107994Z", 2022-09-13T19:12:27.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:27.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:27.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:02:02Z" 2022-09-13T19:12:27.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:27.033 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:27.034 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:27.035 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:27.035 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:27.035 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.036 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.037 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:27.037 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.037 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:27.038 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.039 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.039 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.039 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.040 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:27.040 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.041 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.041 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.041 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.042 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.043 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.043 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:27.044 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.045 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.046 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:27.046 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.047 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.048 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.048 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:27.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.049 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.050 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.050 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.050 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:27.051 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.051 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.052 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.107994_222.1", 2022-09-13T19:12:27.052 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.052 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.053 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.053 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.107994Z", 2022-09-13T19:12:27.054 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.054 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.055 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.055 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.055 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:27.056 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.056 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.057 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.057 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.058 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.058 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.058 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.059 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.059 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.060 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.060 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:27.060 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.061 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:27.061 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:27.062 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:27.062 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.063 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:02:02Z" 2022-09-13T19:12:27.063 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:27.063 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:27.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.107994_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.107994Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:27.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.107994_222.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.107994_222.1 2022-09-13T19:12:27.065 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:07.950+0000 7f1a24c9a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:07.953+0000 7f1a24c9a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:27.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:27.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:27.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:27.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:27.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:27.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:27.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:27.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:27.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:27.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:27.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:02:07Z" 2022-09-13T19:12:27.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:27.099 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:27.100 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:27.100 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:27.101 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:27.101 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.102 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.102 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.102 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:27.103 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.103 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:27.104 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.104 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.104 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.105 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.105 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:27.106 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.106 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.106 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.107 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.107 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.108 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.108 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:27.108 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.109 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.109 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:27.110 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.110 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.110 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.111 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:27.111 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.112 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.112 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.112 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.113 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:27.113 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.114 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.114 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:27.115 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.115 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.115 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.116 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:27.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.117 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.117 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.117 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.118 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:27.118 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.119 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.119 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.119 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.120 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.120 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.121 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.121 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.121 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.122 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.122 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:27.123 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.123 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:27.123 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:27.124 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:27.124 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.125 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:02:07Z" 2022-09-13T19:12:27.125 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:27.125 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:27.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:27.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:27.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:27.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:27.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:27.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:13.152+0000 7fceec64e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:13.153+0000 7fceec64e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:27.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:27.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:27.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:27.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:27.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:27.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:27.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:27.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:27.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.340682_237.1", 2022-09-13T19:12:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T19:12:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:27.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:27.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:27.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:27.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:27.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:02:13Z" 2022-09-13T19:12:27.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:27.158 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:27.158 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:27.159 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:27.159 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:27.159 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.160 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.160 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.161 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:27.161 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.161 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:27.162 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.162 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.163 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.163 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.163 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:27.164 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.164 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.165 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.165 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.165 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.166 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.166 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:27.167 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.167 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.167 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:27.168 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.168 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.169 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.169 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:27.169 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.170 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.170 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.171 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.171 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:27.171 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.172 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.172 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.340682_237.1", 2022-09-13T19:12:27.173 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.173 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.173 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.174 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T19:12:27.174 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.175 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.175 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.176 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.176 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:27.176 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.177 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.177 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.178 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.178 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.178 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.179 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.179 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.180 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.180 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.180 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:27.181 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.181 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:27.182 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:27.182 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:27.182 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.183 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:02:13Z" 2022-09-13T19:12:27.183 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:27.184 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:27.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.340682_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.340682Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:27.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.340682_237.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.340682_237.1 2022-09-13T19:12:27.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:18.312+0000 7f4c454e6600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:18.313+0000 7f4c454e6600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:27.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:27.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:27.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:27.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:27.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:27.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:27.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:27.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:27.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:27.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:27.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:02:18Z" 2022-09-13T19:12:27.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:27.219 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:27.220 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:27.220 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:27.221 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:27.221 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.222 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.222 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.222 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:27.223 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.223 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:27.224 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.224 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.225 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.225 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.226 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:27.226 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.227 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.227 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.228 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.229 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.229 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.229 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:27.230 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.231 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.232 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:27.232 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.233 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.233 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.234 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:27.234 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.235 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.235 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.236 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.237 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:27.237 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.238 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.239 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:27.239 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.240 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.240 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.241 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:27.242 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.243 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.243 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.244 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.245 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:27.245 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.246 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.247 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.247 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.248 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.250 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.250 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.251 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.251 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.252 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:27.253 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.253 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:27.254 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:27.255 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:27.255 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.255 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:02:18Z" 2022-09-13T19:12:27.256 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:27.256 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:27.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:27.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.990623_217.1 2022-09-13T19:12:27.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:23.487+0000 7f5e914df600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:23.488+0000 7f5e914df600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:27.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:27.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:27.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:27.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:27.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:27.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:27.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:27.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.182451_229.1", 2022-09-13T19:12:27.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T19:12:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:27.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:27.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:27.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:27.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:02:23Z" 2022-09-13T19:12:27.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:27.304 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:27.304 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:27.305 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:27.306 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:27.307 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.308 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.308 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.309 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:27.310 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.311 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:27.311 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.312 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.312 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.313 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.314 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:27.315 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.316 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.317 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.318 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.318 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.319 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.320 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:27.321 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.322 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.322 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:27.323 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.324 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.325 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.325 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:27.326 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.327 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.328 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.329 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.330 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:27.330 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.331 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.331 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.182451_229.1", 2022-09-13T19:12:27.332 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.332 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.332 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.333 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T19:12:27.334 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.334 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.334 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.335 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.335 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:27.336 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.336 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.337 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.337 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.337 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.338 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.338 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.339 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.339 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.340 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.340 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:27.340 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.341 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:27.341 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:27.342 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:27.342 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.343 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:02:23Z" 2022-09-13T19:12:27.343 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:27.344 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:27.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.182451_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.182451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:27.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.182451_229.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.182451_229.1 2022-09-13T19:12:27.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:27.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:28.632+0000 7fe6174b7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:28.633+0000 7fe6174b7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:27.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:27.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:27.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:27.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.187352_231.1", 2022-09-13T19:12:27.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T19:12:27.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:27.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:27.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:27.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:27.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:27.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:02:28Z" 2022-09-13T19:12:27.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:27.389 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:27.389 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:27.390 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:27.391 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:27.392 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.393 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.393 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.394 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:27.394 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.395 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:27.395 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.396 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.397 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.398 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.399 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:27.399 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.400 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.401 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.402 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.403 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.404 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.405 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:27.405 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.406 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.406 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:27.407 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.407 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.408 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.408 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:27.409 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.409 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.409 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.410 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.410 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:27.411 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.411 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.412 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.187352_231.1", 2022-09-13T19:12:27.412 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.412 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.413 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.413 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T19:12:27.414 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.414 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.414 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.415 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.415 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:27.416 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.416 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.417 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.417 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.417 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.418 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.418 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.419 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.419 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.420 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.420 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:27.420 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.421 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:27.421 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:27.422 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:27.422 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.422 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:02:28Z" 2022-09-13T19:12:27.423 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:27.423 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:27.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.187352_231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.187352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:27.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.187352_231.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.187352_231.1 2022-09-13T19:12:27.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:33.761+0000 7fcf35a6c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:33.762+0000 7fcf35a6c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:27.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:27.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:27.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:27.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:27.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:27.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:27.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:27.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:27.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:27.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:27.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:02:33Z" 2022-09-13T19:12:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:27.468 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:27.469 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:27.470 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:27.471 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:27.471 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.472 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.473 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.474 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:27.475 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.475 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:27.476 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.477 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.477 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.478 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.479 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:27.480 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.481 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.481 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.482 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.482 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.483 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.483 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:27.483 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.484 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.484 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:27.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.485 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.486 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.487 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:27.487 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.487 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.488 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.488 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.489 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:27.489 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.490 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.490 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:27.491 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.491 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.491 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.492 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:27.493 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.493 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.494 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.495 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.496 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:27.497 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.497 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.498 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.499 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.500 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.501 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.501 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.502 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.503 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.504 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.504 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:27.505 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.506 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:27.507 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:27.507 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:27.508 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.509 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:02:33Z" 2022-09-13T19:12:27.510 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:27.510 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:27.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:27.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:27.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.515 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:27.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:27.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:27.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:38.935+0000 7fc53bfa7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:38.936+0000 7fc53bfa7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:27.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:27.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:27.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:27.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:27.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:27.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:27.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:27.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:27.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:27.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:27.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:27.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:02:38Z" 2022-09-13T19:12:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:27.554 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:27.555 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:27.555 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:27.556 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:27.556 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.557 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.557 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.557 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:27.558 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.558 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:27.559 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.559 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.560 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.560 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.561 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:27.561 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.562 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.562 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.563 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.563 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.563 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.564 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:27.564 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.565 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.565 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:27.565 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.566 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.566 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.567 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:27.567 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.568 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.568 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.568 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.569 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:27.569 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.570 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.570 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:27.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.571 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.571 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.572 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:27.573 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.573 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.573 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.574 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.574 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:27.575 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.575 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.576 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.576 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.577 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.577 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.578 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.578 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.579 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.580 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.581 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:27.582 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.582 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:27.583 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:27.583 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:27.584 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.584 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:02:38Z" 2022-09-13T19:12:27.585 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:27.585 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:27.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:27.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.953175_216.1 2022-09-13T19:12:27.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:27.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:27.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:27.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:44.080+0000 7f0e772b3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:44.081+0000 7f0e772b3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:27.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:27.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:27.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:27.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:27.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:27.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:27.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:27.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:27.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.109037_223.1", 2022-09-13T19:12:27.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.109037Z", 2022-09-13T19:12:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:27.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:27.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:27.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:02:44Z" 2022-09-13T19:12:27.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:27.618 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:27.618 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:27.619 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:27.619 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:27.620 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.620 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.620 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.621 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:27.621 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.622 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:27.622 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.623 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.623 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.623 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.624 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:27.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.625 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.625 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.626 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.626 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.626 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.627 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:27.627 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.628 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.628 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:27.629 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.629 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.629 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.630 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:27.630 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.631 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.631 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.631 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.632 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:27.632 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.633 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.109037_223.1", 2022-09-13T19:12:27.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.634 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.634 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.635 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.109037Z", 2022-09-13T19:12:27.635 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.636 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.636 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.636 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.637 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:27.637 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.638 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.638 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.638 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.639 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.639 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.640 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.640 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.641 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.641 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.641 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:27.642 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.642 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:27.643 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:27.643 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:27.643 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.644 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:02:44Z" 2022-09-13T19:12:27.644 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:27.645 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:27.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.109037_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.109037Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:27.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.109037_223.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.109037_223.1 2022-09-13T19:12:27.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:27.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:49.233+0000 7f178b4dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:49.234+0000 7f178b4dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:27.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:27.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:27.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:27.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:27.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:27.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:27.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:27.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:27.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:27.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:27.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:27.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:27.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:02:49Z" 2022-09-13T19:12:27.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:27.678 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:27.679 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:27.680 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:27.680 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:27.680 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.681 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.681 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.682 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:27.682 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.683 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:27.683 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.683 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.684 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.684 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.685 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:27.685 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.686 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.686 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.687 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.687 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.688 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:27.688 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.688 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.689 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:27.689 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.690 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.690 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.691 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:27.691 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.691 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.692 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.692 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.693 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:27.693 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.693 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.694 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:27.694 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.695 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.695 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.696 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:27.696 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.696 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.697 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.697 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.698 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:27.698 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.698 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.699 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.699 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.700 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.701 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.701 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.702 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.702 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.702 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:27.703 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.703 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:27.704 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:27.704 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:27.705 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.705 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:02:49Z" 2022-09-13T19:12:27.705 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:27.706 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:27.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:27.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T19:12:27.708 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.708 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:27.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:27.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:27.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:54.369+0000 7f462a0ee600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:54.370+0000 7f462a0ee600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:27.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:27.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:27.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:27.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:27.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:27.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:27.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:02:54Z" 2022-09-13T19:12:27.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:27.751 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:27.751 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:27.752 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:27.752 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:27.753 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.753 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.754 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.754 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:27.755 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.755 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:27.756 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.756 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.756 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.757 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.757 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:27.758 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.758 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.759 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.759 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.760 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.760 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.760 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:27.761 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.761 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.762 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:27.762 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.762 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.763 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.763 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:27.764 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.764 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.765 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.765 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.765 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:27.766 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.766 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.767 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:27.767 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.767 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.768 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.768 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:27.769 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.769 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.769 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.770 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.770 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:27.771 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.771 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.772 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.772 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.772 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.773 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.773 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.774 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.774 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.774 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.775 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:27.775 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.776 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:27.776 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:27.776 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:27.777 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.777 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:02:54Z" 2022-09-13T19:12:27.778 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:27.778 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:27.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:27.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.070128_219.1 2022-09-13T19:12:27.781 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.781 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:27.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:27.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:27.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:59.517+0000 7fe4d8e2e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:02:59.518+0000 7fe4d8e2e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:27.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:27.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:27.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:27.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:27.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:27.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:27.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:27.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:27.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.434813_240.1", 2022-09-13T19:12:27.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.434813Z", 2022-09-13T19:12:27.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:27.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:27.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:27.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:27.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:02:59Z" 2022-09-13T19:12:27.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:27.815 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:27.815 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:27.816 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:27.816 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:27.816 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.817 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.817 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.818 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:27.818 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.818 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:27.819 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.819 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.820 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.820 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.820 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:27.821 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.821 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.822 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.822 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.823 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.823 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.823 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:27.824 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.824 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.825 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:27.825 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.825 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.826 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.826 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:27.827 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.827 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.828 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.828 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.829 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:27.829 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.830 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.830 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.434813_240.1", 2022-09-13T19:12:27.831 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.831 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.832 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.434813Z", 2022-09-13T19:12:27.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.833 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.834 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.834 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.835 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:27.835 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.836 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.836 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.837 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.837 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.838 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.838 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.839 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.839 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.840 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.840 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:27.841 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.841 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:27.841 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:27.842 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:27.842 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.843 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:02:59Z" 2022-09-13T19:12:27.843 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:27.843 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:27.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.434813_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.434813Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:27.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.434813_240.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.434813_240.1 2022-09-13T19:12:27.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.846 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:27.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:27.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:27.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:04.652+0000 7f83f1510600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:04.653+0000 7f83f1510600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:27.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:27.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:27.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:27.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:27.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:27.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:27.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:27.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:27.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:27.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:27.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:27.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:27.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:27.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:27.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:03:04Z" 2022-09-13T19:12:27.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:27.876 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:27.876 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:27.876 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:27.877 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:27.877 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.878 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.878 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.878 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:27.879 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.879 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:27.880 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.880 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.880 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.881 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.881 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:27.882 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.882 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.883 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.883 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.883 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.884 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.884 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:27.885 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.885 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.885 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:27.886 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.886 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.887 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:27.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.888 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.888 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.889 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.889 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:27.889 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.890 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.890 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:27.891 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.891 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.891 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.892 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:27.892 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.893 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.893 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.894 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.894 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:27.894 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.895 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.895 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.896 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.896 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.897 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.897 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.898 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.898 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.899 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:27.899 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.899 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:27.900 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:27.900 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:27.901 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.901 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:03:04Z" 2022-09-13T19:12:27.901 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:27.902 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:27.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:27.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:27.904 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.904 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:27.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:27.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:27.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:09.797+0000 7f32eedba600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:09.798+0000 7f32eedba600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:27.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:27.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:27.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:27.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:27.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:27.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:27.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.187352_231.1", 2022-09-13T19:12:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T19:12:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:27.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:03:09Z" 2022-09-13T19:12:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:27.934 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:27.935 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:27.935 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:27.935 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:27.936 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.936 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.937 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.937 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:27.937 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.938 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:27.938 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.939 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.939 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.939 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.940 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:27.940 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.941 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.941 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.941 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.942 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.942 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.943 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:27.943 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.943 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.944 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:27.944 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.945 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.945 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.946 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:27.946 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.947 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.947 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.947 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.948 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:27.948 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.949 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.949 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.187352_231.1", 2022-09-13T19:12:27.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.950 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.950 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.951 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T19:12:27.951 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.951 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.952 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.952 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.953 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:27.953 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.954 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.954 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.954 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.955 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:27.955 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:27.956 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.956 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.956 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.957 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:27.957 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:27.958 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.958 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:27.959 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:27.959 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:27.959 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.960 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:03:09Z" 2022-09-13T19:12:27.960 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:27.961 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:27.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.187352_231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.187352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:27.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.187352_231.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:27.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.187352_231.1 2022-09-13T19:12:27.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:14.942+0000 7fcfef69b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:14.943+0000 7fcfef69b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:27.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:27.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:27.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:27.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:27.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:27.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:27.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:27.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.111513_224.1", 2022-09-13T19:12:27.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.111513Z", 2022-09-13T19:12:27.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:27.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:27.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:27.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:27.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:27.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:27.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:27.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:03:14Z" 2022-09-13T19:12:27.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:27.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:27.992 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:27.993 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:27.993 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:27.994 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:27.994 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:27.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:27.995 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:27.996 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:27.996 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:27.996 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:27.997 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:27.997 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:27.998 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:27.998 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:27.999 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:27.999 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.000 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.000 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.000 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.001 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.001 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.002 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.002 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.003 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.003 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.003 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.004 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.004 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.005 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.005 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.005 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.006 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.006 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.007 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.007 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.008 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.008 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.111513_224.1", 2022-09-13T19:12:28.008 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.009 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.009 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.010 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.111513Z", 2022-09-13T19:12:28.010 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.010 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.011 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.011 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.012 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.012 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.012 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.013 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.014 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.014 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.014 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.015 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.015 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.016 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.016 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.016 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.017 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.017 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.018 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.018 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.019 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:03:14Z" 2022-09-13T19:12:28.019 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.019 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.111513_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.111513Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.111513_224.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.111513_224.1 2022-09-13T19:12:28.021 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.022 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:20.065+0000 7ff30888f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:20.066+0000 7ff30888f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.187352_231.1", 2022-09-13T19:12:28.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T19:12:28.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:03:20Z" 2022-09-13T19:12:28.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.053 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.054 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.054 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.055 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.055 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.055 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.056 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.056 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.057 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.057 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.058 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.058 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.058 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.059 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.059 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.060 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.060 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.060 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.061 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.061 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.062 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.062 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.063 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.063 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.065 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.065 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.066 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.066 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.066 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.067 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.067 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.068 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.068 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.069 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.187352_231.1", 2022-09-13T19:12:28.069 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.070 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.071 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.071 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T19:12:28.071 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.072 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.072 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.073 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.073 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.074 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.075 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.076 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.076 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.077 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.077 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.078 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.078 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.079 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.079 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.079 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.080 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.080 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:03:20Z" 2022-09-13T19:12:28.081 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.081 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.081 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.187352_231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.187352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.187352_231.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.187352_231.1 2022-09-13T19:12:28.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:25.200+0000 7ff2cb06a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:25.201+0000 7ff2cb06a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:03:25Z" 2022-09-13T19:12:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.114 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.114 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.115 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.115 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.116 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.117 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.117 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.118 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.118 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.119 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.119 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.120 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.120 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.121 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.121 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.122 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.122 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.123 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.123 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.124 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.124 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.124 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.125 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.125 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.126 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.126 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.127 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.127 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.127 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.128 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.128 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.129 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.129 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.129 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.130 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.130 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:28.131 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.131 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.132 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.132 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:28.132 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.133 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.133 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.134 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.134 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.135 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.135 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.135 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.136 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.136 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.137 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.137 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.137 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.138 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.138 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.139 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.139 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.140 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.140 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.140 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.141 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.141 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:03:25Z" 2022-09-13T19:12:28.142 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.142 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:28.144 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.144 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:30.329+0000 7f8c431b7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:30.330+0000 7f8c431b7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:28.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:03:30Z" 2022-09-13T19:12:28.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.178 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.178 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.179 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.179 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.180 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.180 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.181 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.181 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.182 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.182 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.183 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.183 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.183 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.184 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.184 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.185 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.185 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.186 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.187 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.187 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.187 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.188 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.188 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.189 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.189 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.189 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.190 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.190 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.191 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.191 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.192 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.193 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.193 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.193 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.194 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.194 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.195 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:28.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.196 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.196 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.197 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:28.197 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.197 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.198 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.198 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.199 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.199 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.200 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.200 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.201 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.201 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.202 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.202 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.203 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.203 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.203 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.204 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.204 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.205 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.205 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.205 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.206 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.206 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:03:30Z" 2022-09-13T19:12:28.207 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.207 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.070128_219.1 2022-09-13T19:12:28.209 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.209 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:35.461+0000 7f6dbb012600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:35.462+0000 7f6dbb012600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:28.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:28.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:03:35Z" 2022-09-13T19:12:28.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.241 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.242 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.242 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.243 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.243 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.244 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.244 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.245 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.245 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.246 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.246 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.247 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.247 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.247 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.248 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.250 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.250 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.250 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.251 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.251 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.252 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.252 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.253 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.253 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.254 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.254 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.254 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.255 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.255 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.256 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.256 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.257 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.257 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.257 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.258 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.258 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:28.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.259 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.259 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.260 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:28.260 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.261 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.261 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.262 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.262 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.263 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.263 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.263 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.264 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.264 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.265 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.265 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.266 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.266 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.267 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.267 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.268 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.268 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.268 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.269 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.269 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:03:35Z" 2022-09-13T19:12:28.270 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.270 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.911619_214.1 2022-09-13T19:12:28.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:40.586+0000 7ff9c5576600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:40.587+0000 7ff9c5576600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.134902_225.1", 2022-09-13T19:12:28.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.134902Z", 2022-09-13T19:12:28.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:03:40Z" 2022-09-13T19:12:28.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.303 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.303 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.304 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.304 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.305 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.305 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.305 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.306 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.306 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.307 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.307 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.308 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.308 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.308 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.309 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.309 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.310 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.310 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.311 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.311 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.311 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.312 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.312 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.313 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.313 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.314 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.314 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.315 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.315 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.315 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.316 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.316 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.317 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.317 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.317 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.318 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.318 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.134902_225.1", 2022-09-13T19:12:28.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.319 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.320 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.320 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.134902Z", 2022-09-13T19:12:28.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.321 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.321 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.322 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.322 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.322 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.323 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.323 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.324 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.324 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.324 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.325 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.325 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.326 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.326 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.326 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.327 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.327 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.328 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.328 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.328 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.329 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:03:40Z" 2022-09-13T19:12:28.329 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.330 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.134902_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.134902Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.134902_225.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.134902_225.1 2022-09-13T19:12:28.332 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.332 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:45.736+0000 7f16223cc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:45.737+0000 7f16223cc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.186446_230.1", 2022-09-13T19:12:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.186446Z", 2022-09-13T19:12:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:03:45Z" 2022-09-13T19:12:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.363 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.364 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.364 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.364 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.365 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.365 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.366 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.366 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.367 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.367 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.367 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.368 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.368 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.369 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.369 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.370 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.370 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.370 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.371 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.371 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.372 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.372 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.372 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.373 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.373 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.374 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.374 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.375 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.375 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.375 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.376 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.376 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.377 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.377 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.377 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.378 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.378 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.186446_230.1", 2022-09-13T19:12:28.379 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.379 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.380 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.380 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.186446Z", 2022-09-13T19:12:28.380 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.381 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.381 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.381 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.382 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.382 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.383 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.383 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.384 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.384 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.385 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.385 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.386 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.386 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.386 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.387 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.387 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.388 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.388 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.388 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.389 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:03:45Z" 2022-09-13T19:12:28.389 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.390 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.186446_230.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.186446Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.186446_230.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.186446_230.1 2022-09-13T19:12:28.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:50.871+0000 7fc13963a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:50.873+0000 7fc13963a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.072712_220.1", 2022-09-13T19:12:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.072712Z", 2022-09-13T19:12:28.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:03:50Z" 2022-09-13T19:12:28.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.423 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.423 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.423 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.424 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.424 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.425 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.425 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.425 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.426 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.426 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.427 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.427 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.428 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.428 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.429 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.429 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.429 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.430 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.430 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.431 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.431 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.432 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.432 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.433 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.433 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.433 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.434 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.434 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.435 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.435 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.436 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.436 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.436 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.437 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.437 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.438 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.438 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.072712_220.1", 2022-09-13T19:12:28.439 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.439 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.440 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.440 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.072712Z", 2022-09-13T19:12:28.441 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.441 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.441 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.442 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.442 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.443 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.443 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.443 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.444 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.444 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.445 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.445 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.446 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.446 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.447 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.447 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.448 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.448 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.449 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.449 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.450 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:03:50Z" 2022-09-13T19:12:28.450 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.450 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.072712_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.072712Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.072712_220.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.072712_220.1 2022-09-13T19:12:28.453 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.453 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:56.083+0000 7f87118ad600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:03:56.084+0000 7f87118ad600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:28.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:28.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:03:56Z" 2022-09-13T19:12:28.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.484 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.485 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.485 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.485 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.486 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.486 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.487 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.487 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.488 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.488 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.489 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.489 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.490 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.490 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.490 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.491 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.491 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.492 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.492 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.493 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.493 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.493 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.494 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.494 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.495 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.495 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.496 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.496 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.497 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.497 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.498 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.498 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.499 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.499 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.499 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.500 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.500 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:28.501 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.501 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.501 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.502 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:28.502 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.503 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.503 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.504 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.504 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.505 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.505 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.506 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.506 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.507 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.508 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.508 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.509 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.509 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.510 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.511 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.511 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.512 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.513 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.513 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.514 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.515 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:03:56Z" 2022-09-13T19:12:28.515 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.516 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.911619_214.1 2022-09-13T19:12:28.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.519 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:01.229+0000 7f6194770600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:01.230+0000 7f6194770600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.105395_221.1", 2022-09-13T19:12:28.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T19:12:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:04:01Z" 2022-09-13T19:12:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.550 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.551 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.551 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.552 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.552 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.552 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.553 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.553 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.554 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.554 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.554 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.555 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.555 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.556 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.556 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.557 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.557 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.557 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.558 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.558 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.559 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.559 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.559 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.560 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.560 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.561 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.561 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.561 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.562 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.562 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.563 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.563 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.563 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.564 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.564 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.565 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.565 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.105395_221.1", 2022-09-13T19:12:28.566 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.566 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.566 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.567 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T19:12:28.567 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.568 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.569 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.569 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.570 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.570 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.570 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.571 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.572 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.572 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.573 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.573 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.573 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.574 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.574 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.575 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.575 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.575 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.576 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.576 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.577 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:04:01Z" 2022-09-13T19:12:28.577 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.578 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.105395_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.105395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.105395_221.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.105395_221.1 2022-09-13T19:12:28.580 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:06.364+0000 7f840dce5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:06.365+0000 7f840dce5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:28.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:04:06Z" 2022-09-13T19:12:28.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.611 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.612 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.612 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.612 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.613 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.613 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.614 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.614 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.615 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.615 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.615 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.616 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.616 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.617 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.617 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.618 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.618 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.618 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.619 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.619 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.620 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.620 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.620 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.621 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.621 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.622 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.623 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.623 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.623 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.624 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.624 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.625 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.625 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.625 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.626 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.626 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.627 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:28.627 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.628 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.628 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.628 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:28.629 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.629 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.630 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.630 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.631 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.631 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.631 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.632 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.633 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.633 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.634 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.634 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.635 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.635 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.636 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.636 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.636 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.637 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.637 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.638 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:04:06Z" 2022-09-13T19:12:28.638 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.639 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:28.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:11.487+0000 7fb2b6e68600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:11.488+0000 7fb2b6e68600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:04:11Z" 2022-09-13T19:12:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.671 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.671 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.672 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.672 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.672 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.673 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.673 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.674 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.674 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.674 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.675 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.675 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.676 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.676 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.676 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.677 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.677 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.678 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.678 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.678 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.679 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.679 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.680 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.680 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.681 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.681 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.681 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.682 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.682 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.683 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.683 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.683 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.684 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.684 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.685 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.685 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.685 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:28.686 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.686 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.687 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.687 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:28.687 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.688 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.688 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.689 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.689 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.689 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.690 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.690 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.691 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.691 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.692 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.692 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.692 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.693 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.693 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.694 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.694 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.694 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.695 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.695 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.696 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.696 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:04:11Z" 2022-09-13T19:12:28.696 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.697 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:28.699 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.699 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:16.637+0000 7f262bb5c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:16.638+0000 7f262bb5c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.134902_225.1", 2022-09-13T19:12:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.134902Z", 2022-09-13T19:12:28.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:04:16Z" 2022-09-13T19:12:28.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.729 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.730 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.730 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.731 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.731 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.732 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.732 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.732 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.733 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.733 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.734 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.734 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.735 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.735 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.735 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.736 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.736 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.737 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.737 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.738 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.738 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.738 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.739 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.739 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.740 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.741 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.741 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.742 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.742 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.743 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.743 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.744 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.744 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.744 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.745 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.134902_225.1", 2022-09-13T19:12:28.745 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.746 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.746 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.747 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.134902Z", 2022-09-13T19:12:28.747 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.747 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.748 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.748 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.749 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.749 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.749 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.750 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.751 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.752 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.752 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.753 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.753 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.754 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.754 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.754 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.755 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.755 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.756 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.756 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:04:16Z" 2022-09-13T19:12:28.756 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.757 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.134902_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.134902Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.134902_225.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.134902_225.1 2022-09-13T19:12:28.759 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.759 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:21.817+0000 7f9b0fd0f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:21.817+0000 7f9b0fd0f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:28.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:28.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:04:21Z" 2022-09-13T19:12:28.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.790 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.791 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.791 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.792 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.792 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.792 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.793 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.793 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.794 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.794 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.794 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.795 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.795 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.796 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.796 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.796 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.797 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.797 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.798 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.798 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.799 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.799 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.799 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.800 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.800 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.801 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.801 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.802 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.802 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.803 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.803 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.804 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.804 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.805 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.805 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.805 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.806 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:28.806 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.807 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.807 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.808 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:28.808 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.809 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.809 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.809 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.810 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.810 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.811 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.811 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.812 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.812 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.813 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.813 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.814 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.814 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.814 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.815 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.815 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.816 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.816 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.816 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.817 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.817 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:04:21Z" 2022-09-13T19:12:28.818 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.818 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.911619_214.1 2022-09-13T19:12:28.820 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:26.961+0000 7f5684bf3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:26.962+0000 7f5684bf3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:28.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:04:26Z" 2022-09-13T19:12:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.853 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.854 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.854 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.854 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.855 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.856 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.856 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.856 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.857 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.857 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.858 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.858 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.872 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.872 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.873 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.874 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.874 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.875 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.875 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.875 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.876 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.876 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.877 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.877 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.878 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.878 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.879 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.879 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.879 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.880 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.880 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.881 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.881 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.881 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.882 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:28.882 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.883 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.883 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.884 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:28.884 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.884 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.885 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.885 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.886 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.886 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.886 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.887 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.887 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.888 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.888 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.888 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.889 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.889 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.890 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.890 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.891 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.891 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.892 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.892 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.893 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.894 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:04:26Z" 2022-09-13T19:12:28.894 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.895 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:28.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:32.110+0000 7f4a538ad600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:32.111+0000 7f4a538ad600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:28.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:04:32Z" 2022-09-13T19:12:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.930 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.930 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.931 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.931 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.932 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.932 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.932 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.933 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.933 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.934 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.934 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.935 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.935 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.935 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.936 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.936 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.937 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.937 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.938 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.938 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.939 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:28.939 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.940 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.940 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:28.940 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.941 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.941 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.942 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:28.942 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.943 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.943 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.943 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.944 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:28.944 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.945 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.945 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:28.945 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.946 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.946 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.947 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:28.947 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.948 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.948 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.948 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.949 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:28.949 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.950 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.950 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.951 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.951 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:28.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.952 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.952 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.953 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.953 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.953 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:28.954 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.954 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:28.955 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:28.955 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:28.956 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.956 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:04:32Z" 2022-09-13T19:12:28.956 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:28.957 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:28.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:28.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:28.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.070128_219.1 2022-09-13T19:12:28.959 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.959 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:28.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:28.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:28.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:28.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:37.247+0000 7fe506ca2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:37.249+0000 7fe506ca2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:28.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:28.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:28.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:28.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:28.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:28.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:28.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:28.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:28.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:28.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:28.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:28.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:28.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:28.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:28.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:28.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:28.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:28.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:28.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:28.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:28.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:28.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:28.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:28.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:28.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:28.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:28.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:28.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:28.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:04:37Z" 2022-09-13T19:12:28.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:28.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:28.990 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:28.990 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:28.991 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:28.991 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:28.992 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:28.992 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.993 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.993 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:28.994 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.994 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:28.995 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:28.995 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:28.996 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:28.996 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:28.996 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:28.997 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:28.997 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:28.998 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:28.998 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:28.998 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:28.999 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:28.999 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.000 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.000 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.001 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.001 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.001 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.002 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.002 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.003 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.003 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.004 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.004 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.005 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.005 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.006 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.006 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:29.006 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.007 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.007 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.008 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:29.008 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.009 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.009 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.009 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.010 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.010 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.011 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.011 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.012 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.012 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.012 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.013 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.014 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.014 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.015 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.015 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.016 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.016 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.017 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.017 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.017 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:04:37Z" 2022-09-13T19:12:29.018 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.018 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:29.021 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.021 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:42.432+0000 7f902a8ab600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:42.433+0000 7f902a8ab600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.351164_238.1", 2022-09-13T19:12:29.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.351164Z", 2022-09-13T19:12:29.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:04:42Z" 2022-09-13T19:12:29.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.054 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.054 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.055 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.055 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.056 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.056 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.056 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.057 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.057 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.058 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.058 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.059 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.059 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.059 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.060 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.060 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.061 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.061 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.061 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.062 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.062 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.063 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.063 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.064 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.065 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.065 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.066 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.066 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.066 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.067 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.067 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.068 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.068 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.068 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.069 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.351164_238.1", 2022-09-13T19:12:29.069 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.070 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.070 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.071 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.351164Z", 2022-09-13T19:12:29.071 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.071 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.072 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.072 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.073 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.073 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.073 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.075 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.075 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.076 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.077 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.077 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.078 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.078 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.078 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.079 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.079 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.080 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:04:42Z" 2022-09-13T19:12:29.080 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.080 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.081 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.351164_238.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.351164Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.081 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.351164_238.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.351164_238.1 2022-09-13T19:12:29.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:47.622+0000 7f1a878a1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:47.623+0000 7f1a878a1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:29.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:29.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:04:47Z" 2022-09-13T19:12:29.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.114 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.115 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.115 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.116 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.116 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.117 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.117 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.118 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.118 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.119 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.119 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.119 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.120 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.120 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.121 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.122 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.122 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.123 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.123 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.124 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.124 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.124 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.125 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.125 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.126 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.126 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.127 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.127 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.127 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.128 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.128 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.129 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.129 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.130 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.130 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.130 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:29.131 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.131 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.132 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.132 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:29.132 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.133 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.133 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.134 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.134 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.135 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.135 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.135 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.136 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.136 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.137 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.137 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.137 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.138 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.138 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.139 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.139 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.140 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.140 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.140 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.141 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.141 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:04:47Z" 2022-09-13T19:12:29.142 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.142 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.990623_217.1 2022-09-13T19:12:29.144 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.145 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:52.764+0000 7f528a634600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:52.765+0000 7f528a634600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:29.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:29.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:04:52Z" 2022-09-13T19:12:29.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.175 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.176 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.176 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.176 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.177 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.177 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.178 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.178 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.178 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.179 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.179 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.180 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.180 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.181 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.181 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.181 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.182 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.183 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.183 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.183 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.184 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.184 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.185 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.185 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.185 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.186 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.186 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.187 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.187 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.188 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.188 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.188 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.189 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.189 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.190 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.190 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:29.191 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.191 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.191 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.192 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:29.192 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.193 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.193 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.193 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.194 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.194 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.195 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.195 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.196 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.196 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.197 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.197 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.198 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.198 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.198 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.199 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.199 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.200 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.200 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.200 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.201 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:04:52Z" 2022-09-13T19:12:29.201 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.202 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.070128_219.1 2022-09-13T19:12:29.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:57.895+0000 7fb80d415600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:04:57.896+0000 7fb80d415600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.146590_228.1", 2022-09-13T19:12:29.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.146590Z", 2022-09-13T19:12:29.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:04:57Z" 2022-09-13T19:12:29.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.256 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.256 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.257 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.257 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.257 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.258 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.259 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.259 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.260 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.260 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.260 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.261 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.261 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.262 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.262 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.263 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.264 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.264 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.264 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.265 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.266 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.266 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.267 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.267 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.267 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.268 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.268 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.269 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.269 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.270 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.270 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.271 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.271 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.272 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.272 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.272 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.273 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.146590_228.1", 2022-09-13T19:12:29.273 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.274 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.275 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.275 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.146590Z", 2022-09-13T19:12:29.276 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.276 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.277 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.277 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.278 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.279 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.280 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.280 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.281 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.282 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.282 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.283 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.283 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.284 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.284 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.284 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.285 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.285 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.286 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.287 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.287 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:04:57Z" 2022-09-13T19:12:29.287 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.288 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.146590_228.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.146590Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.146590_228.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.146590_228.1 2022-09-13T19:12:29.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:03.084+0000 7f991facc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:03.086+0000 7f991facc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:29.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:29.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:05:03Z" 2022-09-13T19:12:29.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.328 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.329 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.330 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.331 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.331 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.332 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.333 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.333 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.334 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.335 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.335 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.336 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.337 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.338 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.338 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.339 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.340 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.340 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.341 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.341 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.342 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.342 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.343 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.343 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.344 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.344 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.345 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.345 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.345 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.346 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.346 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.347 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.347 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.347 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.348 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.348 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.349 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:29.349 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.350 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.350 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.350 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:29.351 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.352 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.352 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.353 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.353 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.353 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.354 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.355 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.355 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.356 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.357 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.358 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.358 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.359 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.360 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.360 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.361 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.362 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.363 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.363 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.364 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.364 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:05:03Z" 2022-09-13T19:12:29.365 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.365 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.953175_216.1 2022-09-13T19:12:29.367 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.368 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:08.236+0000 7fe1a2a8d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:08.237+0000 7fe1a2a8d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:29.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:29.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:05:08Z" 2022-09-13T19:12:29.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.410 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.411 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.412 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.412 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.413 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.414 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.414 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.415 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.416 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.416 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.417 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.418 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.418 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.419 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.420 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.420 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.420 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.421 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.421 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.422 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.422 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.422 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.423 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.423 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.424 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.424 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.424 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.425 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.425 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.426 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.426 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.427 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.427 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.427 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.428 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.428 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.428 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:29.429 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.429 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.430 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.430 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:29.431 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.431 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.431 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.432 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.432 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.433 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.433 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.433 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.434 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.434 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.435 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.435 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.435 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.436 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.436 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.437 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.437 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.437 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.438 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.438 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.439 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.439 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:05:08Z" 2022-09-13T19:12:29.439 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.440 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.911619_214.1 2022-09-13T19:12:29.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:13.398+0000 7f02aed78600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:13.399+0000 7f02aed78600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.182451_229.1", 2022-09-13T19:12:29.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T19:12:29.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:05:13Z" 2022-09-13T19:12:29.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.473 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.473 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.473 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.474 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.474 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.475 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.475 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.475 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.476 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.476 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.477 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.477 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.477 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.478 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.478 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.479 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.479 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.479 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.480 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.480 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.481 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.481 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.481 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.482 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.482 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.483 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.483 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.483 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.484 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.484 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.485 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.485 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.485 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.486 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.486 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.487 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.487 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.182451_229.1", 2022-09-13T19:12:29.487 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.488 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.488 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.489 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T19:12:29.489 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.489 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.490 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.490 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.491 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.491 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.492 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.492 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.492 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.493 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.493 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.494 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.494 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.495 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.495 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.496 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.496 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.496 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.497 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.497 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.498 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:05:13Z" 2022-09-13T19:12:29.498 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.498 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.182451_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.182451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.182451_229.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.182451_229.1 2022-09-13T19:12:29.500 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.501 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:18.535+0000 7f9cced3f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:18.536+0000 7f9cced3f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.182451_229.1", 2022-09-13T19:12:29.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T19:12:29.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:05:18Z" 2022-09-13T19:12:29.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.530 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.530 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.530 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.531 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.531 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.532 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.532 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.532 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.533 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.533 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.534 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.534 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.535 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.535 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.536 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.537 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.537 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.537 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.538 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.538 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.539 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.539 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.539 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.540 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.540 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.541 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.541 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.541 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.542 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.542 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.543 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.543 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.543 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.544 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.544 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.545 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.182451_229.1", 2022-09-13T19:12:29.545 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.545 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.546 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.546 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T19:12:29.547 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.547 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.547 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.548 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.548 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.549 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.549 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.549 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.550 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.550 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.551 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.551 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.551 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.552 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.552 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.553 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.553 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.553 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.554 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.554 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.555 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.555 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:05:18Z" 2022-09-13T19:12:29.556 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.556 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.182451_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.182451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.182451_229.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.182451_229.1 2022-09-13T19:12:29.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:23.782+0000 7fc0df94d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:23.783+0000 7fc0df94d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:29.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:29.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:05:23Z" 2022-09-13T19:12:29.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.587 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.588 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.588 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.589 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.589 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.589 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.590 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.590 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.591 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.591 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.591 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.592 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.592 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.593 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.593 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.593 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.594 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.594 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.595 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.595 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.595 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.596 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.596 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.597 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.597 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.597 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.598 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.598 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.599 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.599 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.600 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.600 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.600 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.601 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.601 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.602 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.602 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:29.602 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.603 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.603 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.604 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:29.604 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.604 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.605 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.605 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.606 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.606 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.606 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.607 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.607 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.608 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.608 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.608 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.609 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.609 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.610 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.610 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.610 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.611 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.611 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.612 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.612 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.612 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:05:23Z" 2022-09-13T19:12:29.613 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.613 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:29.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.616 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:28.920+0000 7fe388a17600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:28.921+0000 7fe388a17600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:29.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:29.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:05:28Z" 2022-09-13T19:12:29.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.645 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.646 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.646 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.646 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.647 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.647 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.648 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.648 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.648 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.649 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.649 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.650 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.650 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.650 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.651 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.651 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.652 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.652 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.653 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.653 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.654 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.654 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.654 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.655 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.655 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.656 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.656 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.656 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.657 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.657 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.658 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.658 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.658 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.659 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.659 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.660 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:29.660 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.660 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.661 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.661 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:29.662 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.662 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.662 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.663 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.663 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.664 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.664 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.664 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.665 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.665 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.666 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.666 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.666 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.667 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.667 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.668 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.668 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.668 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.669 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.669 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.670 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.670 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:05:28Z" 2022-09-13T19:12:29.670 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.671 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.990623_217.1 2022-09-13T19:12:29.673 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.674 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:34.074+0000 7f4ce1b85600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:34.075+0000 7f4ce1b85600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:29.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:29.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:05:34Z" 2022-09-13T19:12:29.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.704 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.705 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.705 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.706 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.706 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.706 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.707 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.707 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.708 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.708 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.708 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.709 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.709 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.710 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.710 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.711 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.711 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.712 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.712 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.712 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.713 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.713 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.714 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.714 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.714 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.715 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.715 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.716 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.716 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.716 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.717 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.717 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.718 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.718 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.718 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.719 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:29.719 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.720 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.720 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.720 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:29.721 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.721 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.722 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.722 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.722 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.723 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.723 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.724 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.724 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.724 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.725 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.725 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.726 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.726 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.726 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.727 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.727 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.728 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.728 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.728 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.729 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.729 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:05:34Z" 2022-09-13T19:12:29.730 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.730 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.070128_219.1 2022-09-13T19:12:29.732 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.733 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:39.212+0000 7f967da46600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:39.213+0000 7f967da46600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.296844_235.1", 2022-09-13T19:12:29.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.296844Z", 2022-09-13T19:12:29.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:05:39Z" 2022-09-13T19:12:29.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.762 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.762 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.762 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.763 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.763 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.764 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.764 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.765 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.765 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.765 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.766 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.766 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.767 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.767 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.767 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.768 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.768 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.769 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.769 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.769 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.770 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.770 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.771 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.771 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.771 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.772 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.772 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.773 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.773 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.773 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.774 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.774 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.775 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.775 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.775 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.776 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.776 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.296844_235.1", 2022-09-13T19:12:29.777 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.777 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.777 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.778 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.296844Z", 2022-09-13T19:12:29.778 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.779 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.779 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.779 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.780 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.780 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.781 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.781 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.781 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.782 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.782 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.783 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.783 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.783 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.784 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.784 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.785 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.785 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.785 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.786 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.786 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.787 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:05:39Z" 2022-09-13T19:12:29.787 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.787 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.296844_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.296844Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.296844_235.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.296844_235.1 2022-09-13T19:12:29.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.790 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:44.379+0000 7fe22cdbf600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:44.381+0000 7fe22cdbf600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:29.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:29.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:05:44Z" 2022-09-13T19:12:29.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.819 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.820 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.820 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.820 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.821 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.821 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.822 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.822 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.822 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.823 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.823 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.824 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.824 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.824 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.825 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.825 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.826 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.826 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.826 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.827 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.827 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.828 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.828 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.828 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.829 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.829 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.830 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.830 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.830 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.831 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.831 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.832 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.832 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.832 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.833 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.833 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.834 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:29.834 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.834 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.835 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.835 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:29.836 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.836 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.836 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.837 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.837 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.838 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.838 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.838 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.839 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.839 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.840 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.840 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.840 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.841 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.841 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.842 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.842 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.842 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.843 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.843 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.844 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.844 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:05:44Z" 2022-09-13T19:12:29.844 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.845 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.990623_217.1 2022-09-13T19:12:29.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:49.517+0000 7f5002ed6600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:49.518+0000 7f5002ed6600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:29.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:29.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:05:49Z" 2022-09-13T19:12:29.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.877 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.877 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.878 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.878 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.878 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.879 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.879 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.880 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.880 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.880 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.881 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.881 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.882 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.882 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.882 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.883 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.883 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.884 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.884 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.884 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.885 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.885 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.886 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.886 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.886 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.887 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.887 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.888 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.888 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.888 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.889 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.889 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.890 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.890 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.891 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.891 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.891 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:29.892 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.892 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.893 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.893 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:29.893 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.894 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.894 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.895 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.895 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.895 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.896 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.896 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.897 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.898 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.898 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.899 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.899 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.899 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.900 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.900 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.901 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.901 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.902 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.902 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:05:49Z" 2022-09-13T19:12:29.903 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.903 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T19:12:29.905 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.906 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:54.646+0000 7f0861d2d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:54.647+0000 7f0861d2d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:29.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:29.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:05:54Z" 2022-09-13T19:12:29.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.935 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.936 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.936 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.936 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.937 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.938 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.938 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.939 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.939 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.939 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.940 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.940 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.941 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.941 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.941 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.942 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.942 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.943 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.943 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.943 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.944 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:29.944 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.945 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.945 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:29.945 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.946 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.946 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.947 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:29.947 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.947 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.948 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.948 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.949 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:29.949 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.949 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.950 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:29.950 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.951 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.951 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:29.952 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.952 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.953 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.953 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.953 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:29.954 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.954 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.955 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.955 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.955 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:29.956 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.956 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.957 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.957 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.957 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:29.958 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:29.958 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.959 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:29.959 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:29.959 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:29.960 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.960 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:05:54Z" 2022-09-13T19:12:29.961 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:29.961 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:29.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:29.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:29.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:29.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.964 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:29.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:29.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:29.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:29.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:59.793+0000 7f7319f25600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:05:59.794+0000 7f7319f25600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:29.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:29.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:29.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:29.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:29.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:29.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:29.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:29.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:29.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:29.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:29.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:29.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:29.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:29.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:29.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:29.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:29.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:29.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:29.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:29.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:29.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:29.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:29.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:29.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:29.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:29.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:29.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:29.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:29.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:29.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:05:59Z" 2022-09-13T19:12:29.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:29.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:29.993 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:29.993 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:29.994 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:29.994 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:29.994 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:29.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:29.995 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:29.996 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:29.996 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:29.996 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:29.997 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:29.997 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:29.998 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:29.998 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:29.998 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:29.999 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:29.999 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.000 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.000 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.000 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.001 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.001 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:30.002 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.002 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.003 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:30.003 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.003 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.004 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.004 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:30.005 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.005 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.005 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.006 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.006 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:30.007 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.007 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.007 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.990623_217.1", 2022-09-13T19:12:30.008 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.008 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.009 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.009 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.990623Z", 2022-09-13T19:12:30.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.010 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.010 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.011 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.011 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:30.011 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.012 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.012 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.013 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.013 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.014 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.014 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.015 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.015 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.015 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:30.016 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.016 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:30.017 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:30.017 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:30.018 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.019 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:05:59Z" 2022-09-13T19:12:30.020 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:30.020 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:30.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.990623_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.990623Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:30.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.990623_217.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.990623_217.1 2022-09-13T19:12:30.024 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.024 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:30.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:30.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:30.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:04.950+0000 7f19ddda2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:04.951+0000 7f19ddda2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:30.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:30.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:30.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:30.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:30.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:30.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:30.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:30.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:30.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:30.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:30.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.111513_224.1", 2022-09-13T19:12:30.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.111513Z", 2022-09-13T19:12:30.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:30.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:30.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:30.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:30.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:30.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:06:04Z" 2022-09-13T19:12:30.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:30.081 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:30.082 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:30.082 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:30.083 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:30.084 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.085 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.086 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.086 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:30.087 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.088 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:30.089 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.090 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.090 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.091 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.092 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:30.093 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.094 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.095 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.096 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.097 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.098 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.099 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:30.100 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.100 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.101 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:30.101 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.102 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.103 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.104 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:30.104 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.105 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.106 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.107 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.107 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:30.108 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.109 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.109 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.111513_224.1", 2022-09-13T19:12:30.110 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.110 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.111 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.111 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.111513Z", 2022-09-13T19:12:30.112 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.112 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.112 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.113 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.113 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:30.114 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.114 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.115 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.116 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.117 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.118 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.118 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.119 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.120 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.120 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.121 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:30.122 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.122 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:30.123 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:30.123 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:30.124 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.124 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:06:04Z" 2022-09-13T19:12:30.125 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:30.125 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:30.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.111513_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.111513Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:30.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.111513_224.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.111513_224.1 2022-09-13T19:12:30.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.129 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:30.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:30.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:30.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:10.149+0000 7f74c4bf2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:10.150+0000 7f74c4bf2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:30.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:30.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:30.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:30.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:30.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:30.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:30.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:30.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:30.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:30.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:30.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:30.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:30.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:30.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:30.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:30.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:30.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:30.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:06:10Z" 2022-09-13T19:12:30.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:30.172 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:30.173 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:30.173 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:30.174 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:30.174 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.175 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.175 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.176 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:30.176 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.176 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:30.177 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.177 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.178 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.178 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.179 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:30.179 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.180 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.180 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.180 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.181 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.181 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.182 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:30.182 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.183 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.183 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:30.184 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.184 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.185 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.185 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:30.185 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.186 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.186 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.187 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.187 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:30.188 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.188 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.188 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.949151_215.1", 2022-09-13T19:12:30.189 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.190 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.190 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.191 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.949151Z", 2022-09-13T19:12:30.191 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.191 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.192 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.192 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.193 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:30.193 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.194 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.194 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.195 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.196 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.196 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.196 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.197 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.198 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.199 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:30.199 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.200 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:30.200 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:30.201 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:30.201 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.202 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:06:10Z" 2022-09-13T19:12:30.202 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:30.203 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:30.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.949151_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.949151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:30.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.949151_215.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.949151_215.1 2022-09-13T19:12:30.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:30.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:30.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:30.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:15.286+0000 7f4f9c36a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:15.287+0000 7f4f9c36a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:30.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:30.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:30.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:30.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:30.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:30.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:30.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:30.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:30.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:30.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:30.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.109037_223.1", 2022-09-13T19:12:30.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.109037Z", 2022-09-13T19:12:30.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:30.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:30.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:30.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:30.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:30.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:06:15Z" 2022-09-13T19:12:30.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:30.249 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:30.250 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:30.251 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:30.251 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:30.252 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.252 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.253 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.254 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:30.255 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.256 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:30.256 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.257 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.257 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.257 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.258 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:30.258 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.259 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.259 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.260 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.260 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.261 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.261 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:30.261 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.262 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.262 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:30.263 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.264 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.264 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.265 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:30.266 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.266 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.267 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.267 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.268 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:30.269 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.269 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.270 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.109037_223.1", 2022-09-13T19:12:30.270 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.271 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.271 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.272 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.109037Z", 2022-09-13T19:12:30.272 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.272 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.273 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.273 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.274 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:30.275 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.275 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.276 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.276 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.277 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.277 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.278 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.278 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.278 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.279 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.280 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:30.280 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.281 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:30.281 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:30.282 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:30.282 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.283 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:06:15Z" 2022-09-13T19:12:30.283 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:30.284 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:30.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.109037_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.109037Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:30.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.109037_223.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.109037_223.1 2022-09-13T19:12:30.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:30.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:30.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:30.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:20.419+0000 7f824e747600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:20.420+0000 7f824e747600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:30.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:30.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:30.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:30.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:30.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:30.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:30.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:30.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:30.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:30.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:30.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.296844_235.1", 2022-09-13T19:12:30.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.296844Z", 2022-09-13T19:12:30.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:30.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:30.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:30.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:30.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:30.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:06:20Z" 2022-09-13T19:12:30.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:30.335 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:30.335 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:30.336 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:30.336 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:30.337 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.337 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.338 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.338 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:30.339 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.339 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:30.339 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.340 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.340 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.341 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.341 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:30.342 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.342 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.343 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.344 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.345 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.346 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.346 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:30.347 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.347 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.348 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:30.349 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.349 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.350 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.351 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:30.352 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.353 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.354 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.355 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.355 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:30.356 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.357 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.358 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.296844_235.1", 2022-09-13T19:12:30.359 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.360 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.361 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.361 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.296844Z", 2022-09-13T19:12:30.362 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.362 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.363 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.363 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.364 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:30.364 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.365 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.365 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.367 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.368 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.370 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.371 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.371 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.372 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:30.373 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.374 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:30.374 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:30.375 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:30.375 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.375 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:06:20Z" 2022-09-13T19:12:30.376 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:30.376 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:30.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.296844_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.296844Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:30.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.296844_235.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.296844_235.1 2022-09-13T19:12:30.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:30.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:30.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:30.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:25.552+0000 7f813b602600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:25.553+0000 7f813b602600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:30.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:30.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:30.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:30.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:30.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:30.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:30.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:30.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:30.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:30.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:30.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:30.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:30.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:30.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:30.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:30.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:30.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:30.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:06:25Z" 2022-09-13T19:12:30.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:30.416 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:30.417 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:30.417 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:30.418 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:30.418 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.418 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.419 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.419 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:30.420 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.420 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:30.421 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.422 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.423 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.424 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.425 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:30.425 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.426 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.427 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.427 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.428 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.429 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.430 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:30.431 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.432 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.433 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:30.434 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.435 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.436 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.437 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:30.438 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.438 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.439 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.441 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.442 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:30.443 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.444 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.445 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:30.446 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.447 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.447 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.448 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:30.449 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.450 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.451 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.452 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.453 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:30.453 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.454 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.455 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.456 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.457 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.458 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.459 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.460 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.460 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.462 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.463 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:30.464 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.465 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:30.466 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:30.467 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:30.468 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.469 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:06:25Z" 2022-09-13T19:12:30.470 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:30.470 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:30.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:30.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.953175_216.1 2022-09-13T19:12:30.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.476 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:30.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:30.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:30.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:30.711+0000 7f370604e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:30.712+0000 7f370604e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:30.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:30.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:30.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:30.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:30.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:30.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:30.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:30.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:30.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:30.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:30.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.105395_221.1", 2022-09-13T19:12:30.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T19:12:30.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:30.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:30.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:30.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:30.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:30.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:06:30Z" 2022-09-13T19:12:30.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.526 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:30.527 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:30.527 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:30.528 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:30.529 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:30.530 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.531 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.531 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.532 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:30.532 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.532 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:30.533 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.534 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.534 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.535 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:30.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.536 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.537 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.537 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.538 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.538 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.539 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:30.539 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.540 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.540 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:30.541 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.541 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.542 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.542 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:30.543 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.543 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.544 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.544 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.545 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:30.545 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.546 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.546 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.105395_221.1", 2022-09-13T19:12:30.547 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.548 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.548 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.549 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.105395Z", 2022-09-13T19:12:30.549 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.550 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.550 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.551 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.551 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:30.552 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.552 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.552 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.553 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.553 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.554 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.554 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.555 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.556 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.556 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:30.556 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.557 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:30.557 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:30.558 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:30.558 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.559 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:06:30Z" 2022-09-13T19:12:30.559 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:30.560 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:30.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.105395_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.105395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:30.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.105395_221.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.105395_221.1 2022-09-13T19:12:30.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.563 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:30.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:30.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:30.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:35.865+0000 7ff51a19d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:35.866+0000 7ff51a19d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:30.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:30.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:30.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:30.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:30.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:30.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:30.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:30.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:30.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:30.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:30.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:30.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:30.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:30.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:30.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:30.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:30.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:30.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:06:35Z" 2022-09-13T19:12:30.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:30.596 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:30.596 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:30.597 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:30.597 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:30.597 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.598 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.599 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.599 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:30.600 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.600 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:30.600 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.601 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.601 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.602 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.602 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:30.603 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.603 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.603 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.604 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.604 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.605 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.605 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:30.605 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.606 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.606 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:30.607 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.607 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.608 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.608 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:30.608 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.609 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.609 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.610 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.610 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:30.611 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.612 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.070128_219.1", 2022-09-13T19:12:30.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.613 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.613 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.614 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.070128Z", 2022-09-13T19:12:30.614 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.614 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.615 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.615 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.616 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:30.616 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.617 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.617 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.617 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.618 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.618 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.619 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.619 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.620 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.620 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.621 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:30.621 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.622 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:30.622 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:30.623 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:30.623 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.624 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:06:35Z" 2022-09-13T19:12:30.624 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:30.625 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:30.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.070128_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.070128Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:30.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.070128_219.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.070128_219.1 2022-09-13T19:12:30.627 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:30.628 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:30.628 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:30.629 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_zonegroup_remove 2022-09-13T19:12:30.630 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:30.632 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:30.633 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:30.633 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:30.634 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1056, in test_zonegroup_remove 2022-09-13T19:12:30.634 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2022-09-13T19:12:30.635 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:30.635 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2022-09-13T19:12:30.636 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 196, in zone_meta_checkpoint 2022-09-13T19:12:30.636 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2022-09-13T19:12:30.636 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2022-09-13T19:12:30.637 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:30.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:30.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:30.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:30.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:30.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:30.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:30.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:30.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:30.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:30.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:30.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:30.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:30.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:30.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663056091.338062_357.1", 2022-09-13T19:12:30.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T08:01:31.338062Z" 2022-09-13T19:12:30.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:30.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:30.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:30.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:30.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:30.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:30.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:30.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:30.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:30.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:41.128+0000 7f013ca3d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:41.129+0000 7f013ca3d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:30.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:30.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:30.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:30.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:30.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:30.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:30.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:30.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:30.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:30.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:30.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:30.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:30.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:30.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:30.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:30.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:30.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:30.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:06:41Z" 2022-09-13T19:12:30.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:30.683 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:30.684 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:30.684 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:30.684 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:30.685 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.685 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.686 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.686 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:30.687 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.687 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:30.688 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.688 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.689 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.689 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.689 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:30.690 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.690 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.691 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.691 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.692 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.692 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.692 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:30.693 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.693 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.694 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:30.694 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.695 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.695 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.695 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:30.696 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.696 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.697 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.698 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.698 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:30.698 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.699 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.699 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:30.700 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.700 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.701 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.701 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:30.701 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.702 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.702 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.703 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.703 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:30.704 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.704 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.705 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.705 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.705 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.706 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.706 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.707 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.707 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.708 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.708 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:30.709 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.709 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:30.709 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:30.710 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:30.710 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.711 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:06:41Z" 2022-09-13T19:12:30.711 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:30.712 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:30.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:30.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.953175_216.1 2022-09-13T19:12:30.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:30.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:30.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:30.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:46.273+0000 7f5f20e5e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:46.274+0000 7f5f20e5e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:30.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:30.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:30.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:30.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:30.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:30.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:30.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:30.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:30.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:30.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:30.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:30.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:30.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:30.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:30.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:30.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:30.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:30.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:06:46Z" 2022-09-13T19:12:30.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:30.747 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:30.748 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:30.748 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:30.749 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:30.749 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.750 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.750 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.751 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:30.752 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.752 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:30.753 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.753 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.754 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.754 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.755 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:30.755 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.756 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.756 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.757 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.757 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.757 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.758 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:30.759 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.759 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.760 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:30.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.762 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:30.762 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.763 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.764 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.765 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.765 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:30.766 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.767 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.768 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.953175_216.1", 2022-09-13T19:12:30.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.770 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.772 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.953175Z", 2022-09-13T19:12:30.773 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.774 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.775 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.776 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.777 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:30.778 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.779 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.780 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.781 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.781 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.782 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.782 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.783 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.783 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.784 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.784 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:30.785 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.785 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:30.786 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:30.786 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:30.787 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.788 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:06:46Z" 2022-09-13T19:12:30.788 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:30.789 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:30.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.953175_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.953175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:30.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.953175_216.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.953175_216.1 2022-09-13T19:12:30.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:30.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:30.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:30.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:51.693+0000 7fbc4ae08600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:51.694+0000 7fbc4ae08600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:30.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:30.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:30.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:30.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:30.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:30.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:30.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:30.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:30.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:30.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:30.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.134902_225.1", 2022-09-13T19:12:30.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.134902Z", 2022-09-13T19:12:30.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:30.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:30.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:30.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:30.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:30.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:06:51Z" 2022-09-13T19:12:30.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:30.826 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:30.826 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:30.827 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:30.828 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:30.829 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.829 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.830 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.831 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:30.831 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.832 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:30.832 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.833 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.833 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.834 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.834 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:30.835 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.835 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.836 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.837 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.838 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.839 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.840 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:30.840 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.841 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.842 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:30.843 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.844 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.845 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:30.846 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.847 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.848 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.848 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.849 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:30.849 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.849 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.850 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.134902_225.1", 2022-09-13T19:12:30.850 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.851 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.851 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.852 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.134902Z", 2022-09-13T19:12:30.852 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.853 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.853 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.854 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.855 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:30.855 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.856 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.857 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.858 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.859 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.859 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.860 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.861 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.862 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.863 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.863 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:30.864 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.864 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:30.865 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:30.865 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:30.866 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.867 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:06:51Z" 2022-09-13T19:12:30.867 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:30.868 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:30.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.134902_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.134902Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:30.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.134902_225.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.134902_225.1 2022-09-13T19:12:30.873 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.874 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:30.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:30.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:30.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:56.852+0000 7f367befa600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:06:56.853+0000 7f367befa600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:30.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:30.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:30.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:30.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:30.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:30.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:30.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:30.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:30.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:30.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:30.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.111513_224.1", 2022-09-13T19:12:30.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.111513Z", 2022-09-13T19:12:30.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:30.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:30.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:30.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:30.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:30.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:30.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:30.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:30.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:30.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:30.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:30.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:06:56Z" 2022-09-13T19:12:30.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:30.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:30.946 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:30.947 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:30.947 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:30.948 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:30.949 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.950 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.951 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.952 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:30.953 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.955 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:30.956 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.957 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.958 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:30.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.960 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:30.961 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.962 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:30.962 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.963 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.964 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.964 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.965 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:30.965 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.966 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.967 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:30.968 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.969 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.970 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.970 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:30.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.972 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.973 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.974 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.975 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:30.975 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.976 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.976 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.111513_224.1", 2022-09-13T19:12:30.977 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.977 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.978 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.978 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.111513Z", 2022-09-13T19:12:30.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.980 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.980 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.981 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:30.981 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:30.982 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:30.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:30.983 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:30.983 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:30.984 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:30.984 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:30.985 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:30.985 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:30.986 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.986 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:30.987 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:30.987 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.988 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:30.988 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:30.988 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:30.989 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:30.989 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:06:56Z" 2022-09-13T19:12:30.990 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:30.990 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:30.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.111513_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.111513Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:30.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.111513_224.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:30.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.111513_224.1 2022-09-13T19:12:30.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:30.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:30.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:30.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:30.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:01.998+0000 7fc25a5d0600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:01.999+0000 7fc25a5d0600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:30.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:30.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:30.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:30.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:30.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:30.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:30.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:30.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:31.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:31.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:31.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:31.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:31.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:31.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:31.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.182451_229.1", 2022-09-13T19:12:31.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T19:12:31.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:31.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:31.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:31.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:31.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:31.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:07:02Z" 2022-09-13T19:12:31.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:31.027 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:31.028 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:31.028 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:31.028 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:31.029 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.029 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.030 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.030 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:31.031 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.032 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:31.032 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.033 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.035 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.036 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.037 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:31.038 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.038 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.039 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.040 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.041 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.042 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.043 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:31.043 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.044 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.046 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:31.046 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.047 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.048 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.049 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:31.050 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.050 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.051 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.052 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.052 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:31.053 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.054 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.055 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.182451_229.1", 2022-09-13T19:12:31.056 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.056 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.057 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.057 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T19:12:31.058 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.058 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.059 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.059 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.060 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:31.060 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.061 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.062 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.063 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.066 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.067 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.067 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.068 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:31.069 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.069 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:31.070 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:31.071 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:31.071 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.072 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:07:02Z" 2022-09-13T19:12:31.073 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:31.074 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:31.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.182451_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.182451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:31.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.182451_229.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.182451_229.1 2022-09-13T19:12:31.078 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.079 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:31.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:31.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:31.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:07.144+0000 7f0c29226600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:07.145+0000 7f0c29226600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:31.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:31.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:31.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:31.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:31.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:31.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:31.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:31.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:31.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:31.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:31.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.351164_238.1", 2022-09-13T19:12:31.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.351164Z", 2022-09-13T19:12:31.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:31.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:31.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:31.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:31.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:31.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:07:07Z" 2022-09-13T19:12:31.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:31.136 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:31.137 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:31.138 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:31.139 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:31.139 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.140 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.141 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.142 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:31.142 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.143 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:31.143 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.144 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.144 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.145 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.145 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:31.145 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.146 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.146 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.147 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.147 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.148 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.148 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:31.149 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.149 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.150 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:31.150 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.151 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.151 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.151 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:31.152 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.152 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.153 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.153 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.154 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:31.154 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.155 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.155 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.351164_238.1", 2022-09-13T19:12:31.155 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.156 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.156 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.157 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.351164Z", 2022-09-13T19:12:31.157 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.157 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.158 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.159 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.159 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:31.159 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.160 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.160 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.161 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.161 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.162 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.162 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.163 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.163 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.164 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.164 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:31.165 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.165 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:31.165 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:31.166 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:31.166 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.167 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:07:07Z" 2022-09-13T19:12:31.167 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:31.168 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:31.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.351164_238.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.351164Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:31.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.351164_238.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.351164_238.1 2022-09-13T19:12:31.170 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.171 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:31.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:31.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:31.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:12.281+0000 7fb436a33600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:12.282+0000 7fb436a33600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:31.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:31.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:31.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:31.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:31.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:31.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:31.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:31.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:31.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:31.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:31.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:31.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:31.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:31.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:31.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:31.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:31.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:31.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:07:12Z" 2022-09-13T19:12:31.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:31.203 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:31.203 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:31.203 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:31.204 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:31.204 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.205 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.206 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.206 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:31.207 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.208 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:31.209 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.209 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.210 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.211 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.212 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:31.213 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.213 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.215 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.216 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.216 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.217 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:31.217 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.218 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.218 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:31.219 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.219 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.220 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.221 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:31.222 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.223 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.224 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.225 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.225 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:31.226 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.226 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.227 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.911619_214.1", 2022-09-13T19:12:31.227 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.227 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.228 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.228 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.911619Z", 2022-09-13T19:12:31.229 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.229 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.230 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.230 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.231 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:31.231 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.232 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.232 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.233 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.233 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.233 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.234 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.234 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.235 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.235 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.236 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:31.237 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.237 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:31.238 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:31.238 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:31.239 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.239 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:07:12Z" 2022-09-13T19:12:31.240 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:31.240 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:31.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.911619_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.911619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:31.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.911619_214.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.911619_214.1 2022-09-13T19:12:31.243 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.243 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:31.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:31.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:31.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:17.414+0000 7efe9bcdd600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:17.415+0000 7efe9bcdd600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:31.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:31.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:31.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:31.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:31.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:31.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:31.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:31.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:31.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:31.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:31.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.107994_222.1", 2022-09-13T19:12:31.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.107994Z", 2022-09-13T19:12:31.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:31.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:31.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:31.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:31.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:31.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:07:17Z" 2022-09-13T19:12:31.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:31.279 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:31.280 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:31.281 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:31.282 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:31.282 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.284 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.285 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.285 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:31.286 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.288 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:31.288 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.289 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.290 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.291 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.291 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:31.292 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.293 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.294 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.294 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.295 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.295 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.296 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:31.297 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.297 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.298 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:31.298 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.298 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.299 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.299 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:31.300 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.301 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.302 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.302 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.303 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:31.304 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.304 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.305 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.107994_222.1", 2022-09-13T19:12:31.306 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.307 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.307 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.308 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.107994Z", 2022-09-13T19:12:31.309 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.310 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.310 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.311 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.312 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:31.313 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.313 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.314 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.315 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.315 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.316 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.318 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.319 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.319 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.320 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.321 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:31.322 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.323 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:31.324 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:31.325 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:31.325 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.326 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:07:17Z" 2022-09-13T19:12:31.327 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:31.327 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:31.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.107994_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.107994Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:31.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.107994_222.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.107994_222.1 2022-09-13T19:12:31.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.332 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:31.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:31.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:31.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:22.568+0000 7f0122568600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:22.570+0000 7f0122568600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:31.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:31.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:31.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:31.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:31.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:31.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:31.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:31.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:31.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:31.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:31.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.182451_229.1", 2022-09-13T19:12:31.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T19:12:31.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:31.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:31.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:31.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:31.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:31.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:07:22Z" 2022-09-13T19:12:31.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:31.373 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:31.374 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:31.374 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:31.375 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:31.375 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.376 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.376 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.376 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:31.377 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.377 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:31.378 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.378 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.379 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.379 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.380 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:31.380 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.380 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.381 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.381 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.382 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.382 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.383 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:31.383 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.383 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.384 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:31.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.385 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.385 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.386 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:31.386 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.386 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.387 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.387 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.388 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:31.388 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.388 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.389 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.182451_229.1", 2022-09-13T19:12:31.389 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.390 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.390 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.391 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T19:12:31.391 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.391 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.392 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.392 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.393 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:31.393 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.394 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.394 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.395 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.395 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.396 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.396 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.396 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.397 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.397 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.398 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:31.398 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.399 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:31.399 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:31.400 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:31.400 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.401 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:07:22Z" 2022-09-13T19:12:31.401 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:31.401 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:31.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.182451_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.182451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:31.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.182451_229.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.182451_229.1 2022-09-13T19:12:31.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:31.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:31.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:31.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:27.790+0000 7f7679ff6600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:27.791+0000 7f7679ff6600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:31.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:31.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:31.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:31.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:31.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:31.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:31.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:31.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:31.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:31.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:31.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055098.994620_218.1", 2022-09-13T19:12:31.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:58.994620Z", 2022-09-13T19:12:31.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:31.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:31.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:31.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:31.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:31.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:07:27Z" 2022-09-13T19:12:31.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:31.440 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:31.441 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:31.441 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:31.442 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:31.443 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.444 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.445 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.446 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:31.447 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.447 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:31.448 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.449 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.450 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.450 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.451 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:31.452 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.452 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.453 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.454 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.454 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.455 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.456 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:31.457 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.458 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.459 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:31.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.460 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.461 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.462 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:31.462 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.463 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.464 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.465 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.466 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:31.467 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.467 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.468 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055098.994620_218.1", 2022-09-13T19:12:31.469 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.470 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.470 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.471 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:58.994620Z", 2022-09-13T19:12:31.472 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.473 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.475 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.475 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.476 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:31.477 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.478 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.479 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.479 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.480 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.481 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.481 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.482 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.484 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.485 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.485 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:31.486 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.486 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:31.487 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:31.487 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:31.488 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.488 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:07:27Z" 2022-09-13T19:12:31.489 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:31.489 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:31.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055098.994620_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:58.994620Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:31.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055098.994620_218.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055098.994620_218.1 2022-09-13T19:12:31.491 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:31.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:31.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:31.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:32.919+0000 7f19936c1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:32.921+0000 7f19936c1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:31.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:31.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:31.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:31.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:31.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:31.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:31.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:31.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:31.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:31.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:31.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.432486_239.1", 2022-09-13T19:12:31.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.432486Z", 2022-09-13T19:12:31.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:31.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:31.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:31.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:31.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:31.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:07:32Z" 2022-09-13T19:12:31.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:31.550 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:31.550 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:31.551 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:31.552 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:31.553 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.553 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.554 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.555 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:31.555 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.556 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:31.557 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.558 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.558 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.559 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.560 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:31.561 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.562 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.563 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.564 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.564 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.565 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.566 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:31.566 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.567 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.568 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:31.568 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.569 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.570 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.571 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:31.572 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.572 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.573 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.574 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.575 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:31.576 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.577 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.577 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.432486_239.1", 2022-09-13T19:12:31.577 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.578 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.578 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.579 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.432486Z", 2022-09-13T19:12:31.579 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.580 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.580 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.581 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.581 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:31.582 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.582 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.582 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.583 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.583 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.584 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.584 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.585 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.585 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.585 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.586 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:31.586 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.587 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:31.588 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:31.588 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:31.589 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.590 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:07:32Z" 2022-09-13T19:12:31.591 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:31.592 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:31.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.432486_239.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.432486Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:31.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.432486_239.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.432486_239.1 2022-09-13T19:12:31.594 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.594 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:31.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:31.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:31.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:38.055+0000 7f6d8a2ab600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:38.056+0000 7f6d8a2ab600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:31.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:31.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:31.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:31.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:31.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:31.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:31.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:31.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:31.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:31.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:31.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.182451_229.1", 2022-09-13T19:12:31.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T19:12:31.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:31.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:31.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:31.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:31.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:31.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:07:38Z" 2022-09-13T19:12:31.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:31.635 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:31.636 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:31.636 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:31.636 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:31.637 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.637 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.638 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.638 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:31.639 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.639 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:31.639 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.640 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.640 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.641 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.641 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:31.641 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.642 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.642 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.643 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.643 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.643 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.644 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:31.644 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.645 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.645 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:31.645 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.646 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.646 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.647 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:31.647 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.647 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.648 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.648 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.649 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:31.649 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.649 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.650 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.182451_229.1", 2022-09-13T19:12:31.650 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.651 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.651 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.652 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T19:12:31.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.652 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.653 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.653 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.653 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:31.654 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.654 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.655 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.655 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.656 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.656 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.656 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.657 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.657 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.658 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.658 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:31.659 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.659 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:31.659 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:31.660 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:31.660 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.661 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:07:38Z" 2022-09-13T19:12:31.661 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:31.661 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:31.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.182451_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.182451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:31.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.182451_229.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.182451_229.1 2022-09-13T19:12:31.663 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.664 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:31.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:31.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:31.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:43.262+0000 7f2b4dad8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:43.266+0000 7f2b4dad8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:31.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:31.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:31.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:31.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:31.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:31.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:31.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:31.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:31.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:31.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:31.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.302317_236.1", 2022-09-13T19:12:31.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.302317Z", 2022-09-13T19:12:31.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:31.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:31.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:31.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:31.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:31.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:07:43Z" 2022-09-13T19:12:31.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:31.702 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:31.703 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:31.704 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:31.704 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:31.705 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.706 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.707 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.707 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:31.708 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.709 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:31.709 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.710 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.711 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.711 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.712 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:31.713 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.713 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.714 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.715 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.715 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.716 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.717 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:31.717 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.718 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.718 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:31.719 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.719 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.719 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.720 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:31.721 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.721 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.722 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.723 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.724 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:31.724 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.725 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.726 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.302317_236.1", 2022-09-13T19:12:31.726 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.727 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.728 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.728 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.302317Z", 2022-09-13T19:12:31.729 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.729 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.730 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.731 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.731 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:31.732 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.733 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.733 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.734 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.735 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.736 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.736 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.737 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.738 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.739 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.740 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:31.740 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.741 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:31.742 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:31.743 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:31.744 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.744 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:07:43Z" 2022-09-13T19:12:31.745 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:31.746 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:31.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.302317_236.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.302317Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:31.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.302317_236.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.302317_236.1 2022-09-13T19:12:31.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.750 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:31.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:31.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:31.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:48.435+0000 7f7e4bf3b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:48.436+0000 7f7e4bf3b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:31.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:31.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:31.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:31.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:31.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:31.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:31.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:31.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:31.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:31.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:31.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.296844_235.1", 2022-09-13T19:12:31.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.296844Z", 2022-09-13T19:12:31.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:31.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:31.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:31.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:31.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:31.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:07:48Z" 2022-09-13T19:12:31.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:31.805 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:31.811 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:31.812 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:31.812 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:31.813 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.813 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.814 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.814 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:31.815 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.815 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:31.816 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.816 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.817 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.817 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.818 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:31.818 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.819 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.819 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.820 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.820 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.821 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.821 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:31.821 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.822 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.823 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:31.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.823 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.824 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:31.825 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.833 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.833 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.834 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.834 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:31.835 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.835 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.836 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.296844_235.1", 2022-09-13T19:12:31.836 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.837 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.837 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.838 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.296844Z", 2022-09-13T19:12:31.842 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.843 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.844 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.845 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.845 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:31.846 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.847 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.848 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.849 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.850 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.851 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.852 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.853 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.854 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.854 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.855 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:31.856 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.857 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:31.858 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:31.858 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:31.859 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.860 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:07:48Z" 2022-09-13T19:12:31.861 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:31.862 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:31.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.296844_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.296844Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:31.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.864 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.296844_235.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.296844_235.1 2022-09-13T19:12:31.865 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.866 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:31.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:31.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:31.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:53.584+0000 7fde939dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:53.585+0000 7fde939dc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:31.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:31.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:31.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:31.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:31.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:31.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:31.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:31.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:31.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:31.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:31.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.514523_248.1", 2022-09-13T19:12:31.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.514523Z", 2022-09-13T19:12:31.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:31.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:31.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:31.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:31.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:31.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:07:53Z" 2022-09-13T19:12:31.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:31.905 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:31.905 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:31.906 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:31.906 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:31.907 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.907 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.908 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:31.908 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.909 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:31.909 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.910 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.911 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.912 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.912 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:31.913 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.914 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.915 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.915 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.916 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.916 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.917 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:31.917 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.918 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.918 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:31.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.919 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.920 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.920 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:31.921 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.921 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.921 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.922 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.923 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:31.923 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.924 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.924 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.514523_248.1", 2022-09-13T19:12:31.924 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.925 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.925 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.926 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.514523Z", 2022-09-13T19:12:31.926 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.927 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.927 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.928 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.928 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:31.929 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.929 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.929 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.930 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.930 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.931 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.934 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.935 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.935 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.936 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.936 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:31.937 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.938 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:31.938 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:31.940 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:31.941 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.941 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:07:53Z" 2022-09-13T19:12:31.942 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:31.943 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:31.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.514523_248.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.514523Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:31.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.514523_248.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:31.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.514523_248.1 2022-09-13T19:12:31.945 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.945 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:31.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:31.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:31.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:31.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:58.729+0000 7f1bc2d3c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:07:58.730+0000 7f1bc2d3c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:31.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:31.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:31.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:31.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:31.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:31.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:31.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:31.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:31.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:31.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:31.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:31.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.340682_237.1", 2022-09-13T19:12:31.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T19:12:31.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:31.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:31.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:31.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:31.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:31.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:31.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:31.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:31.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:31.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:31.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:31.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:31.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:31.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:31.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:31.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:07:58Z" 2022-09-13T19:12:31.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:31.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:31.982 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:31.982 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:31.983 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:31.983 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:31.984 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:31.984 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.985 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.985 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:31.986 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.987 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:31.987 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.988 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.988 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:31.989 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.989 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:31.990 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.990 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:31.991 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.991 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.992 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.992 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.993 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:31.993 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:31.994 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:31.995 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:31.995 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:31.996 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:31.996 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:31.997 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:31.997 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:31.998 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:31.998 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:31.998 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:31.999 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:31.999 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.000 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.000 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.340682_237.1", 2022-09-13T19:12:32.001 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.001 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.002 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.002 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T19:12:32.003 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.003 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.004 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.004 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.005 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:32.006 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.006 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.007 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.007 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.008 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.008 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.009 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.010 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.011 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.011 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.012 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:32.013 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.013 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:32.014 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:32.015 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:32.016 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.017 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:07:58Z" 2022-09-13T19:12:32.018 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:32.019 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:32.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.340682_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.340682Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:32.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.340682_237.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.340682_237.1 2022-09-13T19:12:32.022 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.023 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:32.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:32.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:32.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:03.956+0000 7f8c02ce7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:03.957+0000 7f8c02ce7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:32.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:32.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:32.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:32.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:32.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:32.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:32.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:32.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:32.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:32.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:32.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:32.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:32.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:32.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:32.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:32.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:32.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:32.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:08:03Z" 2022-09-13T19:12:32.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:32.057 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:32.057 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:32.057 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:32.058 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:32.058 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.059 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.059 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.060 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:32.060 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.060 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:32.061 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.061 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.062 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.063 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:32.063 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.064 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.064 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.065 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.065 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.066 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.066 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:32.066 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.067 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.067 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:32.068 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.068 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.069 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.069 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:32.069 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.070 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.070 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.071 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.071 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:32.072 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.072 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.073 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:32.073 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.074 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.074 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.075 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:32.075 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.076 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.076 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.076 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.077 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:32.077 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.078 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.078 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.079 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.079 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.079 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.080 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.080 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.081 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.081 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.082 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:32.082 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.082 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:32.083 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:32.083 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:32.084 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.084 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:08:03Z" 2022-09-13T19:12:32.085 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:32.085 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:32.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:32.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T19:12:32.087 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:32.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:32.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:32.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:09.117+0000 7fe139be3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:09.118+0000 7fe139be3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:32.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:32.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:32.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:32.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:32.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:32.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:32.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:32.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:32.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:32.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:32.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:32.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:32.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:32.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:32.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:32.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:32.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:32.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:08:09Z" 2022-09-13T19:12:32.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:32.119 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:32.119 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:32.119 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:32.120 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:32.120 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.121 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.121 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.121 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:32.122 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.122 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:32.123 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.123 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.124 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.124 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.124 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:32.125 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.125 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.126 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.126 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.126 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.127 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.127 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:32.128 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.128 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.129 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:32.129 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.129 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.130 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.130 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:32.131 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.131 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.132 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.132 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.132 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:32.133 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.133 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.134 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:32.134 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.135 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.135 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.135 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:32.136 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.136 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.137 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.137 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.137 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:32.138 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.138 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.139 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.139 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.140 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.140 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.140 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.141 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.141 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.142 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.142 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:32.143 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.143 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:32.143 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:32.144 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:32.144 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.145 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:08:09Z" 2022-09-13T19:12:32.145 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:32.146 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:32.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:32.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T19:12:32.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:32.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:32.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:32.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:14.242+0000 7f0fae10e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:14.243+0000 7f0fae10e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:32.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:32.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:32.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:32.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:32.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:32.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:32.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:32.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:32.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:32.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:32.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:32.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:32.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:32.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:32.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:32.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:32.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:32.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:08:14Z" 2022-09-13T19:12:32.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:32.180 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:32.181 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:32.181 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:32.181 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:32.182 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.183 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.183 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:32.184 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.184 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:32.184 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.185 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.185 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.186 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.186 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:32.187 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.187 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.187 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.188 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.188 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.189 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.189 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:32.190 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.190 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.190 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:32.191 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.191 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.192 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.192 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:32.193 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.193 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.193 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.194 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.194 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:32.195 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.195 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.196 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:32.196 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.196 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.197 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.197 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:32.198 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.198 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.199 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.199 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.200 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:32.200 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.201 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.201 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.201 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.202 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.203 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.203 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.203 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.204 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.204 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.205 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:32.205 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.206 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:32.206 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:32.206 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:32.207 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.207 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:08:14Z" 2022-09-13T19:12:32.208 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:32.208 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:32.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:32.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T19:12:32.210 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.211 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:32.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:32.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:32.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:19.450+0000 7fe4b9a4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:19.453+0000 7fe4b9a4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:32.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:32.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:32.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:32.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:32.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:32.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:32.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:32.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:32.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:32.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:32.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.296844_235.1", 2022-09-13T19:12:32.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.296844Z", 2022-09-13T19:12:32.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:32.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:32.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:32.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:32.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:32.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:08:19Z" 2022-09-13T19:12:32.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:32.241 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:32.242 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:32.242 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:32.243 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:32.243 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.244 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.244 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.244 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:32.245 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.246 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:32.246 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.247 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.248 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.249 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:32.250 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.251 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.251 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.252 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.252 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.253 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.253 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:32.253 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.254 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.254 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:32.255 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.255 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.256 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.256 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:32.256 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.257 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.257 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.258 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.258 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:32.259 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.259 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.259 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.296844_235.1", 2022-09-13T19:12:32.260 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.261 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.261 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.296844Z", 2022-09-13T19:12:32.262 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.262 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.262 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.263 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.263 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:32.264 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.264 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.265 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.265 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.265 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.266 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.266 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.267 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.267 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.268 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.268 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:32.268 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.269 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:32.269 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:32.270 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:32.270 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.271 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:08:19Z" 2022-09-13T19:12:32.271 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:32.272 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:32.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.296844_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.296844Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:32.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.296844_235.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.296844_235.1 2022-09-13T19:12:32.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:32.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:32.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:32.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:24.585+0000 7f31c7149600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:24.585+0000 7f31c7149600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:32.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:32.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:32.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:32.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:32.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:32.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:32.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:32.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:32.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:32.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:32.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.340682_237.1", 2022-09-13T19:12:32.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T19:12:32.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:32.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:32.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:32.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:32.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:32.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:08:24Z" 2022-09-13T19:12:32.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:32.326 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:32.327 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:32.327 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:32.328 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:32.329 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.329 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.330 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.332 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:32.333 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.334 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:32.335 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.336 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.336 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.337 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.338 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:32.339 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.340 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.340 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.341 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.342 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.343 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.345 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:32.346 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.347 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.347 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:32.348 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.349 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.350 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.351 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:32.351 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.352 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.353 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.354 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.355 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:32.356 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.357 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.358 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.340682_237.1", 2022-09-13T19:12:32.359 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.360 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.361 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.362 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T19:12:32.363 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.364 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.364 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.364 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.365 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:32.365 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.366 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.366 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.367 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.368 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.369 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.369 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.370 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.370 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:32.371 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.371 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:32.371 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:32.372 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:32.372 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.373 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:08:24Z" 2022-09-13T19:12:32.373 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:32.374 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:32.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.340682_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.340682Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:32.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.340682_237.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.340682_237.1 2022-09-13T19:12:32.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:32.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:32.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:32.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:29.726+0000 7f6b2ae19600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:29.727+0000 7f6b2ae19600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:32.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:32.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:32.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:32.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:32.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:32.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:32.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:32.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:32.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:32.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:32.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.187352_231.1", 2022-09-13T19:12:32.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T19:12:32.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:32.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:32.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:32.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:32.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:32.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:08:29Z" 2022-09-13T19:12:32.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:32.409 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:32.410 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:32.411 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:32.411 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:32.411 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.412 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.412 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.413 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:32.413 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.414 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:32.414 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.415 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.415 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.415 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.416 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:32.416 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.417 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.417 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.417 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.418 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.418 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.419 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:32.419 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.420 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.421 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:32.421 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.421 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.422 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.422 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:32.423 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.423 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.424 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.424 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.425 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:32.425 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.425 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.426 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.187352_231.1", 2022-09-13T19:12:32.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.427 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.428 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.428 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T19:12:32.429 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.429 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.430 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.430 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.430 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:32.431 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.431 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.432 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.432 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.433 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.433 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.434 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.434 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.435 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.435 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.435 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:32.436 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.436 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:32.437 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:32.437 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:32.438 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.438 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:08:29Z" 2022-09-13T19:12:32.438 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:32.439 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:32.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.187352_231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.187352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:32.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.187352_231.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.187352_231.1 2022-09-13T19:12:32.441 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.441 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:32.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:32.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:32.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:34.940+0000 7fd31713f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:34.941+0000 7fd31713f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:32.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:32.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:32.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:32.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:32.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:32.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:32.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:32.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:32.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:32.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:32.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.186446_230.1", 2022-09-13T19:12:32.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.186446Z", 2022-09-13T19:12:32.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:32.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:32.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:32.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:32.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:32.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:08:34Z" 2022-09-13T19:12:32.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:32.473 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:32.475 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:32.475 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:32.476 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:32.476 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.476 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.477 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.477 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:32.478 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.478 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:32.479 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.479 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.480 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.480 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.480 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:32.481 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.481 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.482 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.482 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.482 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.483 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.483 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:32.484 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.484 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.485 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:32.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.485 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.486 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.486 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:32.487 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.487 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.487 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.488 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.488 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:32.489 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.489 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.489 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.186446_230.1", 2022-09-13T19:12:32.490 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.490 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.491 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.491 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.186446Z", 2022-09-13T19:12:32.492 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.492 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.492 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.493 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.493 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:32.494 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.494 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.494 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.495 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.495 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.496 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.496 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.496 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.497 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.497 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.498 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:32.498 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.498 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:32.499 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:32.499 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:32.500 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.500 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:08:34Z" 2022-09-13T19:12:32.500 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:32.501 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:32.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.186446_230.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.186446Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:32.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.186446_230.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.186446_230.1 2022-09-13T19:12:32.503 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.503 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:32.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:32.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:32.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:40.088+0000 7f5c13bc4600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:40.089+0000 7f5c13bc4600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:32.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:32.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:32.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:32.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:32.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:32.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:32.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:32.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:32.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:32.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:32.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:32.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:32.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:32.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:32.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:32.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:32.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:32.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:08:40Z" 2022-09-13T19:12:32.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:32.534 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:32.534 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:32.534 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:32.535 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:32.535 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.536 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.536 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.536 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:32.537 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.537 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:32.538 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.538 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.539 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.539 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.539 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:32.540 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.540 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.541 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.541 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.541 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.542 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.542 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:32.543 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.543 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.544 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:32.544 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.545 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.545 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.545 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:32.546 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.546 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.547 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.547 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.547 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:32.548 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.548 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.549 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:32.549 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.549 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.550 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.550 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:32.551 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.551 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.551 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.552 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.552 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:32.553 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.553 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.553 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.554 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.554 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.555 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.555 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.556 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.556 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.556 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.557 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:32.557 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.558 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:32.558 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:32.559 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:32.559 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.560 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:08:40Z" 2022-09-13T19:12:32.560 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:32.560 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:32.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:32.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T19:12:32.563 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:32.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:32.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:32.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:45.239+0000 7fd0e6d1c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:45.240+0000 7fd0e6d1c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:32.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:32.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:32.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:32.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:32.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:32.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:32.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:32.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:32.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:32.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:32.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.432486_239.1", 2022-09-13T19:12:32.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.432486Z", 2022-09-13T19:12:32.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:32.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:32.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:32.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:32.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:32.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:08:45Z" 2022-09-13T19:12:32.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:32.595 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:32.595 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:32.596 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:32.596 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:32.596 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.597 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.597 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.598 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:32.598 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.598 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:32.599 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.599 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.600 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.600 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.600 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:32.601 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.601 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.602 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.602 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.603 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.603 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.604 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:32.605 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.606 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.606 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:32.607 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.608 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.609 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.610 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:32.610 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.611 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.611 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.611 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.612 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:32.612 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.613 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.613 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.432486_239.1", 2022-09-13T19:12:32.613 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.614 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.614 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.615 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.432486Z", 2022-09-13T19:12:32.615 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.616 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.616 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.616 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.617 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:32.617 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.618 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.618 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.618 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.619 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.619 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.620 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.620 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.620 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.621 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.621 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:32.622 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.622 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:32.622 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:32.623 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:32.623 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.624 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:08:45Z" 2022-09-13T19:12:32.624 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:32.625 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:32.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.432486_239.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.432486Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:32.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.432486_239.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.432486_239.1 2022-09-13T19:12:32.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.627 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:32.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:32.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:32.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:50.381+0000 7f4ee2057600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:50.382+0000 7f4ee2057600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:32.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:32.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:32.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:32.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:32.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:32.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:32.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:32.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:32.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:32.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:32.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:32.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:32.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:32.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:32.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:32.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:32.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:32.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:08:50Z" 2022-09-13T19:12:32.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:32.658 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:32.658 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:32.658 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:32.659 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:32.659 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.660 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.660 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.661 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:32.661 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.661 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:32.662 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.662 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.663 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.663 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.663 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:32.664 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.664 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.665 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.666 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.666 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.666 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:32.667 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.667 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.668 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:32.668 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.668 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.669 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.669 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:32.670 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.670 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.671 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.671 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.672 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:32.672 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.672 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.673 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:32.674 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.674 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.674 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.675 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:32.675 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.676 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.676 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.676 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.677 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:32.677 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.678 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.678 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.679 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.679 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.679 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.680 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.680 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.681 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.681 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.681 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:32.682 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.682 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:32.683 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:32.683 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:32.684 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.684 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:08:50Z" 2022-09-13T19:12:32.684 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:32.685 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:32.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:32.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T19:12:32.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:32.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:32.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:32.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:55.545+0000 7f6888539600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:08:55.546+0000 7f6888539600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:32.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:32.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:32.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:32.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:32.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:32.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:32.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:32.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:32.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:32.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:32.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:32.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:32.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:32.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:32.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:32.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:32.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:32.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:08:55Z" 2022-09-13T19:12:32.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:32.725 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:32.725 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:32.726 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:32.726 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:32.727 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.727 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.728 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.728 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:32.728 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.729 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:32.729 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.730 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.730 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.730 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.731 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:32.731 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.732 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.732 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.732 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.733 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.733 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.734 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:32.734 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.734 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.735 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:32.735 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.736 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.736 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.737 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:32.737 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.738 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.738 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.739 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.739 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:32.740 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.740 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.741 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:32.741 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.742 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.742 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.743 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:32.743 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.744 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.744 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.745 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.745 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:32.746 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.746 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.747 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.748 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.748 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.749 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.749 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.750 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.750 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.751 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.751 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:32.752 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.752 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:32.753 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:32.753 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:32.753 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.754 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:08:55Z" 2022-09-13T19:12:32.754 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:32.755 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:32.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:32.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T19:12:32.757 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.757 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:32.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:32.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:32.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:00.691+0000 7fa6ba581600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:00.692+0000 7fa6ba581600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:32.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:32.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:32.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:32.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:32.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:32.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:32.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:32.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:32.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:32.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:32.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:32.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:32.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:32.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:32.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:32.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:32.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:32.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:09:00Z" 2022-09-13T19:12:32.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:32.788 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:32.788 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:32.789 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:32.789 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:32.789 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.790 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.790 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.791 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:32.791 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.791 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:32.792 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.792 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.793 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.793 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.793 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:32.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.794 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.795 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.795 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.795 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.796 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.796 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:32.797 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.797 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.797 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:32.798 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.798 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.799 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.799 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:32.800 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.800 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.801 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.801 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.802 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:32.802 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.803 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.803 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:32.803 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.804 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.804 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.805 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:32.805 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.806 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.806 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.806 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.807 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:32.807 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.808 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.808 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.809 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.809 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.810 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.810 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.810 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.811 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.811 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.812 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:32.812 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.813 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:32.813 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:32.814 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:32.814 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.815 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:09:00Z" 2022-09-13T19:12:32.815 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:32.816 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:32.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:32.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T19:12:32.818 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.818 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:32.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:32.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:32.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:05.845+0000 7fee78fae600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:05.846+0000 7fee78fae600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:32.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:32.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:32.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:32.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:32.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:32.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:32.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:32.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:32.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:32.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:32.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.432486_239.1", 2022-09-13T19:12:32.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.432486Z", 2022-09-13T19:12:32.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:32.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:32.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:32.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:32.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:32.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:09:05Z" 2022-09-13T19:12:32.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:32.874 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:32.874 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:32.875 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:32.875 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:32.876 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.876 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.877 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.877 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:32.877 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.878 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:32.878 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.879 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.879 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.879 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.880 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:32.881 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.881 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.882 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.882 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.882 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.883 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.883 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:32.884 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.884 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.886 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:32.886 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.887 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.888 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.889 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:32.890 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.891 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.892 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.892 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.894 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:32.895 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.895 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.895 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.432486_239.1", 2022-09-13T19:12:32.896 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.896 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.897 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.432486Z", 2022-09-13T19:12:32.898 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.898 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.899 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.899 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.900 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:32.900 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.900 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.901 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.901 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.902 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.902 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.903 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.903 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.903 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.904 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.904 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:32.905 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.905 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:32.906 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:32.906 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:32.907 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.907 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:09:05Z" 2022-09-13T19:12:32.908 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:32.908 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:32.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.432486_239.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.432486Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:32.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.432486_239.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:32.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.432486_239.1 2022-09-13T19:12:32.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:32.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:32.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:32.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:32.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:10.992+0000 7fccd5dc8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:10.993+0000 7fccd5dc8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:32.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:32.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:32.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:32.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:32.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:32.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:32.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:32.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:32.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:32.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:32.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:32.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:32.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:32.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:32.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:32.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:32.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:32.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:32.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:32.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:32.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:32.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:32.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:32.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:32.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:32.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:32.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:09:10Z" 2022-09-13T19:12:32.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:32.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:32.952 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:32.952 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:32.953 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:32.954 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:32.954 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:32.955 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.956 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.957 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:32.957 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.958 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:32.959 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.960 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.972 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:32.973 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.973 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:32.974 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.974 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:32.975 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.975 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.976 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.976 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.977 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:32.977 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.978 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.978 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:32.979 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.979 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.980 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.980 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:32.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.981 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.982 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.982 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.983 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:32.983 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.984 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.984 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:32.985 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.985 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.986 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.987 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:32.987 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.988 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.988 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.989 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:32.990 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:32.990 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:32.990 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:32.991 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:32.991 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:32.994 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:32.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:32.995 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:32.996 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:32.996 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.997 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:32.997 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:32.998 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:32.998 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:32.998 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:32.999 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:32.999 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.000 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:09:10Z" 2022-09-13T19:12:33.000 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:33.001 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:33.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:33.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T19:12:33.003 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.003 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:33.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:33.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:33.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:16.227+0000 7fe7604ff600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:16.228+0000 7fe7604ff600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:33.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:33.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:33.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:33.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:33.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:33.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:33.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:33.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:33.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:33.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:33.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.223597_234.1", 2022-09-13T19:12:33.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.223597Z", 2022-09-13T19:12:33.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:33.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:33.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:33.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:33.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:33.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:09:16Z" 2022-09-13T19:12:33.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:33.045 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:33.046 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:33.046 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:33.047 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:33.047 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.048 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.048 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:33.049 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.049 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:33.050 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.050 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.050 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.051 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:33.052 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.052 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.053 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.053 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.054 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.055 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:33.055 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.055 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.056 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:33.056 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.057 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.057 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.057 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:33.058 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.058 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.059 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.059 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.060 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:33.060 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.061 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.062 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.223597_234.1", 2022-09-13T19:12:33.062 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.063 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.223597Z", 2022-09-13T19:12:33.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.066 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.067 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.067 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.068 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:33.068 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.069 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.069 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.070 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.070 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.070 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.071 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.072 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.073 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.074 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.075 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:33.076 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.076 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:33.077 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:33.078 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:33.079 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.080 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:09:16Z" 2022-09-13T19:12:33.080 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:33.081 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:33.081 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.223597_234.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.223597Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:33.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.223597_234.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.223597_234.1 2022-09-13T19:12:33.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.084 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:33.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:33.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:33.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:21.366+0000 7f7371985600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:21.367+0000 7f7371985600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:33.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:33.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:33.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:33.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:33.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:33.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:33.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:33.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:33.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:33.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:33.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:33.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:33.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:33.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:33.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:33.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:33.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:33.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:09:21Z" 2022-09-13T19:12:33.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:33.139 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:33.140 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:33.141 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:33.142 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:33.142 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.143 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.144 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.145 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:33.146 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.147 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:33.147 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.148 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.149 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.150 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.150 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:33.151 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.152 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.153 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.154 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.154 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.155 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.156 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:33.157 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.157 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.158 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:33.159 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.160 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.161 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.161 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:33.162 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.163 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.163 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.164 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.165 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:33.165 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.166 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.167 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:33.168 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.168 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.169 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.170 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:33.170 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.171 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.172 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.173 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.174 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:33.174 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.175 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.176 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.177 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.178 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.178 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.179 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.179 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.179 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.180 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.180 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:33.181 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.181 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:33.181 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:33.182 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:33.182 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.183 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:09:21Z" 2022-09-13T19:12:33.183 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:33.184 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:33.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:33.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T19:12:33.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.187 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:33.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:33.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:33.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:26.504+0000 7f129af35600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:26.505+0000 7f129af35600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:33.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:33.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:33.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:33.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:33.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:33.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:33.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:33.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:33.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:33.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:33.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:33.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:33.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:33.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:33.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:33.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:33.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:33.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:09:26Z" 2022-09-13T19:12:33.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:33.229 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:33.229 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:33.230 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:33.231 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:33.232 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.232 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.233 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.234 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:33.235 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.235 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:33.236 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.237 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.238 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.239 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.239 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:33.240 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.241 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.242 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.242 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.243 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.244 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.245 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:33.245 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.246 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.247 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:33.248 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.248 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.250 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:33.251 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.251 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.252 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.253 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.254 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:33.254 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.255 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.256 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:33.256 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.257 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.258 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.258 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:33.259 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.260 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.260 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.261 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.262 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:33.263 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.263 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.264 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.264 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.265 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.266 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.267 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.267 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.267 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.268 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:33.269 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.270 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:33.271 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:33.272 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:33.272 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.273 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:09:26Z" 2022-09-13T19:12:33.274 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:33.275 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:33.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:33.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:33.278 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:33.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:33.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:33.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:31.680+0000 7f5073e0c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:31.681+0000 7f5073e0c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:33.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:33.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:33.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:33.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:33.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:33.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:33.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:33.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:33.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:33.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:33.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.351164_238.1", 2022-09-13T19:12:33.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.351164Z", 2022-09-13T19:12:33.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:33.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:33.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:33.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:33.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:33.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:09:31Z" 2022-09-13T19:12:33.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:33.341 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:33.342 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:33.343 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:33.344 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:33.345 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.345 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.346 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.347 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:33.348 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.349 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:33.350 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.351 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.351 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.352 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.352 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:33.352 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.353 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.353 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.354 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.354 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.355 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.355 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:33.355 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.356 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.356 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:33.357 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.357 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.357 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.358 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:33.358 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.359 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.359 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.360 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.360 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:33.360 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.361 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.361 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.351164_238.1", 2022-09-13T19:12:33.362 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.362 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.363 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.363 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.351164Z", 2022-09-13T19:12:33.364 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.364 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.364 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.365 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.365 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:33.366 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.366 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.367 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.368 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.368 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.369 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.369 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.370 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.370 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:33.371 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.371 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:33.372 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:33.372 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:33.373 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.373 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:09:31Z" 2022-09-13T19:12:33.374 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:33.374 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:33.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.351164_238.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.351164Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:33.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.351164_238.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.351164_238.1 2022-09-13T19:12:33.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:33.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:33.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:33.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:36.830+0000 7f5d39624600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:36.831+0000 7f5d39624600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:33.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:33.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:33.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:33.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:33.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:33.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:33.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:33.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:33.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:33.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:33.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:33.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:33.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:33.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:33.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:33.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:33.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:33.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:09:36Z" 2022-09-13T19:12:33.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:33.415 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:33.416 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:33.418 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:33.418 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:33.419 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.420 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.421 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.422 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:33.422 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.423 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:33.424 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.424 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.425 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.426 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.426 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:33.427 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.428 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.428 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.429 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.429 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.430 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.430 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:33.431 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.431 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.432 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:33.432 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.433 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.433 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.433 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:33.434 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.434 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.435 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.435 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.436 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:33.436 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.436 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.437 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:33.437 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.438 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.438 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.439 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:33.439 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.440 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.440 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.441 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.441 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:33.441 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.442 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.442 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.443 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.443 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.444 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.445 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.446 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.446 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:33.447 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.447 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:33.448 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:33.448 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:33.449 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.449 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:09:36Z" 2022-09-13T19:12:33.449 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:33.450 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:33.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:33.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T19:12:33.452 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.453 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:33.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:33.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:33.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:41.980+0000 7f48d2dd2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:41.981+0000 7f48d2dd2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:33.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:33.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:33.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:33.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:33.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:33.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:33.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:33.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:33.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:33.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:33.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.459269_243.1", 2022-09-13T19:12:33.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.459269Z", 2022-09-13T19:12:33.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:33.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:33.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:33.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:33.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:33.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:09:41Z" 2022-09-13T19:12:33.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:33.487 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:33.487 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:33.488 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:33.488 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:33.489 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.489 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.489 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.490 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:33.490 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.491 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:33.491 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.492 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.492 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.493 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.493 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:33.493 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.494 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.495 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.495 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.496 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.496 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.497 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:33.497 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.497 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.498 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:33.498 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.499 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.499 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.500 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:33.500 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.501 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.501 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.502 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.502 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:33.502 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.503 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.503 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.459269_243.1", 2022-09-13T19:12:33.504 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.504 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.505 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.459269Z", 2022-09-13T19:12:33.506 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.506 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.506 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.507 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.507 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:33.508 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.508 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.508 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.509 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.509 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.510 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.510 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.511 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.511 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.512 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.512 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:33.512 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.513 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:33.513 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:33.514 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:33.514 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.515 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:09:41Z" 2022-09-13T19:12:33.515 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:33.515 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:33.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.459269_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.459269Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:33.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.459269_243.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.459269_243.1 2022-09-13T19:12:33.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:33.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:33.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:33.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:47.159+0000 7f95fe8b7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:47.160+0000 7f95fe8b7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:33.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:33.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:33.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:33.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:33.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:33.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:33.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:33.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:33.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:33.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:33.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.146590_228.1", 2022-09-13T19:12:33.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.146590Z", 2022-09-13T19:12:33.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:33.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:33.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:33.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:33.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:33.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:09:47Z" 2022-09-13T19:12:33.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:33.577 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:33.578 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:33.579 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:33.580 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:33.582 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.583 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.583 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.586 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:33.587 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.587 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:33.588 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.589 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.590 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.591 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.592 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:33.593 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.593 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.594 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.595 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.596 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.597 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.598 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:33.599 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.600 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.601 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:33.601 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.603 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.603 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.604 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:33.605 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.606 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.607 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.608 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.609 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:33.609 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.610 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.611 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.146590_228.1", 2022-09-13T19:12:33.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.613 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.614 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.614 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.146590Z", 2022-09-13T19:12:33.615 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.616 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.617 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.618 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.618 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:33.619 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.619 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.620 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.620 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.621 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.622 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.622 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.623 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.624 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.625 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.625 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:33.626 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.626 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:33.627 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:33.627 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:33.628 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.628 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:09:47Z" 2022-09-13T19:12:33.629 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:33.629 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:33.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.146590_228.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.146590Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:33.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.146590_228.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.146590_228.1 2022-09-13T19:12:33.631 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.632 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:33.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:33.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:33.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:52.362+0000 7fb6ca414600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:52.364+0000 7fb6ca414600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:33.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:33.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:33.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:33.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:33.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:33.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:33.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:33.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:33.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:33.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:33.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055101.685819_259.1", 2022-09-13T19:12:33.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:45:01.685819Z", 2022-09-13T19:12:33.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:33.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:33.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:33.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:33.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:33.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:09:52Z" 2022-09-13T19:12:33.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:33.666 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:33.666 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:33.666 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:33.667 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:33.667 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.668 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.668 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.669 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:33.669 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.670 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:33.670 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.670 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.671 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.671 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.672 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:33.672 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.673 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.673 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.674 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.674 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.675 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.675 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:33.676 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.676 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.677 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:33.677 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.678 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.678 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.679 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:33.679 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.680 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.680 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.681 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.682 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:33.682 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.683 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.683 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055101.685819_259.1", 2022-09-13T19:12:33.684 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.684 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.685 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.686 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:45:01.685819Z", 2022-09-13T19:12:33.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.687 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.687 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.688 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.688 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:33.689 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.689 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.690 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.691 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.691 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.692 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.692 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.694 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.694 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.695 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.696 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:33.697 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.697 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:33.698 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:33.700 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:33.700 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.701 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:09:52Z" 2022-09-13T19:12:33.701 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:33.701 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:33.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055101.685819_259.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:45:01.685819Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:33.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.685819_259.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055101.685819_259.1 2022-09-13T19:12:33.704 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.705 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:33.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:33.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:33.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:57.502+0000 7f33e14b9600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:09:57.503+0000 7f33e14b9600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:33.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:33.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:33.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:33.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:33.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:33.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:33.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:33.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:33.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:33.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:33.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.223597_234.1", 2022-09-13T19:12:33.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.223597Z", 2022-09-13T19:12:33.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:33.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:33.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:33.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:33.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:33.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:09:57Z" 2022-09-13T19:12:33.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:33.741 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:33.742 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:33.742 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:33.743 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:33.743 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.743 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.744 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.744 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:33.745 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.746 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:33.746 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.747 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.747 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.748 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.748 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:33.749 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.749 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.750 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.750 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.751 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.751 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.752 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:33.752 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.753 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.753 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:33.754 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.754 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.755 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.755 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:33.756 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.756 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.756 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.757 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.757 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:33.758 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.758 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.759 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.223597_234.1", 2022-09-13T19:12:33.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.760 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.760 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.761 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.223597Z", 2022-09-13T19:12:33.761 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.762 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.762 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.763 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.763 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:33.764 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.764 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.765 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.765 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.766 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.766 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.767 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.767 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.768 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.768 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.769 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:33.769 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.770 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:33.770 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:33.771 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:33.771 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.772 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:09:57Z" 2022-09-13T19:12:33.772 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:33.773 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:33.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.223597_234.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.223597Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:33.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.223597_234.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.223597_234.1 2022-09-13T19:12:33.775 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.776 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:33.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:33.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:33.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:02.641+0000 7f560ca44600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:02.642+0000 7f560ca44600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:33.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:33.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:33.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:33.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:33.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:33.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:33.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:33.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:33.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:33.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:33.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.459269_243.1", 2022-09-13T19:12:33.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.459269Z", 2022-09-13T19:12:33.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:33.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:33.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:33.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:33.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:33.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:10:02Z" 2022-09-13T19:12:33.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:33.811 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:33.812 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:33.812 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:33.813 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:33.813 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.814 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.814 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.815 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:33.815 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.816 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:33.816 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.817 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.817 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.818 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.818 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:33.819 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.819 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.819 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.820 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.820 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.821 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.821 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:33.822 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.823 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.823 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:33.824 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.824 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.825 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:33.825 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.826 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.826 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.827 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.827 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:33.828 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.828 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.829 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.459269_243.1", 2022-09-13T19:12:33.829 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.829 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.830 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.831 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.459269Z", 2022-09-13T19:12:33.831 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.832 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.832 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.833 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.833 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:33.834 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.834 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.835 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.835 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.836 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.836 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.837 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.837 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.838 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.838 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.839 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:33.839 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.839 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:33.840 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:33.841 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:33.841 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.842 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:10:02Z" 2022-09-13T19:12:33.842 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:33.843 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:33.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.459269_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.459269Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:33.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.459269_243.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.459269_243.1 2022-09-13T19:12:33.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.846 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:33.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:33.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:33.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:07.894+0000 7f3c85475600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:07.895+0000 7f3c85475600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:33.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:33.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:33.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:33.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:33.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:33.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:33.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:33.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:33.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:33.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:33.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:33.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:33.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:33.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:33.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:33.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:33.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:33.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:10:07Z" 2022-09-13T19:12:33.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:33.883 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:33.883 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:33.883 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:33.884 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:33.884 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.885 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.885 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.886 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:33.886 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.887 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:33.887 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.888 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.888 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.889 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.889 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:33.890 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.890 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.890 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.891 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.891 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.892 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.892 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:33.893 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.893 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.894 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:33.894 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.895 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.895 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.896 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:33.897 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.897 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.898 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.898 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.899 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:33.899 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.900 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.900 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.142561_227.1", 2022-09-13T19:12:33.901 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.902 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.902 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.903 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.142561Z", 2022-09-13T19:12:33.903 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.904 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.904 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.905 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.905 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:33.906 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.906 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.907 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.908 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.909 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.909 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.910 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.910 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.911 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.911 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.912 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:33.912 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.913 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:33.914 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:33.914 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:33.915 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.915 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:10:07Z" 2022-09-13T19:12:33.916 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:33.916 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:33.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.142561_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.142561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:33.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.142561_227.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:33.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.142561_227.1 2022-09-13T19:12:33.919 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.919 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:33.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:33.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:33.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:33.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:13.028+0000 7f9ec2d93600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:13.029+0000 7f9ec2d93600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:33.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:33.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:33.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:33.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:33.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:33.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:33.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:33.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:33.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:33.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:33.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:33.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.187352_231.1", 2022-09-13T19:12:33.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T19:12:33.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:33.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:33.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:33.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:33.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:33.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:33.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:33.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:33.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:33.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:33.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:33.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:33.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:33.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:10:13Z" 2022-09-13T19:12:33.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:33.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:33.955 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:33.956 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:33.957 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:33.958 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:33.959 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:33.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.960 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.961 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:33.962 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.963 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:33.964 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.964 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.965 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:33.966 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.967 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:33.968 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.969 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:33.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.970 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.971 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.972 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.972 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:33.973 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.974 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.974 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:33.975 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.975 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.976 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.976 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:33.977 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.978 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.979 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.980 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.980 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:33.981 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.981 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.981 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.187352_231.1", 2022-09-13T19:12:33.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.982 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.983 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.983 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.187352Z", 2022-09-13T19:12:33.984 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.985 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.985 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.985 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:33.986 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:33.987 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:33.987 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:33.988 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:33.989 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:33.989 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:33.990 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:33.991 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:33.991 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:33.991 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.992 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:33.992 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:33.993 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.993 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:33.994 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:33.995 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:33.995 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:33.996 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:10:13Z" 2022-09-13T19:12:33.996 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:33.997 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:33.998 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.187352_231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.187352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:33.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.187352_231.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.187352_231.1 2022-09-13T19:12:34.001 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:34.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:34.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:34.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:18.168+0000 7f663e527600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:18.169+0000 7f663e527600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:34.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:34.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:34.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:34.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:34.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:34.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:34.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:34.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:34.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:34.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:34.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:34.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:34.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:34.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:34.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:34.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:34.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:34.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:10:18Z" 2022-09-13T19:12:34.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:34.059 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:34.060 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:34.060 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:34.061 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:34.062 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.064 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.064 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:34.065 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.065 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:34.066 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.066 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.067 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.067 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.068 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:34.068 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.069 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.069 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.070 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.070 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.071 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.072 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:34.073 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.073 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.074 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:34.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.076 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.077 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.078 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:34.079 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.080 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.081 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.081 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.082 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:34.083 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.084 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.084 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:34.085 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.086 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.087 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.088 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:34.089 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.090 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.091 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.091 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.091 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:34.092 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.092 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.093 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.093 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.094 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.094 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.095 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.096 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.096 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.097 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.098 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:34.099 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.100 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:34.101 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:34.102 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:34.102 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.103 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:10:18Z" 2022-09-13T19:12:34.103 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:34.104 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:34.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:34.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:34.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:34.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:34.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:34.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:23.322+0000 7fb392415600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:23.323+0000 7fb392415600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:34.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:34.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:34.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:34.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:34.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:34.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:34.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:34.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:34.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:34.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:34.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.340682_237.1", 2022-09-13T19:12:34.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T19:12:34.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:34.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:34.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:34.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:34.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:34.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:10:23Z" 2022-09-13T19:12:34.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:34.167 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:34.168 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:34.168 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:34.169 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:34.170 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.171 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.171 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.172 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:34.173 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.174 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:34.174 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.175 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.176 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.177 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.177 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:34.178 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.179 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.180 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.180 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.181 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.182 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.183 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:34.184 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.184 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.184 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:34.185 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.185 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.186 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.186 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:34.187 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.187 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.187 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.188 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.188 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:34.189 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.189 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.189 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.340682_237.1", 2022-09-13T19:12:34.190 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.190 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.191 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.191 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T19:12:34.192 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.192 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.193 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.193 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.193 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:34.194 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.194 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.195 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.195 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.196 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.196 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.197 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.197 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.198 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.198 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:34.199 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.199 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:34.200 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:34.200 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:34.200 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.201 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:10:23Z" 2022-09-13T19:12:34.201 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:34.202 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:34.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.340682_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.340682Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:34.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.340682_237.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.340682_237.1 2022-09-13T19:12:34.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:34.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:34.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:34.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:28.464+0000 7fbf0116e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:28.466+0000 7fbf0116e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:34.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:34.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:34.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:34.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:34.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:34.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:34.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:34.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:34.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:34.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:34.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.351164_238.1", 2022-09-13T19:12:34.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.351164Z", 2022-09-13T19:12:34.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:34.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:34.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:34.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:34.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:34.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:10:28Z" 2022-09-13T19:12:34.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:34.243 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:34.244 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:34.244 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:34.244 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:34.245 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.245 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.246 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.246 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:34.247 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.247 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:34.247 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.248 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.248 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.249 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:34.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.250 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.250 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.251 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.251 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.251 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.265 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:34.266 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.266 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.266 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:34.267 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.267 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.268 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.268 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:34.269 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.269 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.269 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.270 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.270 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:34.271 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.271 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.272 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.351164_238.1", 2022-09-13T19:12:34.272 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.272 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.273 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.273 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.351164Z", 2022-09-13T19:12:34.274 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.274 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.275 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.275 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.275 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:34.276 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.276 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.277 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.277 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.277 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.278 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.278 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.279 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.279 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.279 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.280 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:34.280 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.281 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:34.281 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:34.282 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:34.282 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.282 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:10:28Z" 2022-09-13T19:12:34.283 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:34.283 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:34.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.351164_238.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.351164Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:34.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.351164_238.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.351164_238.1 2022-09-13T19:12:34.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.286 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:34.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:34.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:34.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:33.605+0000 7f1b4c8fc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:33.606+0000 7f1b4c8fc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:34.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:34.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:34.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:34.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:34.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:34.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:34.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:34.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:34.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:34.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:34.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.223597_234.1", 2022-09-13T19:12:34.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.223597Z", 2022-09-13T19:12:34.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:34.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:34.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:34.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:34.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:34.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:10:33Z" 2022-09-13T19:12:34.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:34.316 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:34.316 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:34.317 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:34.317 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:34.318 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.318 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.318 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.319 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:34.319 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.320 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:34.320 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.321 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.321 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.321 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.322 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:34.322 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.323 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.323 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.324 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.324 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.324 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.325 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:34.325 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.326 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.326 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:34.326 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.327 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.327 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.328 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:34.328 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.328 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.329 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.329 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.330 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:34.330 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.331 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.331 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.223597_234.1", 2022-09-13T19:12:34.332 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.332 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.332 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.333 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.223597Z", 2022-09-13T19:12:34.333 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.334 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.334 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.335 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.335 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:34.336 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.336 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.336 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.337 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.337 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.338 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.338 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.339 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.339 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.339 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.340 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:34.340 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.341 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:34.341 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:34.342 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:34.342 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.342 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:10:33Z" 2022-09-13T19:12:34.343 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:34.343 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:34.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.223597_234.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.223597Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:34.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.223597_234.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.223597_234.1 2022-09-13T19:12:34.345 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:34.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:34.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:34.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:38.738+0000 7ff50c74f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:38.739+0000 7ff50c74f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:34.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:34.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:34.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:34.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:34.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:34.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:34.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:34.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:34.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:34.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:34.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:34.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:34.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:34.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:34.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:34.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:34.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:34.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:10:38Z" 2022-09-13T19:12:34.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:34.381 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:34.382 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:34.383 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:34.384 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:34.385 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.385 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.386 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.387 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:34.388 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.389 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:34.390 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.391 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.392 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.393 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:34.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.395 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.396 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.397 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.398 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.399 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.400 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:34.400 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.401 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.402 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:34.403 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.404 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.406 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:34.407 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.408 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.409 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.409 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.410 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:34.411 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.412 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.413 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:34.413 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.414 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.415 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.416 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:34.417 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.417 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.418 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.419 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.420 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:34.421 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.422 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.422 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.423 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.424 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.425 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.426 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.428 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.429 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.430 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.431 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:34.432 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.432 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:34.433 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:34.434 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:34.435 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.436 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:10:38Z" 2022-09-13T19:12:34.437 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:34.438 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:34.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:34.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T19:12:34.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.443 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:34.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:34.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:34.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:43.896+0000 7f5681e3c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:43.898+0000 7f5681e3c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:34.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:34.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:34.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:34.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:34.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:34.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:34.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:34.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:34.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:34.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:34.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055101.622388_255.1", 2022-09-13T19:12:34.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:45:01.622388Z", 2022-09-13T19:12:34.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:34.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:34.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:34.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:34.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:34.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:10:43Z" 2022-09-13T19:12:34.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:34.487 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:34.488 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:34.488 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:34.489 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:34.489 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.489 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.490 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.491 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:34.491 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.492 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:34.492 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.493 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.493 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.493 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.494 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:34.494 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.495 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.495 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.496 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.496 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.497 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.497 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:34.497 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.498 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.498 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:34.499 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.499 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.500 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.500 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:34.500 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.501 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.501 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.502 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.503 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:34.504 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.504 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.504 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055101.622388_255.1", 2022-09-13T19:12:34.505 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.506 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.506 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.506 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:45:01.622388Z", 2022-09-13T19:12:34.507 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.507 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.508 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.508 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.509 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:34.509 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.510 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.510 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.510 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.511 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.511 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.512 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.513 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.513 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.514 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.514 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:34.515 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.515 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:34.516 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:34.516 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:34.516 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.517 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:10:43Z" 2022-09-13T19:12:34.517 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:34.518 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:34.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055101.622388_255.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:45:01.622388Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:34.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055101.622388_255.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055101.622388_255.1 2022-09-13T19:12:34.520 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:34.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:34.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:34.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:49.040+0000 7fc3d0cdc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:49.041+0000 7fc3d0cdc600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:34.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:34.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:34.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:34.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:34.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:34.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:34.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:34.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:34.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:34.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:34.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.223597_234.1", 2022-09-13T19:12:34.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.223597Z", 2022-09-13T19:12:34.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:34.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:34.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:34.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:34.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:34.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:10:49Z" 2022-09-13T19:12:34.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:34.553 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:34.553 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:34.554 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:34.554 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:34.555 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.556 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.556 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:34.556 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.557 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:34.557 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.558 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.558 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.559 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.559 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:34.560 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.560 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.560 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.561 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.561 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.562 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.562 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:34.563 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.563 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.563 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:34.564 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.564 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.565 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.565 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:34.565 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.566 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.566 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.567 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.567 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:34.568 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.568 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.569 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.223597_234.1", 2022-09-13T19:12:34.569 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.570 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.570 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.571 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.223597Z", 2022-09-13T19:12:34.571 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.571 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.572 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.572 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.573 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:34.573 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.574 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.574 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.575 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.575 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.576 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.576 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.577 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.577 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.577 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.578 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:34.578 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.579 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:34.579 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:34.580 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:34.580 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.580 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:10:49Z" 2022-09-13T19:12:34.581 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:34.581 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:34.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.223597_234.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.223597Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:34.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.223597_234.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.223597_234.1 2022-09-13T19:12:34.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:34.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:34.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:34.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:54.201+0000 7f050b5b2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:54.202+0000 7f050b5b2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:34.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:34.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:34.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:34.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:34.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:34.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:34.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:34.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:34.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:34.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:34.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:34.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:34.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:34.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:34.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:34.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:34.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:34.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:10:54Z" 2022-09-13T19:12:34.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:34.631 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:34.632 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:34.632 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:34.633 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:34.633 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.634 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.634 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.635 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:34.635 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.636 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:34.636 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.636 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.637 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.637 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.638 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:34.639 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.639 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.640 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.640 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.641 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.641 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.642 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:34.642 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.643 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.643 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:34.644 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.644 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.645 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.645 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:34.645 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.646 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.646 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.647 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.647 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:34.648 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.648 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.649 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:34.649 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.650 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.651 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.651 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:34.651 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.652 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.652 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.653 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.653 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:34.654 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.654 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.655 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.656 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.656 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.657 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.657 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.658 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.658 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.659 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.659 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:34.660 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.660 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:34.661 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:34.661 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:34.662 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.662 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:10:54Z" 2022-09-13T19:12:34.663 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:34.663 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:34.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:34.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T19:12:34.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.667 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:34.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:34.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:34.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:59.357+0000 7f9f94718600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:10:59.358+0000 7f9f94718600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:34.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:34.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:34.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:34.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:34.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:34.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:34.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:34.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:34.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:34.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:34.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.340682_237.1", 2022-09-13T19:12:34.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T19:12:34.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:34.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:34.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:34.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:34.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:34.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:10:59Z" 2022-09-13T19:12:34.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:34.704 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:34.704 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:34.705 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:34.705 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:34.705 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.707 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.707 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.708 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:34.708 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.709 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:34.709 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.710 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.710 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.711 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.711 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:34.712 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.712 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.713 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.713 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.714 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.714 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.715 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:34.716 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.716 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.717 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:34.718 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.719 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.720 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.720 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:34.721 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.723 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.723 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.724 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.725 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:34.726 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.726 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.727 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.340682_237.1", 2022-09-13T19:12:34.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.728 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.729 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.730 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.340682Z", 2022-09-13T19:12:34.731 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.731 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.732 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.733 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.733 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:34.734 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.735 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.736 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.736 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.736 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.737 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.737 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.738 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.738 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.739 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.739 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:34.739 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.740 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:34.740 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:34.741 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:34.741 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.742 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:10:59Z" 2022-09-13T19:12:34.742 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:34.742 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:34.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.340682_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.340682Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:34.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.340682_237.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.340682_237.1 2022-09-13T19:12:34.744 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.745 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:34.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:34.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:34.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:04.571+0000 7f06496b5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:04.572+0000 7f06496b5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:34.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:34.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:34.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:34.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:34.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:34.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:34.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:34.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:34.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:34.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:34.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:34.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:34.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:34.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:34.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:34.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:34.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:34.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:11:04Z" 2022-09-13T19:12:34.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:34.776 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:34.776 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:34.777 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:34.777 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:34.777 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.778 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.778 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.779 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:34.779 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.779 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:34.780 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.780 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.781 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.781 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.782 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:34.782 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.782 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.783 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.783 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.784 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.784 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.784 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:34.785 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.785 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.786 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:34.786 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.787 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.787 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.787 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:34.788 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.788 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.789 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.789 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.790 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:34.790 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.790 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.791 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:34.791 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.792 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.792 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:34.793 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.793 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.794 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.794 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.795 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:34.795 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.795 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.796 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.796 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.797 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.797 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.798 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.798 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.798 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.799 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.799 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:34.800 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.800 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:34.800 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:34.801 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:34.801 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.802 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:11:04Z" 2022-09-13T19:12:34.802 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:34.803 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:34.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:34.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T19:12:34.805 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.805 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:34.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:34.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:34.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:09.720+0000 7f3d2ce02600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:09.721+0000 7f3d2ce02600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:34.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:34.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:34.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:34.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:34.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:34.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:34.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:34.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:34.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:34.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:34.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055100.589473_253.1", 2022-09-13T19:12:34.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:45:00.589473Z", 2022-09-13T19:12:34.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:34.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:34.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:34.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:34.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:34.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:11:09Z" 2022-09-13T19:12:34.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:34.837 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:34.837 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:34.838 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:34.838 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:34.839 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.839 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.840 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.841 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:34.841 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.842 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:34.842 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.843 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.843 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.844 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.845 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:34.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.846 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.846 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.847 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.847 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.848 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.848 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:34.848 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.850 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.851 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:34.852 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.853 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.854 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.855 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:34.856 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.857 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.860 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.861 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.862 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:34.863 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.864 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.865 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055100.589473_253.1", 2022-09-13T19:12:34.865 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.866 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.866 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.867 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:45:00.589473Z", 2022-09-13T19:12:34.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.869 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.870 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.871 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.871 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:34.872 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.873 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.874 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.874 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.875 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.876 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.877 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.877 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.878 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.879 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.880 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:34.881 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.881 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:34.882 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:34.883 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:34.884 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.884 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:11:09Z" 2022-09-13T19:12:34.886 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:34.887 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:34.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055100.589473_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:45:00.589473Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:34.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055100.589473_253.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:34.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055100.589473_253.1 2022-09-13T19:12:34.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:34.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:34.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:34.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:34.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:14.871+0000 7fc3ecf6a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:14.872+0000 7fc3ecf6a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:34.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:34.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:34.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:34.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:34.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:34.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:34.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:34.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:34.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:34.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:34.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:34.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.223597_234.1", 2022-09-13T19:12:34.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.223597Z", 2022-09-13T19:12:34.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:34.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:34.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:34.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:34.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:34.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:34.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:34.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:34.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:34.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:34.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:34.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:34.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:34.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:34.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:11:14Z" 2022-09-13T19:12:34.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:34.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:34.955 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:34.955 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:34.956 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:34.957 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:34.958 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:34.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.960 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.960 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:34.961 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.962 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:34.962 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.963 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.964 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:34.965 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.966 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:34.967 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.968 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:34.968 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.969 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.969 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.970 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.971 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:34.971 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.972 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.972 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:34.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.973 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.973 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.974 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:34.974 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.975 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.975 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.976 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.976 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:34.977 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.977 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.977 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.223597_234.1", 2022-09-13T19:12:34.978 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.978 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.979 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:34.980 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.223597Z", 2022-09-13T19:12:34.986 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:34.987 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:34.988 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:34.989 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:34.989 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:34.993 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:34.994 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:34.995 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:34.996 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:34.996 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:34.997 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.000 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.002 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.014 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.015 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.016 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:35.017 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.018 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:35.019 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:35.020 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:35.021 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.022 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:11:14Z" 2022-09-13T19:12:35.023 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:35.024 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.223597_234.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.223597Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:35.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.223597_234.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.223597_234.1 2022-09-13T19:12:35.030 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.031 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:35.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:35.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:35.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:20.011+0000 7f2f552e7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:20.013+0000 7f2f552e7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:35.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:35.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:35.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:35.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:35.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:35.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:35.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:35.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:35.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:35.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:35.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:35.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:35.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:35.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:35.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:35.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:35.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:35.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:11:20Z" 2022-09-13T19:12:35.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:35.102 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:35.103 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:35.103 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:35.104 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:35.105 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.106 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.107 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.107 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:35.108 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.109 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:35.110 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.111 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.112 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.112 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.113 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:35.114 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.115 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.116 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.117 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.118 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.119 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:35.119 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.120 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.121 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:35.121 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.122 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.123 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.124 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:35.125 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.125 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.126 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.127 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.128 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:35.128 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.129 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.130 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:35.131 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.132 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.132 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.133 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:35.134 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.135 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.135 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.136 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.137 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:35.138 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.138 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.139 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.140 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.141 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.141 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.143 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.144 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.144 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.145 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.146 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:35.147 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.148 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:35.148 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:35.149 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:35.150 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.151 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:11:20Z" 2022-09-13T19:12:35.152 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:35.153 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:35.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T19:12:35.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:35.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:35.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:35.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:25.238+0000 7f319537f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:25.239+0000 7f319537f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:35.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:35.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:35.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:35.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:35.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:35.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:35.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:35.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:35.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:35.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:35.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.182451_229.1", 2022-09-13T19:12:35.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T19:12:35.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:35.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:35.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:35.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:35.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:35.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:11:25Z" 2022-09-13T19:12:35.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:35.202 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:35.203 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:35.203 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:35.203 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:35.204 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.204 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.205 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.205 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:35.206 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.206 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:35.206 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.207 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.207 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.208 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.208 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:35.208 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.209 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.209 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.210 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.210 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.211 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.211 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:35.211 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.212 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.212 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:35.213 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.213 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.214 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.214 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:35.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.215 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.215 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.216 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.216 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:35.216 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.217 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.217 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.182451_229.1", 2022-09-13T19:12:35.218 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.218 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.219 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.219 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.182451Z", 2022-09-13T19:12:35.219 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.220 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.220 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.221 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.221 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:35.222 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.222 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.222 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.223 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.223 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.224 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.224 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.225 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.225 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.225 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.226 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:35.226 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.227 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:35.227 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:35.228 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:35.228 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.228 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:11:25Z" 2022-09-13T19:12:35.229 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:35.229 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.182451_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.182451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:35.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.182451_229.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.182451_229.1 2022-09-13T19:12:35.231 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.232 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:35.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:35.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:35.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:30.386+0000 7f8767004600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:30.387+0000 7f8767004600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:35.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:35.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:35.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:35.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:35.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:35.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:35.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:35.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:35.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:35.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:35.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:35.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:35.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:35.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:35.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:35.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:35.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:35.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:11:30Z" 2022-09-13T19:12:35.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:35.268 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:35.269 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:35.269 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:35.270 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:35.271 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.272 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.272 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.273 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:35.274 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.275 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:35.276 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.277 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.278 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.278 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.279 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:35.280 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.281 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.282 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.282 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.283 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.284 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.285 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:35.286 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.287 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.288 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:35.289 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.290 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.291 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.292 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:35.293 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.293 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.294 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.295 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.296 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:35.297 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.298 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.299 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:35.300 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.301 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.302 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.303 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:35.303 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.304 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.305 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.306 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.307 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:35.308 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.309 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.310 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.311 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.312 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.312 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.313 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.314 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.315 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.316 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.316 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:35.317 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.318 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:35.319 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:35.320 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:35.321 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.322 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:11:30Z" 2022-09-13T19:12:35.323 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:35.323 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:35.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T19:12:35.327 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:35.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:35.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:35.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:35.550+0000 7f0b49dd5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:35.551+0000 7f0b49dd5600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:35.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:35.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:35.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:35.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:35.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:35.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:35.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:35.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:35.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:35.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:35.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.468110_246.1", 2022-09-13T19:12:35.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.468110Z", 2022-09-13T19:12:35.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:35.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:35.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:35.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:35.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:35.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:11:35Z" 2022-09-13T19:12:35.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:35.366 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:35.367 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:35.367 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:35.368 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:35.368 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.369 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.369 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.369 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:35.372 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.372 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:35.373 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.373 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.375 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.375 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.377 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:35.378 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.379 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.380 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.381 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.382 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.385 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.386 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:35.387 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.388 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.389 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:35.389 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.390 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.391 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.391 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:35.392 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.392 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.392 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.393 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.394 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:35.394 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.395 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.395 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.468110_246.1", 2022-09-13T19:12:35.395 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.396 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.396 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.397 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.468110Z", 2022-09-13T19:12:35.397 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.398 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.398 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.399 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.399 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:35.400 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.400 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.401 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.401 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.401 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.402 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.402 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.403 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.403 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.403 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.404 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:35.404 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.405 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:35.405 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:35.406 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:35.406 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.408 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:11:35Z" 2022-09-13T19:12:35.408 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:35.409 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.468110_246.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.468110Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:35.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.468110_246.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.468110_246.1 2022-09-13T19:12:35.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:35.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:35.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:35.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:40.704+0000 7fdef4a10600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:40.705+0000 7fdef4a10600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:35.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:35.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:35.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:35.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:35.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:35.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:35.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:35.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:35.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:35.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:35.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:35.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:35.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:35.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:35.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:35.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:35.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:35.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:11:40Z" 2022-09-13T19:12:35.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:35.447 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:35.447 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:35.448 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:35.448 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:35.449 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.449 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.450 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.450 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:35.450 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.451 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:35.451 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.452 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.452 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.453 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:35.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.454 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.454 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.455 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.455 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.456 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.456 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:35.456 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.457 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.457 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:35.458 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.458 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.458 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.459 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:35.459 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.460 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.460 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.461 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.461 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:35.461 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.462 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.462 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.220845_233.1", 2022-09-13T19:12:35.463 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.463 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.464 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.220845Z", 2022-09-13T19:12:35.465 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.465 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.465 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.466 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.467 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:35.467 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.468 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.468 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.469 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.470 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.470 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.471 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.471 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.472 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.472 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.473 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:35.473 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.474 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:35.474 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:35.475 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:35.475 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.475 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:11:40Z" 2022-09-13T19:12:35.476 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:35.476 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.220845_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.220845Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:35.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.220845_233.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.220845_233.1 2022-09-13T19:12:35.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:35.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:35.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:35.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:45.910+0000 7f481fdb7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:45.911+0000 7f481fdb7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:35.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:35.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:35.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:35.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:35.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:35.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:35.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:35.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:35.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:35.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:35.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.189736_232.1", 2022-09-13T19:12:35.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.189736Z", 2022-09-13T19:12:35.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:35.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:35.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:35.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:35.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:35.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:11:45Z" 2022-09-13T19:12:35.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:35.514 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:35.515 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:35.515 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:35.516 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:35.516 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.517 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.517 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.518 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:35.518 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.519 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:35.519 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.519 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.520 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.520 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.521 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:35.521 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.522 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.522 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.523 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.523 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.523 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.524 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:35.524 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.525 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.526 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:35.526 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.526 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.527 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.527 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:35.528 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.528 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.528 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.529 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.529 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:35.530 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.530 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.530 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.189736_232.1", 2022-09-13T19:12:35.531 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.532 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.532 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.533 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.189736Z", 2022-09-13T19:12:35.533 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.533 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.534 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.534 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.535 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:35.535 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.536 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.536 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.537 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.537 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.538 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.538 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.539 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.539 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.540 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.540 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:35.541 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.541 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:35.541 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:35.542 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:35.543 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.543 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:11:45Z" 2022-09-13T19:12:35.544 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:35.544 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.189736_232.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.189736Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:35.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056091.338062_357.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.189736_232.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056091.338062_357.1 target=1_1663055099.189736_232.1 2022-09-13T19:12:35.547 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:35.548 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.548 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:35.549 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable 2022-09-13T19:12:35.549 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:35.552 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:35.552 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:35.553 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:35.553 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1139, in test_bucket_sync_disable 2022-09-13T19:12:35.554 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2022-09-13T19:12:35.555 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:35.555 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2022-09-13T19:12:35.556 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 196, in zone_meta_checkpoint 2022-09-13T19:12:35.556 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2022-09-13T19:12:35.557 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2022-09-13T19:12:35.557 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:35.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=yxvvuh-52 2022-09-13T19:12:35.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-52/ 2022-09-13T19:12:35.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-52/ 2022-09-13T19:12:35.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:35.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-52/ 2022-09-13T19:12:35.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:35.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:35.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:35.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:35.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:35.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:35.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:35.569 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:35.570 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.571 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.572 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:11:55 GMT 2022-09-13T19:12:35.573 INFO:tasks.rgw_multisite_tests:/yxvvuh-52/ 2022-09-13T19:12:35.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:35.576 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:LXBbGOU86tjSaNYkw7yk0O8luI8= 2022-09-13T19:12:35.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:55 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:LXBbGOU86tjSaNYkw7yk0O8luI8=', 'Content-Length': '0'} 2022-09-13T19:12:35.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000a120a50547e3f4e8-0063203b4b-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:35.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=yxvvuh-53 2022-09-13T19:12:35.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-53/ 2022-09-13T19:12:35.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-53/ 2022-09-13T19:12:35.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:35.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-53/ 2022-09-13T19:12:35.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:35.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:35.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi083.front.sepia.ceph.com:8000 2022-09-13T19:12:35.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:35.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:35.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:35.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:35.593 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:35.594 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.595 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.596 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:11:55 GMT 2022-09-13T19:12:35.598 INFO:tasks.rgw_multisite_tests:/yxvvuh-53/ 2022-09-13T19:12:35.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:35.599 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:jQ7zlkmtF1ZpzlO6qgtLc7XpZLU= 2022-09-13T19:12:35.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:55 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:jQ7zlkmtF1ZpzlO6qgtLc7XpZLU=', 'Content-Length': '0'} 2022-09-13T19:12:35.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000a5c68dba85ee1ea4-0063203b4b-1a1a-test-zone2'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:35.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=yxvvuh-54 2022-09-13T19:12:35.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-54/ 2022-09-13T19:12:35.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-54/ 2022-09-13T19:12:35.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:35.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-54/ 2022-09-13T19:12:35.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:35.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:35.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8001 2022-09-13T19:12:35.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2022-09-13T19:12:35.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:35.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:35.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:35.611 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:35.612 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.612 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.613 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:11:55 GMT 2022-09-13T19:12:35.614 INFO:tasks.rgw_multisite_tests:/yxvvuh-54/ 2022-09-13T19:12:35.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:35.615 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:UJ5nlOU47ikEkds9CAs0arW2AZE= 2022-09-13T19:12:35.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:11:55 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:UJ5nlOU47ikEkds9CAs0arW2AZE=', 'Content-Length': '0'} 2022-09-13T19:12:35.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002a86517a0685aa2e-0063203b4b-1450-test-zone3'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:35.618 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.619 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:35.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:35.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:35.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:35.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:35.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:35.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:35.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:35.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:35.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:35.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:35.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:35.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:35.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663056715.246326_454.1", 2022-09-13T19:12:35.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T08:11:55.246326Z" 2022-09-13T19:12:35.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:35.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:35.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:35.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:35.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:35.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:35.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:35.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:35.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:35.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:55.486+0000 7f8355435600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:11:55.488+0000 7f8355435600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:35.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:35.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:35.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:35.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:35.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:35.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:35.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:35.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:35.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:35.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:35.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:35.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:35.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:35.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:35.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:35.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:35.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:35.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:11:55Z" 2022-09-13T19:12:35.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:35.669 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:35.669 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:35.670 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:35.670 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:35.671 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.671 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.672 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.672 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:35.673 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.673 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:35.674 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.674 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.675 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.675 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.675 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:35.676 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.676 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.677 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.677 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.677 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.678 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.678 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:35.679 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.680 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.680 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:35.681 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.682 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.682 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.683 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:35.683 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.684 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.684 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.684 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.685 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:35.685 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.686 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.686 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:35.687 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.687 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.688 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.688 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:35.688 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.689 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.689 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.690 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.690 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:35.691 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.692 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.693 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.693 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.694 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.695 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.695 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.695 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.696 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.696 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.697 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:35.698 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.698 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:35.699 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:35.700 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:35.700 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.701 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:11:55Z" 2022-09-13T19:12:35.701 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:35.702 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:35.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:35.704 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: *** Caught signal (Segmentation fault) ** 2022-09-13T19:12:35.704 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: in thread 7fd5fbfb7700 thread_name:lua_background 2022-09-13T19:12:35.705 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: ceph version 17.0.0-14918-g840d0541 (840d05418c459144c92fb2a90d74a2ded29610da) quincy (dev) 2022-09-13T19:12:35.705 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fd735f91cf0] 2022-09-13T19:12:35.705 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 2: (RGWSI_SysObj::Obj::ROp::read(DoutPrefixProvider const*, long, long, ceph::buffer::v15_2_0::list*, optional_yield)+0x43) [0x55828ed55a33] 2022-09-13T19:12:35.706 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 3: (rgw_get_system_obj(RGWSI_SysObj*, rgw_pool const&, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list&, RGWObjVersionTracker*, std::chrono::time_point > >*, optional_yield, DoutPrefixProvider const*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, rgw_cache_entry_info*, boost::optional, bool)+0x1f7) [0x55828f23b807] 2022-09-13T19:12:35.706 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 4: (rgw::sal::RadosLuaManager::get_script(DoutPrefixProvider const*, optional_yield, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator >&)+0x1a5) [0x55828f225665] 2022-09-13T19:12:35.707 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 5: (rgw::lua::read_script(DoutPrefixProvider const*, rgw::sal::LuaManager*, std::__cxx11::basic_string, std::allocator > const&, optional_yield, rgw::lua::context, std::__cxx11::basic_string, std::allocator >&)+0x75) [0x55828f345d65] 2022-09-13T19:12:35.707 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 6: (rgw::lua::Background::read_script()+0x9f) [0x55828f35132f] 2022-09-13T19:12:35.708 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 7: (rgw::lua::Background::run()+0x6f6) [0x55828f352386] 2022-09-13T19:12:35.708 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 8: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fd73570aba3] 2022-09-13T19:12:35.708 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fd735f871ca] 2022-09-13T19:12:35.709 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 10: clone() 2022-09-13T19:12:35.709 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 2022-09-13T08:11:56.006+0000 7fd5fbfb7700 -1 *** Caught signal (Segmentation fault) ** 2022-09-13T19:12:35.710 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: in thread 7fd5fbfb7700 thread_name:lua_background 2022-09-13T19:12:35.710 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 2022-09-13T19:12:35.710 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: ceph version 17.0.0-14918-g840d0541 (840d05418c459144c92fb2a90d74a2ded29610da) quincy (dev) 2022-09-13T19:12:35.711 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fd735f91cf0] 2022-09-13T19:12:35.711 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 2: (RGWSI_SysObj::Obj::ROp::read(DoutPrefixProvider const*, long, long, ceph::buffer::v15_2_0::list*, optional_yield)+0x43) [0x55828ed55a33] 2022-09-13T19:12:35.712 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 3: (rgw_get_system_obj(RGWSI_SysObj*, rgw_pool const&, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list&, RGWObjVersionTracker*, std::chrono::time_point > >*, optional_yield, DoutPrefixProvider const*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, rgw_cache_entry_info*, boost::optional, bool)+0x1f7) [0x55828f23b807] 2022-09-13T19:12:35.712 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 4: (rgw::sal::RadosLuaManager::get_script(DoutPrefixProvider const*, optional_yield, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator >&)+0x1a5) [0x55828f225665] 2022-09-13T19:12:35.713 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 5: (rgw::lua::read_script(DoutPrefixProvider const*, rgw::sal::LuaManager*, std::__cxx11::basic_string, std::allocator > const&, optional_yield, rgw::lua::context, std::__cxx11::basic_string, std::allocator >&)+0x75) [0x55828f345d65] 2022-09-13T19:12:35.713 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 6: (rgw::lua::Background::read_script()+0x9f) [0x55828f35132f] 2022-09-13T19:12:35.713 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 7: (rgw::lua::Background::run()+0x6f6) [0x55828f352386] 2022-09-13T19:12:35.714 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 8: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fd73570aba3] 2022-09-13T19:12:35.714 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fd735f871ca] 2022-09-13T19:12:35.715 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 10: clone() 2022-09-13T19:12:35.715 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-13T19:12:35.716 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 2022-09-13T19:12:35.716 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 0> 2022-09-13T08:11:56.006+0000 7fd5fbfb7700 -1 *** Caught signal (Segmentation fault) ** 2022-09-13T19:12:35.716 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: in thread 7fd5fbfb7700 thread_name:lua_background 2022-09-13T19:12:35.717 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 2022-09-13T19:12:35.717 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: ceph version 17.0.0-14918-g840d0541 (840d05418c459144c92fb2a90d74a2ded29610da) quincy (dev) 2022-09-13T19:12:35.718 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fd735f91cf0] 2022-09-13T19:12:35.718 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 2: (RGWSI_SysObj::Obj::ROp::read(DoutPrefixProvider const*, long, long, ceph::buffer::v15_2_0::list*, optional_yield)+0x43) [0x55828ed55a33] 2022-09-13T19:12:35.718 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 3: (rgw_get_system_obj(RGWSI_SysObj*, rgw_pool const&, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list&, RGWObjVersionTracker*, std::chrono::time_point > >*, optional_yield, DoutPrefixProvider const*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, rgw_cache_entry_info*, boost::optional, bool)+0x1f7) [0x55828f23b807] 2022-09-13T19:12:35.719 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 4: (rgw::sal::RadosLuaManager::get_script(DoutPrefixProvider const*, optional_yield, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator >&)+0x1a5) [0x55828f225665] 2022-09-13T19:12:35.719 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 5: (rgw::lua::read_script(DoutPrefixProvider const*, rgw::sal::LuaManager*, std::__cxx11::basic_string, std::allocator > const&, optional_yield, rgw::lua::context, std::__cxx11::basic_string, std::allocator >&)+0x75) [0x55828f345d65] 2022-09-13T19:12:35.720 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 6: (rgw::lua::Background::read_script()+0x9f) [0x55828f35132f] 2022-09-13T19:12:35.720 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 7: (rgw::lua::Background::run()+0x6f6) [0x55828f352386] 2022-09-13T19:12:35.721 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 8: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fd73570aba3] 2022-09-13T19:12:35.721 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fd735f871ca] 2022-09-13T19:12:35.721 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 10: clone() 2022-09-13T19:12:35.722 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-13T19:12:35.722 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 2022-09-13T19:12:35.723 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 0> 2022-09-13T08:11:56.006+0000 7fd5fbfb7700 -1 *** Caught signal (Segmentation fault) ** 2022-09-13T19:12:35.723 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: in thread 7fd5fbfb7700 thread_name:lua_background 2022-09-13T19:12:35.723 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 2022-09-13T19:12:35.724 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: ceph version 17.0.0-14918-g840d0541 (840d05418c459144c92fb2a90d74a2ded29610da) quincy (dev) 2022-09-13T19:12:35.724 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fd735f91cf0] 2022-09-13T19:12:35.725 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 2: (RGWSI_SysObj::Obj::ROp::read(DoutPrefixProvider const*, long, long, ceph::buffer::v15_2_0::list*, optional_yield)+0x43) [0x55828ed55a33] 2022-09-13T19:12:35.725 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 3: (rgw_get_system_obj(RGWSI_SysObj*, rgw_pool const&, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list&, RGWObjVersionTracker*, std::chrono::time_point > >*, optional_yield, DoutPrefixProvider const*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, rgw_cache_entry_info*, boost::optional, bool)+0x1f7) [0x55828f23b807] 2022-09-13T19:12:35.726 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 4: (rgw::sal::RadosLuaManager::get_script(DoutPrefixProvider const*, optional_yield, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator >&)+0x1a5) [0x55828f225665] 2022-09-13T19:12:35.726 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 5: (rgw::lua::read_script(DoutPrefixProvider const*, rgw::sal::LuaManager*, std::__cxx11::basic_string, std::allocator > const&, optional_yield, rgw::lua::context, std::__cxx11::basic_string, std::allocator >&)+0x75) [0x55828f345d65] 2022-09-13T19:12:35.726 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 6: (rgw::lua::Background::read_script()+0x9f) [0x55828f35132f] 2022-09-13T19:12:35.727 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 7: (rgw::lua::Background::run()+0x6f6) [0x55828f352386] 2022-09-13T19:12:35.727 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 8: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fd73570aba3] 2022-09-13T19:12:35.728 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fd735f871ca] 2022-09-13T19:12:35.728 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 10: clone() 2022-09-13T19:12:35.729 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-13T19:12:35.729 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stdout: INFO: 2022-09-13T19:12:35.729 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi083.stderr: INFO: daemon-helper: command crashed with signal 11 2022-09-13T19:12:35.730 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~0s 2022-09-13T19:12:35.730 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.731 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:35.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:35.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:35.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:00.636+0000 7fecc5f7e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:00.637+0000 7fecc5f7e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:35.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:35.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:35.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:35.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:35.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:35.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:35.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:35.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:35.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:35.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:35.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:35.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:35.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:35.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:35.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:35.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:35.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:35.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:12:00Z" 2022-09-13T19:12:35.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:35.761 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:35.761 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:35.762 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:35.762 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:35.763 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.763 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.764 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.764 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:35.764 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.765 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:35.765 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.766 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.766 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.767 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.767 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:35.767 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.768 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.768 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.769 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.769 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.769 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.770 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:35.770 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.771 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.771 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:35.772 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.772 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.772 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.773 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:35.773 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.774 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.774 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.775 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.775 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:35.776 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.776 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.777 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:35.777 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.777 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.778 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.778 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:35.779 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.779 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.780 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.780 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.780 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:35.781 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.781 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.782 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.782 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.782 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.783 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.783 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.784 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.784 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.785 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.786 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:35.786 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.787 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:35.787 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:35.788 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:35.788 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.789 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:12:00Z" 2022-09-13T19:12:35.789 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:35.790 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:35.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:35.792 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~6s 2022-09-13T19:12:35.793 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.793 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:35.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:35.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:35.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:05.776+0000 7f165ea73600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:05.778+0000 7f165ea73600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:35.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:35.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:35.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:35.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:35.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:35.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:35.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:35.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:35.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:35.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:35.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:35.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:35.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:35.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:35.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:35.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:35.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:35.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:12:05Z" 2022-09-13T19:12:35.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:35.826 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:35.827 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:35.827 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:35.828 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:35.828 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.829 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.829 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.829 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:35.830 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.830 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:35.831 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.831 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.832 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.833 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:35.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.834 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.834 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.834 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.835 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.835 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.836 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:35.836 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.836 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.837 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:35.837 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.838 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.838 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.839 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:35.839 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.840 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.840 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.840 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.841 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:35.841 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.842 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.842 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:35.843 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.843 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.843 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.844 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:35.844 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.845 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.845 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.846 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.846 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:35.847 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.847 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.847 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.848 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.848 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.849 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.849 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.850 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.850 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.851 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.851 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:35.851 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.852 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:35.852 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:35.853 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:35.853 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.854 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:12:05Z" 2022-09-13T19:12:35.854 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:35.854 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:35.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:35.857 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~11s 2022-09-13T19:12:35.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.858 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:35.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:35.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:35.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:10.930+0000 7f20c3eaf600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:10.931+0000 7f20c3eaf600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:35.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:35.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:35.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:35.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:35.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:35.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:35.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:35.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:35.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:35.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:35.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:35.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:35.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:35.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:35.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:35.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:35.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:35.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:12:10Z" 2022-09-13T19:12:35.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:35.892 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:35.892 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:35.893 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:35.893 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:35.893 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.894 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.894 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.895 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:35.896 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.896 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:35.896 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.897 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.898 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.898 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.899 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:35.899 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.900 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.900 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.901 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.901 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.901 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.902 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:35.902 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.903 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.903 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:35.904 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.904 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.905 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:35.905 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.906 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.906 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.907 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.907 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:35.908 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.908 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.908 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:35.909 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.909 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.910 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.910 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:35.911 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.911 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.911 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.912 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.912 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:35.913 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.913 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.914 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.914 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.915 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.915 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.915 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.916 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.916 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.917 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.917 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:35.918 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.918 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:35.919 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:35.919 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:35.919 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.920 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:12:10Z" 2022-09-13T19:12:35.920 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:35.921 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:35.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:35.923 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.923 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:35.924 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~17s 2022-09-13T19:12:35.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:35.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:35.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:16.070+0000 7f5d1600e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:16.071+0000 7f5d1600e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:35.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:35.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:35.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:35.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:35.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:35.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:35.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:35.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:35.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:35.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:35.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:35.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:35.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:35.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:35.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:35.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:35.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:35.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:35.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:12:16Z" 2022-09-13T19:12:35.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:35.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:35.959 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:35.959 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:35.960 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:35.960 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:35.960 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.961 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.961 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.962 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:35.962 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.963 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:35.963 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.963 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.964 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.964 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.965 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:35.965 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.965 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:35.966 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.966 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.967 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.967 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.968 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:35.968 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.969 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.969 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:35.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.970 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.970 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.971 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:35.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.972 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.972 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.973 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.973 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:35.974 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.974 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.974 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:35.975 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.975 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.976 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.976 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:35.976 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.977 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.977 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.978 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:35.978 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:35.979 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:35.979 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:35.979 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:35.980 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:35.980 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:35.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:35.981 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:35.982 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:35.982 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.983 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:35.983 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:35.984 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.984 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:35.985 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:35.985 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:35.986 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:35.986 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:12:16Z" 2022-09-13T19:12:35.987 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:35.987 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:35.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:35.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:35.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:35.989 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.990 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:35.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:35.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:35.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:35.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:21.211+0000 7f787ea8f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:21.212+0000 7f787ea8f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:35.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:35.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:35.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:35.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:35.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:35.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:35.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:35.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:35.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:35.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:35.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:35.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:35.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:35.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:35.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:35.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:35.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:36.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:36.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:36.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:36.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:36.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:36.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:36.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:36.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:36.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:12:21Z" 2022-09-13T19:12:36.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:36.022 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:36.023 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:36.023 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:36.024 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:36.024 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.025 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.025 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:36.026 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.026 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:36.027 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.027 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.027 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.028 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.028 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:36.029 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.029 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.030 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.030 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.030 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.031 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.032 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:36.033 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.033 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.034 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:36.034 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.035 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.036 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:36.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.037 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.037 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.038 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.038 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:36.039 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.039 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.039 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.040 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.040 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.041 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.041 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.042 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.042 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.043 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.043 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.044 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:36.044 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.044 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.045 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.045 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.046 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.046 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.047 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.048 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.048 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.049 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:36.049 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.050 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:36.051 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:36.051 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:36.052 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.052 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:12:21Z" 2022-09-13T19:12:36.052 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:36.053 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:36.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:36.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:36.055 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~22s 2022-09-13T19:12:36.056 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.056 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:36.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:36.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:36.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:26.343+0000 7f0780951600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:26.344+0000 7f0780951600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:36.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:36.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:36.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:36.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:36.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:36.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:36.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:36.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:36.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:36.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:36.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:36.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:36.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:36.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:36.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:36.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:12:26Z" 2022-09-13T19:12:36.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:36.106 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:36.107 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:36.108 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:36.109 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:36.110 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.110 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.111 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.112 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:36.113 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.114 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:36.115 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.116 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.117 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.118 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.119 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:36.120 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.121 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.122 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.122 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.123 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.124 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.125 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:36.126 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.128 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.128 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:36.130 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.131 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.132 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.133 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:36.134 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.134 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.135 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.136 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.137 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:36.138 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.139 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.140 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.141 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.142 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.143 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.144 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.144 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.145 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.146 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.147 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.148 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:36.149 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.150 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.150 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.151 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.152 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.153 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.154 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.154 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.155 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.156 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.157 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:36.158 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.158 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:36.159 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:36.160 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:36.161 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.162 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:12:26Z" 2022-09-13T19:12:36.163 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:36.164 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:36.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:36.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:36.169 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~28s 2022-09-13T19:12:36.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.170 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:36.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:36.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:36.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:31.503+0000 7f187ad4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:31.504+0000 7f187ad4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:36.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:36.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:36.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:36.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:36.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:36.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:36.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:36.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:36.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:36.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:36.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:36.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:36.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:36.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:36.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:36.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:12:31Z" 2022-09-13T19:12:36.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:36.239 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:36.239 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:36.240 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:36.240 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:36.241 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.241 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.242 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.242 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:36.243 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.243 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:36.244 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.244 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.245 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.245 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.245 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:36.246 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.246 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.247 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.247 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.248 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.248 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.249 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:36.249 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.250 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.250 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:36.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.251 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.251 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.252 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:36.253 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.253 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.254 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.254 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.255 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:36.255 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.256 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.256 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.256 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.261 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.261 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.262 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.262 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.264 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.272 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.272 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:36.274 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.274 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.275 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.275 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.276 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.276 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.276 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.277 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.277 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.278 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.278 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:36.279 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.279 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:36.280 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:36.280 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:36.280 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.281 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:12:31Z" 2022-09-13T19:12:36.281 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:36.282 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:36.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:36.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:36.284 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~34s 2022-09-13T19:12:36.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:36.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:36.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:36.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:36.637+0000 7f65c9a14600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:36.638+0000 7f65c9a14600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:36.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:36.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:36.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:36.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:36.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:36.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:36.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:36.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:36.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:36.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:36.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:36.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:36.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:36.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:36.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:36.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:12:36Z" 2022-09-13T19:12:36.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:36.318 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:36.319 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:36.319 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:36.319 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:36.320 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.321 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.321 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:36.322 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.322 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:36.323 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.323 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.324 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.324 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.325 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:36.325 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.325 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.326 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.326 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.327 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.327 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.328 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:36.328 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.328 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.329 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:36.329 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.330 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.330 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.331 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:36.331 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.332 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.332 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.333 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.333 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:36.337 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.337 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.338 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.338 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.339 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.339 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.340 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.340 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.340 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.341 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.341 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.342 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:36.342 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.343 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.344 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.344 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.344 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.345 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.345 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.346 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.346 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.347 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.347 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:36.348 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.348 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:36.349 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:36.349 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:36.350 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.350 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:12:36Z" 2022-09-13T19:12:36.350 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:36.351 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:36.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:36.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:36.353 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~39s 2022-09-13T19:12:36.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:36.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:36.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:36.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:41.785+0000 7f9e35ced600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:41.786+0000 7f9e35ced600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:36.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:36.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:36.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:36.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:36.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:36.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:36.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:36.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:36.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:36.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:36.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:36.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:36.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:36.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:36.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:36.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:12:41Z" 2022-09-13T19:12:36.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:36.422 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:36.423 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:36.423 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:36.424 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:36.424 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.424 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.425 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.425 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:36.426 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.426 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:36.427 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.428 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.428 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.429 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.429 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:36.432 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.433 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.433 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.434 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.434 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.435 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.435 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:36.436 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.436 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.436 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:36.437 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.437 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.438 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.438 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:36.439 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.439 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.440 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.440 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.441 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:36.441 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.442 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.442 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.443 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.443 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.444 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.445 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.446 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.447 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.447 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:36.448 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.448 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.449 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.449 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.450 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.450 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.454 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.455 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.456 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.456 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.457 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:36.457 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.458 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:36.458 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:36.459 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:36.459 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.459 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:12:41Z" 2022-09-13T19:12:36.460 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:36.460 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:36.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:36.464 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:36.467 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~45s 2022-09-13T19:12:36.467 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:36.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:36.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:36.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:46.940+0000 7fbb82081600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:46.941+0000 7fbb82081600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:36.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:36.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:36.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:36.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:36.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:36.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:36.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:36.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:36.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:36.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:36.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:36.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:36.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:36.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:36.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:36.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:12:46Z" 2022-09-13T19:12:36.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.530 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:36.530 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:36.530 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:36.531 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:36.532 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:36.532 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.533 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.533 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.533 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:36.534 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.534 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:36.535 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.536 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.536 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.537 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.537 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:36.537 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.538 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.538 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.539 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.539 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.540 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.540 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:36.541 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.541 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.541 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:36.542 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.543 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.543 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.544 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:36.544 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.545 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.545 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.546 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.546 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:36.546 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.547 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.547 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.548 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.548 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.549 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.549 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.550 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.550 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.551 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.551 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.551 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:36.552 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.552 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.553 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.553 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.554 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.554 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.555 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.556 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.556 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.557 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:36.557 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.558 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:36.558 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:36.558 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:36.559 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.559 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:12:46Z" 2022-09-13T19:12:36.560 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:36.560 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:36.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:36.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:36.562 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~50s 2022-09-13T19:12:36.563 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.563 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:36.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:36.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:36.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:52.094+0000 7f45eca84600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:52.095+0000 7f45eca84600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:36.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:36.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:36.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:36.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:36.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:36.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:36.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:36.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:36.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:36.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:36.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:36.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:36.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:36.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:36.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:36.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:12:52Z" 2022-09-13T19:12:36.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:36.593 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:36.594 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:36.594 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:36.595 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:36.595 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.596 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.596 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.597 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:36.597 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.597 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:36.598 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.598 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.599 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.599 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.600 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:36.600 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.600 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.601 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.601 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.602 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.602 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.602 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:36.603 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.603 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.604 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:36.604 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.605 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.606 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.607 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:36.607 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.607 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.608 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.608 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.609 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:36.609 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.610 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.610 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.611 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.611 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.612 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.612 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.613 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.613 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.614 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.614 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.615 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:36.615 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.616 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.618 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.618 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.619 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.620 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.620 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.620 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.621 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.622 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.622 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:36.623 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.623 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:36.623 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:36.624 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:36.624 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.625 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:12:52Z" 2022-09-13T19:12:36.626 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:36.626 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:36.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:36.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:36.629 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~56s 2022-09-13T19:12:36.629 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:36.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:36.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:36.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:57.251+0000 7f9c1bcb3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:12:57.252+0000 7f9c1bcb3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:36.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:36.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:36.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:36.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:36.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:36.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:36.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:36.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:36.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:36.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:36.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:36.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:36.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:36.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:36.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:36.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:12:57Z" 2022-09-13T19:12:36.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:36.673 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:36.674 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:36.674 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:36.674 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:36.675 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.675 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.676 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.676 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:36.676 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.677 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:36.677 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.678 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.678 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.678 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.679 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:36.679 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.680 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.680 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.681 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.681 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.681 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.682 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:36.682 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.683 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.683 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:36.683 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.684 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.684 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.685 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:36.685 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.685 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.686 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.686 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.687 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:36.687 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.688 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.688 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.688 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.689 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.689 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.690 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.690 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.690 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.691 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.691 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.692 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:36.692 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.692 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.693 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.693 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.694 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.694 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.695 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.695 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.695 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.696 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.696 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:36.697 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.697 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:36.697 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:36.698 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:36.699 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.699 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:12:57Z" 2022-09-13T19:12:36.699 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:36.700 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:36.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:36.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:36.702 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~62s 2022-09-13T19:12:36.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.703 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:36.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:36.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:36.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:02.394+0000 7f06dd355600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:02.395+0000 7f06dd355600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:36.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:36.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:36.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:36.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:36.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:36.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:36.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:36.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:36.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:36.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:36.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:36.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:36.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:36.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:36.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:36.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:13:02Z" 2022-09-13T19:12:36.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:36.733 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:36.733 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:36.733 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:36.734 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:36.734 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.735 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.735 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.735 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:36.736 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.736 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:36.737 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.737 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.738 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.738 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.738 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:36.739 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.739 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.740 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.740 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.740 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.741 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.741 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:36.742 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.742 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.743 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:36.743 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.743 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.744 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.744 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:36.745 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.745 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.746 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.746 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.746 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:36.747 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.747 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.748 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.748 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.749 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.749 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.749 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.750 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.750 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.751 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.751 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.751 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:36.752 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.752 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.753 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.753 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.754 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.754 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.755 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.755 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.755 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.756 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.756 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:36.757 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.757 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:36.758 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:36.759 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:36.760 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.760 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:13:02Z" 2022-09-13T19:12:36.760 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:36.761 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:36.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:36.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:36.763 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~67s 2022-09-13T19:12:36.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:36.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:36.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:36.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:07.557+0000 7f3380a45600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:07.558+0000 7f3380a45600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:36.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:36.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:36.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:36.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:36.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:36.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:36.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:36.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:36.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:36.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:36.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:36.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:36.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:36.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:13:07Z" 2022-09-13T19:12:36.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:36.811 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:36.811 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:36.812 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:36.812 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:36.813 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.814 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.814 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.815 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:36.816 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.816 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:36.817 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.818 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.819 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.819 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.820 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:36.821 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.821 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.822 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.823 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.823 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.823 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.824 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:36.824 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.825 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.825 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:36.826 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.826 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.827 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.827 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:36.827 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.829 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.830 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.830 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.831 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:36.832 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.833 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.833 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.834 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.835 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.836 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.837 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.837 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.838 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.839 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.840 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.841 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:36.842 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.842 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.843 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.844 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.844 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.845 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.846 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.847 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.847 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.848 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:36.849 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.850 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:36.850 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:36.851 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:36.851 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.852 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:13:07Z" 2022-09-13T19:12:36.852 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:36.853 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:36.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:36.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:36.857 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~73s 2022-09-13T19:12:36.858 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:36.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:36.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:36.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:12.696+0000 7f3a60a31600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:12.697+0000 7f3a60a31600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:36.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:36.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:36.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:36.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:36.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:36.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:36.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:36.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:36.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:36.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:36.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:36.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:36.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:36.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:36.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:36.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:13:12Z" 2022-09-13T19:12:36.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:36.913 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:36.914 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:36.914 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:36.915 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:36.915 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.916 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.916 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.917 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:36.917 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.918 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:36.918 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.919 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.919 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.920 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:36.920 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.921 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.921 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.922 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.922 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.923 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.923 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:36.923 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.924 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.924 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:36.925 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.925 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.926 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.926 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:36.927 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.927 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.928 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.928 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.929 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:36.929 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.929 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.930 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.930 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.931 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.931 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.932 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.932 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.933 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.933 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.933 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.934 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:36.935 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.935 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.936 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.936 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.936 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.937 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.937 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.938 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.939 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.939 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.940 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:36.940 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.941 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:36.941 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:36.942 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:36.942 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.943 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:13:12Z" 2022-09-13T19:12:36.943 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:36.944 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:36.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:36.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:36.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:36.947 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~78s 2022-09-13T19:12:36.947 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.948 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:36.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:36.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:36.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:36.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:17.846+0000 7f765dbf6600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:17.847+0000 7f765dbf6600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:36.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:36.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:36.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:36.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:36.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:36.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:36.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:36.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:36.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:36.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:36.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:36.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:36.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:36.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:36.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:36.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:36.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:36.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:36.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:36.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:36.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:36.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:36.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:36.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:36.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:36.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:36.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:36.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:36.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:13:17Z" 2022-09-13T19:12:36.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:36.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:36.987 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:36.987 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:36.987 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:36.988 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:36.988 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:36.989 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.989 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.990 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:36.990 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.991 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:36.991 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.992 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.992 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:36.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.993 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:36.993 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.994 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:36.994 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:36.995 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:36.995 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:36.996 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:36.996 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:36.996 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:36.997 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:36.997 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:36.998 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:36.998 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:36.998 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:36.999 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:36.999 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.000 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.000 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.000 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.001 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.001 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.002 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.002 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.003 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.003 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.004 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.004 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.005 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.005 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.005 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.006 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:37.006 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.007 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.007 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.007 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.008 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.008 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.009 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.009 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.010 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.010 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:37.011 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.011 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:37.011 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:37.012 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:37.013 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.013 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:13:17Z" 2022-09-13T19:12:37.014 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:37.014 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:37.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:37.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:37.016 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.017 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:37.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:37.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:37.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:22.988+0000 7fe0defb3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:22.989+0000 7fe0defb3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:37.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:37.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:37.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:37.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:37.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:37.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:37.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:37.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:37.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:37.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:37.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:37.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:37.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:37.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:37.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:37.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:13:22Z" 2022-09-13T19:12:37.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:37.065 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:37.065 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:37.066 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:37.066 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:37.067 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.067 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.067 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.068 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:37.068 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.069 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:37.069 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.069 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.070 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.071 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.071 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:37.072 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.072 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.073 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.073 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.074 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.074 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.075 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:37.075 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.076 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.076 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:37.076 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.077 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.077 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.078 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:37.078 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.078 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.079 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.080 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.080 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.081 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.082 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.083 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.083 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.084 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.085 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.086 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.087 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.087 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.088 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.089 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.090 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:37.090 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.091 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.092 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.092 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.093 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.094 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.095 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.095 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.096 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.097 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.098 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:37.098 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.099 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:37.100 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:37.101 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:37.101 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.102 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:13:22Z" 2022-09-13T19:12:37.103 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:37.104 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:37.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:37.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:37.107 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~84s 2022-09-13T19:12:37.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:37.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:37.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:37.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:28.147+0000 7f3608167600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:28.148+0000 7f3608167600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:37.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:37.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:37.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:37.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:37.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:37.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:37.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:37.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:37.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:37.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:37.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:37.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:37.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:37.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:37.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:37.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:13:28Z" 2022-09-13T19:12:37.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:37.141 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:37.141 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:37.142 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:37.142 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:37.143 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.143 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.143 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.144 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:37.144 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.145 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:37.145 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.145 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.146 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.146 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.147 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:37.147 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.147 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.148 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.148 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.149 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.149 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.150 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:37.150 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.150 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.151 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:37.151 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.152 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.152 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.152 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:37.153 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.153 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.154 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.154 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.154 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.155 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.155 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.156 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.156 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.156 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.157 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.157 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.158 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.158 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.159 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.159 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.159 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:37.160 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.160 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.161 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.161 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.161 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.162 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.162 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.163 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.163 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.163 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.164 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:37.164 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.165 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:37.165 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:37.165 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:37.166 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.166 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:13:28Z" 2022-09-13T19:12:37.167 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:37.167 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:37.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:37.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:37.169 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~90s 2022-09-13T19:12:37.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.170 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:37.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:37.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:37.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:33.283+0000 7fd3b91df600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:33.284+0000 7fd3b91df600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:37.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:37.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:37.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:37.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:37.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:37.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:37.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:37.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:37.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:37.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:37.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:37.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:37.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:37.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:37.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:37.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:13:33Z" 2022-09-13T19:12:37.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:37.200 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:37.200 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:37.201 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:37.201 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:37.201 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.202 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.202 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.203 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:37.203 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.203 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:37.204 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.204 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.205 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.205 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.205 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:37.206 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.206 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.207 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.207 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.208 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.208 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.209 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:37.209 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.209 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.210 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:37.210 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.211 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.211 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.211 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:37.212 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.212 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.213 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.213 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.213 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.214 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.214 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.215 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.215 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.215 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.216 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.217 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.218 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.218 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.218 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:37.219 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.219 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.220 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.220 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.220 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.221 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.221 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.222 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.222 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.222 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.223 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:37.223 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.224 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:37.224 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:37.224 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:37.225 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.225 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:13:33Z" 2022-09-13T19:12:37.226 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:37.226 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:37.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:37.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:37.228 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~95s 2022-09-13T19:12:37.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:37.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:37.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:37.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:38.418+0000 7fa7b167b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:38.419+0000 7fa7b167b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:37.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:37.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:37.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:37.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:37.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:37.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:37.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:37.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:37.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:37.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:37.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:37.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:37.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:37.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:37.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:37.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:13:38Z" 2022-09-13T19:12:37.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:37.259 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:37.259 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:37.260 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:37.261 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:37.261 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.262 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.262 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:37.263 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.263 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:37.263 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.264 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.265 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.265 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.265 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:37.266 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.266 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.267 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.267 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.267 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.268 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.268 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:37.269 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.269 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.269 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:37.270 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.270 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.271 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.271 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:37.272 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.272 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.272 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.273 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.273 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.274 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.274 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.274 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.275 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.275 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.276 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.276 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.276 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.277 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.277 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.278 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.278 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:37.278 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.279 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.279 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.280 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.280 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.281 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.281 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.282 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.282 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.283 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:37.283 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.283 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:37.284 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:37.284 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:37.285 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.285 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:13:38Z" 2022-09-13T19:12:37.285 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:37.286 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:37.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:37.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:37.288 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~101s 2022-09-13T19:12:37.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.289 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:37.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:37.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:37.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:43.574+0000 7f3b27702600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:43.575+0000 7f3b27702600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:37.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:37.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:37.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:37.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:37.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:37.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:37.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:37.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:37.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:37.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:37.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:37.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:37.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:37.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:37.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:37.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:13:43Z" 2022-09-13T19:12:37.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:37.320 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:37.321 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:37.321 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:37.322 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:37.322 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.323 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.323 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.324 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:37.324 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.325 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:37.325 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.326 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.326 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.327 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.327 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:37.327 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.328 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.328 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.329 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.329 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.330 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.330 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:37.331 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.331 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.332 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:37.332 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.333 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.333 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.334 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:37.334 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.335 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.335 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.335 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.336 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.336 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.337 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.337 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.338 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.338 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.339 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.339 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.340 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.340 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.341 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.341 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.342 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:37.342 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.343 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.343 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.343 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.344 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.344 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.345 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.345 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.346 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.346 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.347 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:37.347 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.348 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:37.348 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:37.349 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:37.349 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.350 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:13:43Z" 2022-09-13T19:12:37.350 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:37.351 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:37.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:37.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:37.353 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~107s 2022-09-13T19:12:37.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:37.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:37.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:37.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:48.726+0000 7f4f24e2f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:48.727+0000 7f4f24e2f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:37.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:37.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:37.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:37.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:37.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:37.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:37.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:37.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:37.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:37.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:37.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:37.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:37.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:37.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:37.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:37.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:13:48Z" 2022-09-13T19:12:37.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:37.388 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:37.389 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:37.389 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:37.389 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:37.390 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.390 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.391 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.391 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:37.392 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.392 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:37.392 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.393 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.393 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.394 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.394 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:37.395 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.395 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.396 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.396 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.396 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.397 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.397 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:37.398 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.398 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.398 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:37.399 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.399 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.401 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.401 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:37.401 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.402 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.403 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.404 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.404 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.404 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.405 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.405 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.406 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.407 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.407 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.407 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.408 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.408 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.409 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.409 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:37.410 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.410 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.410 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.411 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.411 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.412 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.413 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.413 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.413 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.414 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.414 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:37.415 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.415 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:37.416 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:37.416 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:37.416 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.417 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:13:48Z" 2022-09-13T19:12:37.417 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:37.418 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:37.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:37.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:37.420 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~112s 2022-09-13T19:12:37.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.421 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:37.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:37.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:37.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:53.872+0000 7fe690cb3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:53.873+0000 7fe690cb3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:37.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:37.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:37.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:37.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:37.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:37.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:37.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:37.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:37.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:37.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:37.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:37.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:37.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:37.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:37.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:37.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:13:53Z" 2022-09-13T19:12:37.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:37.453 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:37.454 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:37.454 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:37.455 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:37.455 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.456 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.456 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.457 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:37.457 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.458 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:37.458 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.459 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.459 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.460 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:37.460 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.461 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.461 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.462 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.462 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.462 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.463 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:37.463 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.464 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.464 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:37.465 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.465 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.466 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.466 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:37.466 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.467 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.467 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.468 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.468 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.469 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.469 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.470 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.470 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.471 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.471 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.471 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.472 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.472 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.473 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.473 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.474 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:37.474 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.474 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.475 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.475 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.476 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.476 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.477 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.477 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.477 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.478 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.478 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:37.479 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.479 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:37.480 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:37.480 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:37.481 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.481 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:13:53Z" 2022-09-13T19:12:37.481 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:37.482 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:37.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:37.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:37.484 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~118s 2022-09-13T19:12:37.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:37.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:37.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:37.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:59.018+0000 7f8da7f43600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:13:59.019+0000 7f8da7f43600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:37.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:37.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:37.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:37.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:37.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:37.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:37.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:37.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:37.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:37.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:37.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:37.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:37.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:37.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:37.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:37.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:13:59Z" 2022-09-13T19:12:37.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:37.518 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:37.519 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:37.519 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:37.520 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:37.520 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.521 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.521 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.522 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:37.522 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.522 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:37.523 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.523 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.524 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.524 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.525 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:37.525 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.525 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.526 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.526 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.527 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.527 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.528 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:37.528 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.529 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.529 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:37.529 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.530 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.530 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.531 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:37.531 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.532 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.532 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.532 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.533 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.533 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.534 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.534 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.535 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.536 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.536 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.537 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.537 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.538 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.538 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:37.539 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.539 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.539 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.540 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.540 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.541 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.541 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.542 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.542 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.543 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.543 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:37.543 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.544 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:37.544 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:37.545 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:37.545 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.546 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:13:59Z" 2022-09-13T19:12:37.546 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:37.546 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:37.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:37.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:37.549 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~123s 2022-09-13T19:12:37.549 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.549 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:37.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:37.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:37.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:04.152+0000 7f5e2a85c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:04.153+0000 7f5e2a85c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:37.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:37.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:37.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:37.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:37.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:37.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:37.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:37.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:37.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:37.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:37.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:37.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:37.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:37.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:37.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:37.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:14:04Z" 2022-09-13T19:12:37.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:37.583 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:37.583 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:37.584 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:37.584 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:37.584 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.585 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.585 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.586 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:37.586 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.587 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:37.587 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.588 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.588 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.589 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.589 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:37.590 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.590 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.590 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.591 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.591 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.592 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.592 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:37.593 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.593 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.594 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:37.594 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.595 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.595 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.595 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:37.596 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.596 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.597 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.597 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.598 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.598 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.598 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.599 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.599 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.600 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.600 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.601 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.601 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.602 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.602 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.602 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.603 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:37.603 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.604 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.604 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.605 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.605 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.606 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.606 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.606 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.607 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.607 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.608 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:37.608 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.609 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:37.609 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:37.610 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:37.610 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.611 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:14:04Z" 2022-09-13T19:12:37.611 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:37.612 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:37.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:37.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:37.614 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~129s 2022-09-13T19:12:37.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:37.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:37.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:37.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:09.318+0000 7ff8f6e46600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:09.319+0000 7ff8f6e46600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:37.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:37.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:37.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:37.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:37.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:37.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:37.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:37.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:37.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:37.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:37.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:37.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:37.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:37.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:37.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:37.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:14:09Z" 2022-09-13T19:12:37.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:37.646 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:37.647 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:37.647 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:37.647 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:37.648 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.648 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.649 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.649 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:37.649 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.650 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:37.650 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.651 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.651 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.652 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.652 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:37.653 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.653 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.654 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.654 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.655 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.655 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:37.656 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.656 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.656 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:37.657 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.657 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.658 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.658 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:37.659 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.659 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.659 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.660 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.660 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.661 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.661 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.662 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.662 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.663 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.663 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.663 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.664 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.664 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.665 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.665 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.666 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:37.666 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.666 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.667 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.667 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.668 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.668 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.669 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.669 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.670 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.670 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.670 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:37.671 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.671 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:37.672 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:37.672 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:37.673 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.673 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:14:09Z" 2022-09-13T19:12:37.673 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:37.674 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:37.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:37.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:37.676 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~135s 2022-09-13T19:12:37.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:37.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:37.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:37.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:14.455+0000 7f5263f92600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:14.456+0000 7f5263f92600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:37.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:37.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:37.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:37.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:37.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:37.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:37.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:37.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:37.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:37.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:37.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:37.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:37.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:37.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:37.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:37.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:14:14Z" 2022-09-13T19:12:37.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:37.709 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:37.709 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:37.710 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:37.710 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:37.711 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.711 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.711 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.712 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:37.712 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.713 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:37.713 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.714 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.714 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.715 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.715 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:37.716 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.716 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.716 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.717 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.717 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.718 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.718 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:37.719 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.719 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.720 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:37.720 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.721 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.721 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.721 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:37.722 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.722 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.723 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.723 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.724 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.724 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.725 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.725 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.726 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.726 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.727 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.727 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.727 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.728 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.728 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.729 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.729 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:37.730 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.730 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.731 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.731 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.731 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.732 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.732 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.733 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.733 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.734 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.734 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:37.734 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.735 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:37.735 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:37.736 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:37.736 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.737 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:14:14Z" 2022-09-13T19:12:37.737 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:37.738 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:37.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:37.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:37.740 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~140s 2022-09-13T19:12:37.740 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.741 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:37.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:37.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:37.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:19.604+0000 7fd6d1c99600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:19.606+0000 7fd6d1c99600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:37.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:37.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:37.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:37.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:37.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:37.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:37.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:37.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:37.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:37.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:37.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:37.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:37.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:37.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:37.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:37.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:14:19Z" 2022-09-13T19:12:37.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:37.774 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:37.774 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:37.775 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:37.775 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:37.776 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.776 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.776 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.778 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:37.778 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.778 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:37.779 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.779 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.780 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.780 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.781 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:37.781 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.782 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.782 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.782 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.783 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.783 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.784 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:37.784 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.785 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.785 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:37.785 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.786 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.786 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.787 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:37.787 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.788 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.788 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.789 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.789 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.790 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.790 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.791 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.791 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.792 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.793 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.793 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.794 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.794 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.795 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.795 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.796 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:37.796 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.797 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.797 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.798 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.799 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.799 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.800 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.800 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.801 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.801 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:37.801 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.802 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:37.802 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:37.803 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:37.803 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.804 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:14:19Z" 2022-09-13T19:12:37.804 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:37.804 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:37.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:37.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:37.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:37.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:37.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:37.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:24.756+0000 7f710b79f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:24.757+0000 7f710b79f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:37.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:37.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:37.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:37.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:37.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:37.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:37.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:37.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:37.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:37.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:37.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:37.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:37.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:37.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:37.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:37.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:14:24Z" 2022-09-13T19:12:37.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:37.845 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:37.845 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:37.846 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:37.847 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:37.848 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.849 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.849 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.850 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:37.851 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.852 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:37.853 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.853 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.854 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.855 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.856 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:37.857 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.858 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.858 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.859 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.860 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.861 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.861 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:37.862 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.863 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.863 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:37.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.865 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.866 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:37.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.867 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.868 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.868 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.868 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.869 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.869 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.870 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.870 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.871 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.871 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.872 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.872 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.873 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.873 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.874 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.874 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:37.875 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.875 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.876 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.876 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.877 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.877 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.878 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.878 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.879 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.879 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.880 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:37.880 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.880 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:37.881 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:37.881 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:37.882 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.882 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:14:24Z" 2022-09-13T19:12:37.883 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:37.883 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:37.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:37.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:37.885 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~146s 2022-09-13T19:12:37.885 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.886 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:37.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:37.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:37.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:29.901+0000 7ff968bee600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:29.902+0000 7ff968bee600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:37.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:37.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:37.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:37.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:37.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:37.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:37.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:37.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:37.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:37.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:37.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:37.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:37.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:37.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:37.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:37.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:14:29Z" 2022-09-13T19:12:37.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:37.917 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:37.917 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:37.918 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:37.918 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:37.919 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.919 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.920 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.920 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:37.920 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.921 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:37.921 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.922 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.922 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.923 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.923 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:37.923 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.924 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.924 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.925 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.925 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.925 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.926 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:37.926 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.927 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.927 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:37.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.928 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.928 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.929 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:37.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.930 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.930 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.930 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.931 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.931 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.932 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.932 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.933 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.933 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.933 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.934 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.935 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.935 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.936 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.936 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.936 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:37.937 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.937 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.938 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.938 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.938 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.939 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.939 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.940 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.940 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.941 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.941 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:37.942 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.943 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:37.943 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:37.944 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:37.944 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.944 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:14:29Z" 2022-09-13T19:12:37.945 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:37.945 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:37.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:37.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:37.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:37.948 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~151s 2022-09-13T19:12:37.948 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.948 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:37.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:37.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:37.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:37.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:35.051+0000 7f53bdd4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:35.052+0000 7f53bdd4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:37.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:37.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:37.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:37.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:37.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:37.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:37.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:37.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:37.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:37.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:37.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:37.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:37.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:37.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:37.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:37.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:37.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:37.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:37.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:37.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:37.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:37.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:37.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:14:35Z" 2022-09-13T19:12:37.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:37.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:37.981 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:37.982 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:37.982 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:37.983 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:37.983 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:37.984 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.984 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.984 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:37.985 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.985 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:37.986 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.986 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.987 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:37.987 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.987 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:37.988 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.988 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:37.989 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.989 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.989 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.990 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.990 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:37.991 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.991 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.992 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:37.992 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.992 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.993 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.993 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:37.994 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.994 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.994 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:37.995 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:37.995 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:37.996 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:37.996 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:37.997 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:37.997 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:37.997 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:37.998 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:37.998 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:37.999 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:37.999 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:37.999 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.000 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.000 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:38.001 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.001 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.002 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.002 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.003 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.004 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.004 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.005 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.005 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.006 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:38.006 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.006 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:38.007 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:38.008 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:38.008 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.008 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:14:35Z" 2022-09-13T19:12:38.009 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:38.009 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:38.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:38.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:38.012 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~157s 2022-09-13T19:12:38.012 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.013 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:38.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:38.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:38.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:40.201+0000 7f0701110600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:40.202+0000 7f0701110600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:38.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:38.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:38.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:38.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:38.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:38.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:38.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:38.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:38.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:38.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:38.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:38.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:38.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:14:40Z" 2022-09-13T19:12:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:38.044 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:38.045 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:38.045 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:38.046 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:38.046 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.047 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.047 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:38.048 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.048 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:38.049 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.049 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.049 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.050 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.050 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:38.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.051 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.052 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.052 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.052 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.053 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.053 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:38.054 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.054 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.055 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:38.055 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.056 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.056 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.057 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:38.057 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.057 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.058 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.058 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.059 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:38.059 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.060 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.060 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.061 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.061 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.062 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.062 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.063 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.063 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.064 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.064 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:38.065 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.065 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.066 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.066 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.067 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.067 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.068 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.068 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.069 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.069 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.070 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:38.070 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.071 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:38.071 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:38.072 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:38.072 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.073 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:14:40Z" 2022-09-13T19:12:38.073 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:38.074 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:38.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:38.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:38.076 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~163s 2022-09-13T19:12:38.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.077 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:38.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:38.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:38.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:45.333+0000 7fe73bed4600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:45.334+0000 7fe73bed4600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:38.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:38.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:38.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:38.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:38.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:38.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:38.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:38.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:38.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:38.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:38.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:38.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:38.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:38.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:38.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:14:45Z" 2022-09-13T19:12:38.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:38.109 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:38.110 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:38.110 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:38.111 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:38.111 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.112 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.112 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.113 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:38.113 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.113 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:38.114 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.114 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.115 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.115 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.116 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:38.116 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.117 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.117 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.117 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.118 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.118 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.119 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:38.119 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.120 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.120 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:38.120 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.121 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.121 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.122 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:38.122 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.123 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.123 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.124 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.124 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:38.124 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.125 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.125 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.131 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.131 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.132 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.132 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.133 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.133 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.134 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.135 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.135 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:38.136 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.136 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.137 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.137 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.137 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.138 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.138 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.139 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.139 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.140 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.140 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:38.140 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.141 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:38.142 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:38.142 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:38.143 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.143 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:14:45Z" 2022-09-13T19:12:38.144 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:38.144 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:38.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:38.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:38.151 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~168s 2022-09-13T19:12:38.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:50.492+0000 7f1be223f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:50.493+0000 7f1be223f600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:38.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:38.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:38.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:38.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:38.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:38.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:14:50Z" 2022-09-13T19:12:38.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:38.187 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:38.187 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:38.188 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:38.188 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:38.189 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.189 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.189 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.190 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:38.190 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.191 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:38.191 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.192 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.192 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.193 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.193 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:38.193 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.194 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.194 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.195 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.195 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.196 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.196 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:38.196 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.197 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.197 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:38.198 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.198 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.198 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.199 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:38.199 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.200 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.200 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.201 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.201 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:38.202 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.202 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.202 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.203 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.203 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.204 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.204 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.205 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.205 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.206 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.206 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.206 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:38.207 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.207 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.208 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.208 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.209 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.209 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.210 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.210 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.210 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.211 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.211 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:38.212 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.212 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:38.213 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:38.213 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:38.213 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.214 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:14:50Z" 2022-09-13T19:12:38.214 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:38.215 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:38.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:38.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:38.217 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~174s 2022-09-13T19:12:38.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:38.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:55.646+0000 7efe49817600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:14:55.647+0000 7efe49817600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:38.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:38.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:38.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:38.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:38.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:38.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:38.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:38.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:38.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:38.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:14:55Z" 2022-09-13T19:12:38.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:38.251 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:38.251 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:38.252 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:38.252 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:38.253 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.253 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.253 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.254 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:38.254 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.255 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:38.255 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.256 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.256 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.257 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.257 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:38.258 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.258 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.259 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.259 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.260 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.260 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.260 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:38.261 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.261 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.262 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:38.262 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.263 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.263 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.263 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:38.264 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.264 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.265 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.265 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.266 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:38.266 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.267 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.267 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.267 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.268 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.268 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.269 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.269 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.270 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.270 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.271 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.271 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:38.272 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.272 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.273 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.273 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.274 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.274 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.275 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.275 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.276 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.276 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.277 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:38.277 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.278 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:38.278 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:38.279 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:38.279 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.280 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:14:55Z" 2022-09-13T19:12:38.280 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:38.280 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:38.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:38.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:38.283 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~179s 2022-09-13T19:12:38.283 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.284 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:38.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:38.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:38.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:00.781+0000 7f8474590600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:00.782+0000 7f8474590600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:38.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:38.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:38.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:38.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:38.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:38.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:38.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:38.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:38.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:38.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:15:00Z" 2022-09-13T19:12:38.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:38.320 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:38.320 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:38.321 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:38.321 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:38.322 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.322 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.323 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.323 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:38.324 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.325 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:38.325 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.325 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.326 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.326 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.327 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:38.327 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.328 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.328 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.328 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.329 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.329 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.330 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:38.330 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.330 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.331 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:38.331 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.332 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.332 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.332 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:38.333 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.333 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.334 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.334 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.335 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:38.335 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.335 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.336 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.336 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.337 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.337 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.337 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.338 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.338 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.339 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.339 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.340 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:38.340 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.340 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.341 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.341 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.342 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.342 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.342 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.343 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.343 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.344 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.344 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:38.345 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.345 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:38.345 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:38.346 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:38.346 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.347 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:15:00Z" 2022-09-13T19:12:38.347 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:38.348 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:38.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:38.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:38.350 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~185s 2022-09-13T19:12:38.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:05.926+0000 7f7f7526a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:05.927+0000 7f7f7526a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:38.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:38.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:38.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:38.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:38.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:38.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:38.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:38.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:38.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:38.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:15:05Z" 2022-09-13T19:12:38.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:38.382 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:38.383 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:38.383 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:38.383 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:38.384 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.384 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.385 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.385 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:38.386 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.386 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:38.387 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.387 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.388 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.388 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.388 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:38.389 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.389 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.390 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.390 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.390 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.391 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.391 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:38.392 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.392 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.393 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:38.393 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.393 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.394 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:38.395 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.395 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.396 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.396 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.396 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:38.397 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.397 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.398 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.398 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.399 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.399 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.400 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.400 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.401 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.401 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.401 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.402 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:38.402 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.403 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.403 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.403 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.404 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.404 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.405 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.405 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.406 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.406 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.407 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:38.407 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.408 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:38.408 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:38.408 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:38.409 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.409 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:15:05Z" 2022-09-13T19:12:38.410 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:38.410 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:38.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:38.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:38.412 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~191s 2022-09-13T19:12:38.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:11.098+0000 7f82fea20600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:11.099+0000 7f82fea20600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:38.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:38.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:38.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:38.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:38.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:38.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:38.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:38.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:38.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:38.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:38.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:15:11Z" 2022-09-13T19:12:38.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:38.445 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:38.446 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:38.446 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:38.446 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:38.447 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.447 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.448 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.448 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:38.448 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.449 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:38.449 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.450 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.450 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.451 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.451 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:38.452 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.452 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.452 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.453 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.454 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.454 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.454 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:38.455 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.455 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.456 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:38.456 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.457 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.457 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.457 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:38.458 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.458 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.459 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.459 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.459 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:38.460 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.460 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.461 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.461 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.462 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.462 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.462 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.463 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.463 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.464 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.464 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.464 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:38.465 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.465 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.466 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.466 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.467 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.467 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.467 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.468 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.468 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.469 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.469 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:38.469 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.470 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:38.470 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:38.471 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:38.471 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.471 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:15:11Z" 2022-09-13T19:12:38.472 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:38.472 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:38.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:38.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:38.474 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~196s 2022-09-13T19:12:38.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:38.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:38.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:16.231+0000 7f4ceb7d7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:16.232+0000 7f4ceb7d7600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:38.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:38.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:38.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:38.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:38.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:38.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:38.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:38.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:38.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:38.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:15:16Z" 2022-09-13T19:12:38.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:38.507 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:38.507 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:38.508 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:38.508 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:38.508 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.509 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.509 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.510 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:38.510 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.511 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:38.511 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.512 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.512 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.513 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.513 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:38.514 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.514 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.514 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.515 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.515 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.516 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.516 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:38.517 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.517 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.517 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:38.518 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.518 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.519 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.519 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:38.520 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.520 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.521 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.521 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.522 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:38.522 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.522 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.523 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.523 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.524 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.524 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.524 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.525 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.525 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.526 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.526 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.527 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:38.527 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.527 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.528 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.528 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.529 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.529 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.530 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.530 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.531 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.531 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.532 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:38.533 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.533 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:38.534 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:38.534 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:38.535 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.535 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:15:16Z" 2022-09-13T19:12:38.535 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:38.536 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:38.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:38.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:38.538 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~202s 2022-09-13T19:12:38.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.539 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:38.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:38.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:21.380+0000 7fc9d6cac600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:21.381+0000 7fc9d6cac600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:38.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:38.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:38.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:38.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:38.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:38.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:38.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:38.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:38.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:38.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:15:21Z" 2022-09-13T19:12:38.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:38.600 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:38.601 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:38.603 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:38.603 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:38.605 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.605 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.606 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.606 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:38.607 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.607 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:38.607 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.608 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.608 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.609 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.609 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:38.609 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.610 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.610 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.611 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.611 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.612 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.612 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:38.613 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.613 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.613 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:38.614 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.614 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.615 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.615 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:38.616 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.616 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.616 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.617 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.617 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:38.618 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.618 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.619 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.619 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.619 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.620 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.620 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.621 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.621 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.622 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.622 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.622 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:38.623 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.623 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.624 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.624 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.624 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.625 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.625 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.626 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.626 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.627 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.627 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:38.628 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.628 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:38.629 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:38.629 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:38.629 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.630 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:15:21Z" 2022-09-13T19:12:38.630 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:38.631 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:38.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:38.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:38.633 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:38.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:38.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:26.513+0000 7f39ac91d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:26.515+0000 7f39ac91d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:38.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:38.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:38.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:38.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:38.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:38.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:38.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:38.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:38.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:38.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:15:26Z" 2022-09-13T19:12:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:38.681 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:38.682 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:38.683 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:38.684 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:38.685 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.686 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.687 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:38.688 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.689 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:38.690 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.690 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.692 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.692 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.693 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:38.694 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.695 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.696 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.697 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.698 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.698 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.699 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:38.700 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.700 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.700 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:38.701 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.701 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.702 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.702 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:38.703 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.703 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.704 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.704 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.704 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:38.705 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.705 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.706 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.706 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.707 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.707 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.708 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.708 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.708 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.709 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.709 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.710 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:38.710 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.711 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.711 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.711 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.712 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.712 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.713 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.713 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.714 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.714 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.715 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:38.715 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.715 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:38.716 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:38.716 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:38.717 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.717 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:15:26Z" 2022-09-13T19:12:38.718 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:38.718 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:38.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:38.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:38.720 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~207s 2022-09-13T19:12:38.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:31.680+0000 7fd585f4e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:31.681+0000 7fd585f4e600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:38.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:38.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:38.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:38.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:38.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:38.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:38.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:38.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:38.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:38.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:15:31Z" 2022-09-13T19:12:38.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:38.752 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:38.753 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:38.753 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:38.754 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:38.754 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.755 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.755 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.755 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:38.756 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.756 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:38.757 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.757 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.758 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.758 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.759 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:38.759 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.760 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.760 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.761 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.761 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.761 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.762 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:38.762 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.763 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.763 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:38.764 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.764 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.764 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.765 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:38.765 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.766 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.766 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.766 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.767 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:38.767 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.768 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.768 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.769 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.769 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.770 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.770 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.771 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.771 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.772 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.772 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:38.772 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.773 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.773 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.774 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.774 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.774 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.775 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.775 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.776 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.776 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.777 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:38.777 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.777 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:38.778 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:38.778 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:38.779 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.779 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:15:31Z" 2022-09-13T19:12:38.779 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:38.780 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:38.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:38.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:38.782 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~213s 2022-09-13T19:12:38.782 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.783 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:38.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:38.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:38.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:36.824+0000 7f81b7eef600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:36.826+0000 7f81b7eef600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:38.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:38.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:38.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:38.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:38.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:15:36Z" 2022-09-13T19:12:38.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:38.813 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:38.813 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:38.814 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:38.814 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:38.815 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.815 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.816 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.816 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:38.816 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.817 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:38.817 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.818 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.818 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.819 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.819 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:38.819 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.820 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.820 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.821 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.821 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.821 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.822 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:38.822 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.823 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.823 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:38.824 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.824 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.825 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:38.825 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.826 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.826 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.826 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.827 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:38.827 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.828 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.828 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.829 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.829 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.829 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.830 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.830 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.831 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.831 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.832 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.832 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:38.832 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.833 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.833 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.834 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.834 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.834 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.835 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.835 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.836 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.836 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.837 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:38.837 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.837 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:38.838 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:38.838 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:38.839 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.839 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:15:36Z" 2022-09-13T19:12:38.839 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:38.840 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:38.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:38.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:38.842 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~219s 2022-09-13T19:12:38.843 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.843 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:38.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:38.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:38.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:41.964+0000 7fad411e4600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:41.965+0000 7fad411e4600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:38.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:38.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:38.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:38.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:38.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:38.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:38.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:38.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:38.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:38.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:15:41Z" 2022-09-13T19:12:38.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:38.877 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:38.878 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:38.878 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:38.879 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:38.879 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.880 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.880 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.881 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:38.882 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.882 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:38.883 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.883 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.884 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.884 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.885 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:38.885 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.886 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.886 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.887 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.887 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.887 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.888 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:38.889 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.889 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.889 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:38.890 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.890 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.891 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.891 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:38.891 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.892 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.893 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.893 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.893 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:38.894 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.894 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.895 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.896 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.896 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.896 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.897 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.897 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.898 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.898 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.899 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:38.899 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.899 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.900 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.900 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.901 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.901 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.902 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.902 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.902 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.903 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.903 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:38.904 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.904 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:38.904 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:38.905 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:38.905 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.906 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:15:41Z" 2022-09-13T19:12:38.906 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:38.907 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:38.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:38.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:38.909 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~224s 2022-09-13T19:12:38.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.910 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:38.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:38.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:38.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:47.126+0000 7ff60ad5a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:47.127+0000 7ff60ad5a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:38.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:38.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:38.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:38.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:38.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:38.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:38.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:38.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:38.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:38.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:38.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:15:47Z" 2022-09-13T19:12:38.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:38.944 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:38.944 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:38.944 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:38.945 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:38.945 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.946 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.946 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.947 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:38.947 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.947 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:38.948 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.948 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.949 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.950 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:38.950 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.950 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.951 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.951 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.952 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.952 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.952 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:38.953 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.953 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.954 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:38.954 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.955 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.955 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.955 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:38.956 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.956 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.957 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.957 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.958 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:38.958 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.958 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.959 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.960 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.960 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.961 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.961 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.962 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.962 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:38.963 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:38.963 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:38.963 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:38.964 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.964 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:38.965 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:38.965 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:38.965 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.966 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:38.966 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.967 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:38.967 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:38.968 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.968 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:38.968 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:38.969 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:38.969 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:38.970 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:15:47Z" 2022-09-13T19:12:38.970 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:38.971 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:38.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:38.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.972 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:38.972 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:38.973 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~230s 2022-09-13T19:12:38.973 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.973 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:38.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:38.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:38.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:38.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:52.279+0000 7fc142ae2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:52.280+0000 7fc142ae2600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:38.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:38.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:38.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:38.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:38.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:38.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:38.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:38.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:38.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:38.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:38.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:38.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:38.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:39.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:39.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:39.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:15:52Z" 2022-09-13T19:12:39.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:39.004 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:39.004 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:39.005 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:39.005 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:39.005 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.006 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.007 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.007 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:39.007 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.008 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:39.008 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.009 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.009 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.009 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.011 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:39.012 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.012 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.013 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.013 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.014 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.014 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:39.015 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.015 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.016 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:39.016 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.017 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.018 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.018 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:39.020 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.021 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.022 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.022 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.023 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:39.023 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.024 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.024 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.024 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.025 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.025 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.026 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.026 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.027 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.027 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.028 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.029 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:39.030 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.031 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.031 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.032 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.032 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.033 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.033 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.034 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.034 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.034 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.035 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:39.035 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.048 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:39.048 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:39.049 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:39.049 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.050 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:15:52Z" 2022-09-13T19:12:39.050 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:39.051 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:39.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:39.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:39.054 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~235s 2022-09-13T19:12:39.054 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:39.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:39.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:39.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:57.426+0000 7f74d7ad1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:15:57.427+0000 7f74d7ad1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:39.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:39.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:39.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:39.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:39.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:39.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:39.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:39.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:15:57Z" 2022-09-13T19:12:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:39.092 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:39.093 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:39.093 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:39.094 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:39.094 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.094 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.095 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.095 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:39.098 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.099 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:39.100 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.101 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.102 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.103 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.104 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:39.105 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.105 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.106 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.106 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.107 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.107 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.108 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:39.108 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.109 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.109 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:39.110 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.110 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.110 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.111 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:39.111 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.112 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.114 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.114 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.115 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:39.115 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.116 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.116 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.117 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.117 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.118 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.119 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.119 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.120 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.120 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.121 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.121 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:39.122 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.122 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.123 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.123 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.124 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.124 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.125 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.125 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.126 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.126 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.127 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:39.127 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.128 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:39.128 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:39.129 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:39.129 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.130 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:15:57Z" 2022-09-13T19:12:39.130 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:39.131 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:39.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:39.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:39.133 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~241s 2022-09-13T19:12:39.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:39.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:39.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:02.583+0000 7f70e27a3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:02.584+0000 7f70e27a3600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:39.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:39.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:39.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:39.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:39.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:39.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:39.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:39.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:39.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:16:02Z" 2022-09-13T19:12:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:39.167 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:39.168 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:39.168 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:39.168 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:39.169 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.169 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.170 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.170 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:39.170 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.171 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:39.172 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.172 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.172 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.173 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.173 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:39.174 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.174 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.174 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.175 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.175 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.176 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.176 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:39.176 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.177 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.177 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:39.178 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.178 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.179 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.179 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:39.179 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.180 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.180 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.181 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.181 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:39.181 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.182 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.182 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.183 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.183 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.183 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.184 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.184 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.185 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.185 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.185 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.186 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:39.186 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.187 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.187 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.188 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.188 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.188 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.189 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.189 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.190 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.190 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.190 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:39.191 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.191 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:39.192 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:39.192 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:39.192 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.193 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:16:02Z" 2022-09-13T19:12:39.193 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:39.194 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:39.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:39.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:39.196 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~247s 2022-09-13T19:12:39.196 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.197 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:07.723+0000 7fd306271600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:07.724+0000 7fd306271600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:39.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:39.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:39.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:39.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:39.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:39.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:39.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:39.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:39.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:39.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:39.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:39.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:39.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:39.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:16:07Z" 2022-09-13T19:12:39.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:39.228 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:39.228 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:39.229 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:39.229 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:39.229 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.230 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.230 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.231 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:39.231 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.231 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:39.232 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.232 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.233 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.233 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.234 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:39.234 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.234 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.235 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.235 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.236 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.236 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.237 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:39.237 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.238 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.238 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:39.239 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.239 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.240 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.240 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:39.241 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.241 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.241 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.242 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.242 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:39.243 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.243 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.244 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.244 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.245 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.245 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.246 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.246 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.247 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.247 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.247 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.248 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:39.248 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.249 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.249 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.250 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.250 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.251 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.251 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.251 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.252 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.252 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:39.253 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.253 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:39.254 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:39.254 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:39.255 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.255 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:16:07Z" 2022-09-13T19:12:39.256 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:39.256 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:39.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:39.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:39.258 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~252s 2022-09-13T19:12:39.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:39.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:39.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:39.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:12.885+0000 7f17e226c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:12.886+0000 7f17e226c600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:39.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:39.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:39.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:39.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:39.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:39.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:39.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:39.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:39.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:39.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:16:12Z" 2022-09-13T19:12:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:39.301 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:39.301 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:39.301 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:39.302 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:39.302 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.303 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.303 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.304 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:39.304 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.305 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:39.306 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.306 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.307 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.307 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.308 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:39.308 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.309 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.309 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.310 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.310 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.310 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.311 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:39.311 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.312 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.312 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:39.313 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.313 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.314 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.314 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:39.315 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.315 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.315 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.316 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.316 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:39.317 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.317 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.318 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.318 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.319 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.319 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.320 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.321 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.321 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.321 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.322 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:39.322 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.323 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.323 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.323 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.324 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.324 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.325 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.325 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.326 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.326 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.326 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:39.327 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.327 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:39.328 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:39.328 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:39.329 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.330 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:16:12Z" 2022-09-13T19:12:39.330 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:39.331 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:39.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:39.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:39.335 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~258s 2022-09-13T19:12:39.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:39.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:39.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:39.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:18.023+0000 7fc231b4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:18.024+0000 7fc231b4d600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:39.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:39.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:39.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:39.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:39.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:39.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:39.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:39.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:39.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:39.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:39.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:16:18Z" 2022-09-13T19:12:39.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:39.378 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:39.379 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:39.379 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:39.379 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:39.382 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.382 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.383 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.384 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:39.384 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.385 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:39.386 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.387 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.387 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.389 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.390 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:39.391 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.392 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.393 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.393 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.393 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.394 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.395 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:39.395 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.396 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.397 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:39.397 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.398 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.398 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.399 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:39.399 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.400 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.400 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.401 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.401 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:39.402 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.402 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.403 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.403 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.403 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.404 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.404 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.405 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.405 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.406 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.406 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.407 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:39.407 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.415 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.416 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.417 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.417 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.417 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.418 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.418 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.419 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.419 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.420 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:39.420 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.421 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:39.421 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:39.430 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:39.430 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.431 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:16:18Z" 2022-09-13T19:12:39.431 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:39.432 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:39.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:39.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:39.444 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~264s 2022-09-13T19:12:39.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.446 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:23.169+0000 7ff389905600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:23.170+0000 7ff389905600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:39.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:39.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:39.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:39.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:39.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:39.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:39.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:16:23Z" 2022-09-13T19:12:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:39.492 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:39.492 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:39.493 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:39.493 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:39.494 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.495 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.496 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:39.496 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.497 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:39.497 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.497 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.498 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.498 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.499 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:39.499 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.500 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.500 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.500 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.501 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.501 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.502 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:39.503 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.503 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.504 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:39.504 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.504 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.505 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:39.506 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.508 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.508 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.508 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.509 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:39.509 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.510 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.510 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.511 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.511 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.512 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.512 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.513 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.513 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.513 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.514 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.514 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:39.515 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.515 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.516 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.516 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.517 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.517 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.518 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.518 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.519 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.519 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.520 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:39.520 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.520 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:39.521 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:39.521 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:39.522 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.522 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:16:23Z" 2022-09-13T19:12:39.523 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:39.523 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:39.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:39.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:39.526 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.527 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:39.531 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~269s 2022-09-13T19:12:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:28.330+0000 7f9c97f3a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:28.331+0000 7f9c97f3a600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:39.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:39.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:39.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:39.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:39.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:39.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:39.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:39.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:39.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:39.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:39.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:16:28Z" 2022-09-13T19:12:39.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:39.567 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:39.567 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:39.567 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:39.568 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:39.568 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.569 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.569 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.569 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:39.571 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.571 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:39.571 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.572 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.572 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.573 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.573 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:39.573 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.574 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.574 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.575 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.575 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.576 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.576 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:39.576 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.577 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.577 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:39.578 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.578 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.579 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.579 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:39.580 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.580 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.581 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.582 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.582 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:39.583 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.583 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.583 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.584 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.584 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.585 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.585 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.585 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.586 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.586 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.587 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.587 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:39.587 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.588 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.588 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.589 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.589 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.590 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.590 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.590 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.591 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.592 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.592 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:39.593 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.593 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:39.594 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:39.594 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:39.594 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.595 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:16:28Z" 2022-09-13T19:12:39.595 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:39.596 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:39.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:39.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:39.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:39.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:39.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:39.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:33.468+0000 7fedd52b8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:33.470+0000 7fedd52b8600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:39.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:39.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:39.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:39.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:39.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:39.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:39.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:39.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:16:33Z" 2022-09-13T19:12:39.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:39.630 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:39.631 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:39.631 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:39.632 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:39.632 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.632 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.633 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.633 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:39.634 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.635 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:39.635 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.636 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.636 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.636 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.637 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:39.637 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.638 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.638 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.639 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.639 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.639 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.640 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:39.640 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.641 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.641 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:39.642 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.642 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.643 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:39.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.644 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.645 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.645 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.646 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:39.647 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.647 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.648 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.648 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.649 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.649 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.650 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.650 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.651 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.652 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.652 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.653 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:39.653 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.654 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.655 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.656 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.656 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.657 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.658 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.658 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.659 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.659 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:39.660 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.660 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:39.661 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:39.661 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:39.662 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.663 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:16:33Z" 2022-09-13T19:12:39.663 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:39.664 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:39.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:39.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:39.666 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~275s 2022-09-13T19:12:39.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.667 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:38.603+0000 7f14d0df1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:38.604+0000 7f14d0df1600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:39.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:39.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:39.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:39.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:39.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:39.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:39.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:39.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:39.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:39.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:39.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:39.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:39.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:39.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:39.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:16:38Z" 2022-09-13T19:12:39.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:39.700 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:39.700 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:39.701 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:39.701 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:39.702 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.702 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.702 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.703 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:39.703 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.704 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:39.704 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.705 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.705 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.705 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.706 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:39.706 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.707 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.707 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.708 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.708 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.708 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.709 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:39.709 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.710 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.711 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:39.712 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.712 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.713 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.714 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:39.714 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.715 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.716 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.717 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.717 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:39.719 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.719 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.720 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.721 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.722 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.722 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.723 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.724 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.724 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.725 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.725 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.726 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:39.727 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.728 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.729 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.730 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.730 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.731 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.731 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.732 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.732 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.733 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.733 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:39.733 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.734 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:39.734 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:39.735 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:39.735 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.736 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:16:38Z" 2022-09-13T19:12:39.736 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:39.736 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:39.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:39.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:39.738 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~280s 2022-09-13T19:12:39.739 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.739 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:39.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:39.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:39.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:43.748+0000 7f8cfac40600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:43.749+0000 7f8cfac40600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:39.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:39.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:39.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:39.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:39.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:39.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:39.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:39.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:39.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:39.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:39.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:16:43Z" 2022-09-13T19:12:39.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:39.772 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:39.772 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:39.772 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:39.773 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:39.773 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.774 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.774 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.774 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:39.775 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.775 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:39.776 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.776 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.777 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.777 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.778 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:39.778 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.779 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.779 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.779 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.780 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.781 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.781 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:39.782 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.782 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.783 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:39.783 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.784 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.785 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.785 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:39.785 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.786 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.786 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.787 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.787 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:39.788 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.788 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.788 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.789 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.789 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.790 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.790 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.791 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.791 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.792 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.792 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.792 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:39.793 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.793 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.797 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.798 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.798 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.799 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.799 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.800 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.800 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.801 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:39.801 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.802 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:39.802 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:39.803 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:39.803 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.803 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:16:43Z" 2022-09-13T19:12:39.804 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:39.804 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:39.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:39.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:39.807 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~286s 2022-09-13T19:12:39.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:48.911+0000 7fba68d31600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:48.913+0000 7fba68d31600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:39.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:39.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:39.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:39.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:39.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:39.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:39.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:39.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:39.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:39.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:39.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:16:48Z" 2022-09-13T19:12:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:39.842 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:39.843 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:39.843 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:39.844 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:39.844 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.845 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.845 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.845 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:39.846 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.846 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:39.847 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.847 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.848 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.848 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.848 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:39.849 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.849 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.850 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.850 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.851 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.852 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.852 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:39.853 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.853 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.853 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:39.854 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.854 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.855 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.855 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:39.856 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.856 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.856 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.857 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.857 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:39.858 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.858 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.859 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.859 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.860 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.860 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.860 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.861 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.861 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.862 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.863 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.863 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:39.864 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.870 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.871 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.871 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.872 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.872 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.872 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.873 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.873 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.875 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.876 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:39.876 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.877 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:39.877 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:39.877 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:39.878 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.878 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:16:48Z" 2022-09-13T19:12:39.879 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:39.879 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:39.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:39.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:39.882 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~292s 2022-09-13T19:12:39.882 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:39.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:39.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:54.053+0000 7f700e4ff600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:54.055+0000 7f700e4ff600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:39.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:39.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:39.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:39.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:39.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:39.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:39.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:39.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:39.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:39.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:39.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:39.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:39.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:16:54Z" 2022-09-13T19:12:39.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:39.915 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:39.916 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:39.916 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:39.916 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:39.917 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.917 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.918 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.918 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:39.919 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.919 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:39.920 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.920 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.920 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.921 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.921 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:39.922 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.922 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.923 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.923 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.924 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.924 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.924 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:39.925 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.925 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.926 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:39.926 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.927 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.927 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.927 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:39.928 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.928 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.929 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.929 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.930 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:39.931 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.931 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.932 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.932 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.932 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.933 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.933 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.934 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.934 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.935 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.935 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.936 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:39.936 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.936 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.937 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.937 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.938 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:39.938 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.939 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.939 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.940 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.940 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.941 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:39.941 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.941 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:39.942 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:39.942 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:39.943 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.943 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:16:54Z" 2022-09-13T19:12:39.944 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:39.944 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:39.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:39.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:39.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:39.947 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~297s 2022-09-13T19:12:39.947 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.947 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:39.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:39.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:39.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:39.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:59.215+0000 7f7c9de4b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:16:59.216+0000 7f7c9de4b600 20 RGW-SYNC:meta: read sync status 2022-09-13T19:12:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "sync_status": { 2022-09-13T19:12:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "info": { 2022-09-13T19:12:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "status": "sync", 2022-09-13T19:12:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "num_shards": 4, 2022-09-13T19:12:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "markers": [ 2022-09-13T19:12:39.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 0, 2022-09-13T19:12:39.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 1, 2022-09-13T19:12:39.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 1, 2022-09-13T19:12:39.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "", 2022-09-13T19:12:39.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "0.000000", 2022-09-13T19:12:39.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 2, 2022-09-13T19:12:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:39.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:39.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: { 2022-09-13T19:12:39.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "key": 3, 2022-09-13T19:12:39.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "val": { 2022-09-13T19:12:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "state": 1, 2022-09-13T19:12:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "next_step_marker": "", 2022-09-13T19:12:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total_entries": 0, 2022-09-13T19:12:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "pos": 0, 2022-09-13T19:12:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "realm_epoch": 2 2022-09-13T19:12:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: ] 2022-09-13T19:12:39.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "full_sync": { 2022-09-13T19:12:39.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "total": 1, 2022-09-13T19:12:39.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "complete": 1 2022-09-13T19:12:39.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: }, 2022-09-13T19:12:39.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: "current_time": "2022-09-13T08:16:59Z" 2022-09-13T19:12:39.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stdout: INFO: } 2022-09-13T19:12:39.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2022-09-13T19:12:39.990 INFO:tasks.rgw_multisite_tests: "sync_status": { 2022-09-13T19:12:39.990 INFO:tasks.rgw_multisite_tests: "info": { 2022-09-13T19:12:39.991 INFO:tasks.rgw_multisite_tests: "status": "sync", 2022-09-13T19:12:39.991 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2022-09-13T19:12:39.992 INFO:tasks.rgw_multisite_tests: "period": "3255cc65-e34b-4e83-a436-e06431413191", 2022-09-13T19:12:39.992 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.993 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.993 INFO:tasks.rgw_multisite_tests: "markers": [ 2022-09-13T19:12:39.994 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:39.994 INFO:tasks.rgw_multisite_tests: "key": 0, 2022-09-13T19:12:39.995 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:39.995 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:39.996 INFO:tasks.rgw_multisite_tests: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:39.996 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:39.996 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2022-09-13T19:12:39.997 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:39.997 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:25:52.976234Z", 2022-09-13T19:12:39.998 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:39.998 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:39.999 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:39.999 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:40.000 INFO:tasks.rgw_multisite_tests: "key": 1, 2022-09-13T19:12:40.000 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:40.000 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:40.001 INFO:tasks.rgw_multisite_tests: "marker": "", 2022-09-13T19:12:40.001 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:40.002 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:40.002 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:40.003 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2022-09-13T19:12:40.004 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:40.005 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:40.005 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:40.006 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:40.007 INFO:tasks.rgw_multisite_tests: "key": 2, 2022-09-13T19:12:40.007 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:40.007 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:40.008 INFO:tasks.rgw_multisite_tests: "marker": "1_1663055099.141531_226.1", 2022-09-13T19:12:40.008 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:40.009 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:40.009 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:40.010 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:44:59.141531Z", 2022-09-13T19:12:40.010 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:40.011 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:40.011 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:40.012 INFO:tasks.rgw_multisite_tests: { 2022-09-13T19:12:40.012 INFO:tasks.rgw_multisite_tests: "key": 3, 2022-09-13T19:12:40.013 INFO:tasks.rgw_multisite_tests: "val": { 2022-09-13T19:12:40.014 INFO:tasks.rgw_multisite_tests: "state": 1, 2022-09-13T19:12:40.014 INFO:tasks.rgw_multisite_tests: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:40.015 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2022-09-13T19:12:40.015 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2022-09-13T19:12:40.015 INFO:tasks.rgw_multisite_tests: "pos": 0, 2022-09-13T19:12:40.016 INFO:tasks.rgw_multisite_tests: "timestamp": "2022-09-13T07:28:20.288347Z", 2022-09-13T19:12:40.017 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2022-09-13T19:12:40.018 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:40.018 INFO:tasks.rgw_multisite_tests: } 2022-09-13T19:12:40.018 INFO:tasks.rgw_multisite_tests: ] 2022-09-13T19:12:40.019 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:40.019 INFO:tasks.rgw_multisite_tests: "full_sync": { 2022-09-13T19:12:40.020 INFO:tasks.rgw_multisite_tests: "total": 1, 2022-09-13T19:12:40.020 INFO:tasks.rgw_multisite_tests: "complete": 1 2022-09-13T19:12:40.021 INFO:tasks.rgw_multisite_tests: }, 2022-09-13T19:12:40.028 INFO:tasks.rgw_multisite_tests: "current_time": "2022-09-13T08:16:59Z" 2022-09-13T19:12:40.035 INFO:tasks.rgw_multisite_tests:} 2022-09-13T19:12:40.037 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663053952.976234_34.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-13T07:25:52.976234Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663055099.141531_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:44:59.141531Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663054100.288347_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-13T07:28:20.288347Z', 'realm_epoch': 2}}] 2022-09-13T19:12:40.043 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663056715.246326_454.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:40.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663055099.141531_226.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:40.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1663056715.246326_454.1 target=1_1663055099.141531_226.1 2022-09-13T19:12:40.046 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~303s 2022-09-13T19:12:40.047 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: BARK! unmounting mounts and killing all daemons 2022-09-13T19:12:40.048 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0: INFO: Sent signal 15 2022-09-13T19:12:40.049 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1: INFO: Sent signal 15 2022-09-13T19:12:40.050 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2: INFO: Sent signal 15 2022-09-13T19:12:40.051 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mon.a: INFO: Sent signal 15 2022-09-13T19:12:40.052 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1: INFO: Sent signal 15 2022-09-13T19:12:40.053 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mgr.x: INFO: Sent signal 15 2022-09-13T19:12:40.053 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0.smithi083.stderr: INFO: 2022-09-13T08:17:02.549+0000 7f4466a78700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 0 (PID: 101783) UID: 0 2022-09-13T19:12:40.054 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0.smithi083.stderr: INFO: 2022-09-13T08:17:02.549+0000 7f4466a78700 -1 osd.0 41 *** Got signal Terminated *** 2022-09-13T19:12:40.055 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0.smithi083.stderr: INFO: 2022-09-13T08:17:02.549+0000 7f4466a78700 -1 osd.0 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-13T19:12:40.056 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1.smithi083.stderr: INFO: 2022-09-13T08:17:02.552+0000 7fe2ef316700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 1 (PID: 101790) UID: 0 2022-09-13T19:12:40.057 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1.smithi083.stderr: INFO: 2022-09-13T08:17:02.552+0000 7fe2ef316700 -1 osd.1 41 *** Got signal Terminated *** 2022-09-13T19:12:40.058 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1.smithi083.stderr: INFO: 2022-09-13T08:17:02.552+0000 7fe2ef316700 -1 osd.1 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-13T19:12:40.059 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2.smithi083.stderr: INFO: 2022-09-13T08:17:02.553+0000 7f4ed4ed2700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 2 (PID: 101793) UID: 0 2022-09-13T19:12:40.059 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2.smithi083.stderr: INFO: 2022-09-13T08:17:02.553+0000 7f4ed4ed2700 -1 osd.2 41 *** Got signal Terminated *** 2022-09-13T19:12:40.060 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2.smithi083.stderr: INFO: 2022-09-13T08:17:02.553+0000 7f4ed4ed2700 -1 osd.2 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-13T19:12:40.061 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mon.a.smithi083.stderr: INFO: 2022-09-13T08:17:02.553+0000 7fe0d6cbc700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c2 -i a (PID: 101373) UID: 0 2022-09-13T19:12:40.062 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mon.a.smithi083.stderr: INFO: 2022-09-13T08:17:02.553+0000 7fe0d6cbc700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-09-13T19:12:40.063 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1.smithi083.stdout: INFO: 2022-09-13T08:17:02.554+0000 7fd974c11700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 107345) UID: 0 2022-09-13T19:12:40.064 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1.smithi083.stdout: INFO: 2022-09-13T08:17:02.554+0000 7fd978e7c600 -1 shutting down 2022-09-13T19:12:40.065 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mgr.x.smithi083.stderr: INFO: 2022-09-13T08:17:02.555+0000 7fe76d000700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster c2 -i x (PID: 101380) UID: 0 2022-09-13T19:12:40.066 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mgr.x.smithi083.stderr: INFO: 2022-09-13T08:17:02.555+0000 7fe76d000700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2022-09-13T19:12:40.067 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:40.068 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.069 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:40.070 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_encrypted_object_sync 2022-09-13T19:12:40.071 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:40.072 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:40.073 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:40.074 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:40.075 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1279, in test_encrypted_object_sync 2022-09-13T19:12:40.075 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2022-09-13T19:12:40.076 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:40.077 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2022-09-13T19:12:40.077 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 184, in zone_meta_checkpoint 2022-09-13T19:12:40.084 INFO:tasks.rgw_multisite_tests: period, realm_epoch, num_shards, sync_status = meta_sync_status(zone) 2022-09-13T19:12:40.085 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 141, in meta_sync_status 2022-09-13T19:12:40.087 INFO:tasks.rgw_multisite_tests: assert(retcode == 2) # ENOENT 2022-09-13T19:12:40.088 INFO:tasks.rgw_multisite_tests:AssertionError: 2022-09-13T19:12:40.089 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:40.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=yxvvuh-61 2022-09-13T19:12:40.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-61/ 2022-09-13T19:12:40.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-61/ 2022-09-13T19:12:40.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-61/ 2022-09-13T19:12:40.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:40.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.103 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.104 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.104 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.105 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:20:46 GMT 2022-09-13T19:12:40.105 INFO:tasks.rgw_multisite_tests:/yxvvuh-61/ 2022-09-13T19:12:40.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.106 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:Rwrs7Ck2frcabdpY5LsIbzfai7U= 2022-09-13T19:12:40.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 08:20:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:Rwrs7Ck2frcabdpY5LsIbzfai7U=', 'Content-Length': '0'} 2022-09-13T19:12:40.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000bd454e1dad04be01-0063203d5e-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 08:20:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-61/testobj-sse-c 2022-09-13T19:12:40.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-61/testobj-sse-c 2022-09-13T19:12:40.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-61/testobj-sse-c 2022-09-13T19:12:40.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-13T19:12:40.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.112 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.113 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2022-09-13T19:12:40.114 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:40.114 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:20:46 GMT 2022-09-13T19:12:40.115 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-algorithm:AES256 2022-09-13T19:12:40.116 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= 2022-09-13T19:12:40.117 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== 2022-09-13T19:12:40.118 INFO:tasks.rgw_multisite_tests:/yxvvuh-61/testobj-sse-c 2022-09-13T19:12:40.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.119 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:RgxR8eyCDc310zX1uDs6hF1Nt7E= 2022-09-13T19:12:40.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 08:20:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RgxR8eyCDc310zX1uDs6hF1Nt7E='} 2022-09-13T19:12:40.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000d664d4c11207af30-0063203d5e-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 08:20:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-61/testobj-sse-kms 2022-09-13T19:12:40.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-61/testobj-sse-kms 2022-09-13T19:12:40.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-61/testobj-sse-kms 2022-09-13T19:12:40.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-13T19:12:40.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.125 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.125 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2022-09-13T19:12:40.125 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:40.126 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 08:20:46 GMT 2022-09-13T19:12:40.126 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption:aws:kms 2022-09-13T19:12:40.127 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-aws-kms-key-id:testkey-1 2022-09-13T19:12:40.127 INFO:tasks.rgw_multisite_tests:/yxvvuh-61/testobj-sse-kms 2022-09-13T19:12:40.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.128 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:wvx68MHJhfoKMgf2XSCRLYvbE5w= 2022-09-13T19:12:40.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 08:20:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:wvx68MHJhfoKMgf2XSCRLYvbE5w='} 2022-09-13T19:12:40.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6eXh2dnVoLTYxL3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx000006eef3a7a7c1550f1-0063203d5e-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 08:20:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.129 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:40.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:40.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:40.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:40.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:40.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:40.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663057246.045157_471.1", 2022-09-13T19:12:40.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T08:20:46.045157Z" 2022-09-13T19:12:40.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:40.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663057246.045157_471.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:40.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:40.167 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:25:46.265+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:30:46.265+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:35:46.265+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:40:46.265+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:45:46.266+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:50:46.267+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T08:55:46.266+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T09:00:46.267+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T09:05:46.268+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2022-09-13T19:12:40.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T09:10:46.268+0000 7f6564bbe600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2022-09-13T19:12:40.176 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:40.177 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.177 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:40.178 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_index_log_trim 2022-09-13T19:12:40.181 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:40.181 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:40.182 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:40.182 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:40.183 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1309, in test_bucket_index_log_trim 2022-09-13T19:12:40.183 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2022-09-13T19:12:40.184 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1304, in make_test_bucket 2022-09-13T19:12:40.185 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2022-09-13T19:12:40.185 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:40.186 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2022-09-13T19:12:40.186 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 184, in zone_meta_checkpoint 2022-09-13T19:12:40.187 INFO:tasks.rgw_multisite_tests: period, realm_epoch, num_shards, sync_status = meta_sync_status(zone) 2022-09-13T19:12:40.187 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 141, in meta_sync_status 2022-09-13T19:12:40.187 INFO:tasks.rgw_multisite_tests: assert(retcode == 2) # ENOENT 2022-09-13T19:12:40.188 INFO:tasks.rgw_multisite_tests:AssertionError: 2022-09-13T19:12:40.189 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:40.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=yxvvuh-62 2022-09-13T19:12:40.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-62/ 2022-09-13T19:12:40.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-62/ 2022-09-13T19:12:40.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-62/ 2022-09-13T19:12:40.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:40.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.196 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.196 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.201 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.203 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 09:10:46 GMT 2022-09-13T19:12:40.209 INFO:tasks.rgw_multisite_tests:/yxvvuh-62/ 2022-09-13T19:12:40.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.211 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:pNqW/41uNV1dKuiTrKwumWMMV8A= 2022-09-13T19:12:40.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pNqW/41uNV1dKuiTrKwumWMMV8A=', 'Content-Length': '0'} 2022-09-13T19:12:40.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000006d53d55180661671-0063204916-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-62/ 2022-09-13T19:12:40.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-62/ 2022-09-13T19:12:40.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:40.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-62/ 2022-09-13T19:12:40.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.229 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:40.230 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.231 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.231 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 09:10:46 GMT 2022-09-13T19:12:40.232 INFO:tasks.rgw_multisite_tests:/yxvvuh-62/ 2022-09-13T19:12:40.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.233 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A= 2022-09-13T19:12:40.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A=', 'Content-Length': '0'} 2022-09-13T19:12:40.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000547efb11e7302678-0063204916-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-62/a 2022-09-13T19:12:40.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-62/a 2022-09-13T19:12:40.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-62/a 2022-09-13T19:12:40.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T19:12:40.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.242 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.242 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2022-09-13T19:12:40.243 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:40.244 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 09:10:46 GMT 2022-09-13T19:12:40.244 INFO:tasks.rgw_multisite_tests:/yxvvuh-62/a 2022-09-13T19:12:40.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.245 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:/TxJ28dQWmlNUubCzqhq7GAyh7k= 2022-09-13T19:12:40.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:/TxJ28dQWmlNUubCzqhq7GAyh7k='} 2022-09-13T19:12:40.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000defe9c77bfae17e9-0063204916-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-62/ 2022-09-13T19:12:40.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-62/ 2022-09-13T19:12:40.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:40.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-62/ 2022-09-13T19:12:40.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.254 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:40.254 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.255 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.256 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 09:10:46 GMT 2022-09-13T19:12:40.257 INFO:tasks.rgw_multisite_tests:/yxvvuh-62/ 2022-09-13T19:12:40.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.258 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A= 2022-09-13T19:12:40.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A=', 'Content-Length': '0'} 2022-09-13T19:12:40.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023f90cdfd794bf4c-0063204916-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-62/b 2022-09-13T19:12:40.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-62/b 2022-09-13T19:12:40.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-62/b 2022-09-13T19:12:40.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T19:12:40.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.269 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.270 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2022-09-13T19:12:40.271 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:40.273 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 09:10:46 GMT 2022-09-13T19:12:40.274 INFO:tasks.rgw_multisite_tests:/yxvvuh-62/b 2022-09-13T19:12:40.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.277 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:muhhWkSqWbrUauBK3Eynmy+6juU= 2022-09-13T19:12:40.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:muhhWkSqWbrUauBK3Eynmy+6juU='} 2022-09-13T19:12:40.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000432d817334668fbe-0063204916-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-62/ 2022-09-13T19:12:40.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-62/ 2022-09-13T19:12:40.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:40.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-62/ 2022-09-13T19:12:40.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.290 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:40.291 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.292 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.293 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 09:10:46 GMT 2022-09-13T19:12:40.294 INFO:tasks.rgw_multisite_tests:/yxvvuh-62/ 2022-09-13T19:12:40.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.295 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A= 2022-09-13T19:12:40.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A=', 'Content-Length': '0'} 2022-09-13T19:12:40.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e308ed8db227ce6a-0063204916-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-62/c 2022-09-13T19:12:40.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-62/c 2022-09-13T19:12:40.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-62/c 2022-09-13T19:12:40.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T19:12:40.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.308 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.309 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2022-09-13T19:12:40.310 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:40.311 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 09:10:46 GMT 2022-09-13T19:12:40.312 INFO:tasks.rgw_multisite_tests:/yxvvuh-62/c 2022-09-13T19:12:40.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.314 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:RZfOC1j82BxDftMKcsXZz7SbCwg= 2022-09-13T19:12:40.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:RZfOC1j82BxDftMKcsXZz7SbCwg='} 2022-09-13T19:12:40.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000621f4700de617488-0063204916-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-62/ 2022-09-13T19:12:40.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-62/ 2022-09-13T19:12:40.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:40.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-62/ 2022-09-13T19:12:40.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.326 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:40.327 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.327 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.328 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 09:10:46 GMT 2022-09-13T19:12:40.329 INFO:tasks.rgw_multisite_tests:/yxvvuh-62/ 2022-09-13T19:12:40.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.331 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A= 2022-09-13T19:12:40.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pR49o6643wODVpp2O0GGnCXVl3A=', 'Content-Length': '0'} 2022-09-13T19:12:40.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036902a2b0b40ecb1-0063204916-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-62/d 2022-09-13T19:12:40.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-62/d 2022-09-13T19:12:40.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-62/d 2022-09-13T19:12:40.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T19:12:40.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.344 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.345 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2022-09-13T19:12:40.345 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:40.346 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 09:10:46 GMT 2022-09-13T19:12:40.347 INFO:tasks.rgw_multisite_tests:/yxvvuh-62/d 2022-09-13T19:12:40.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.349 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:7hqFbE7X/sjRvjiIqEXOZ016QyA= 2022-09-13T19:12:40.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 09:10:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:7hqFbE7X/sjRvjiIqEXOZ016QyA='} 2022-09-13T19:12:40.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc5c8c528e927b84-0063204916-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 09:10:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.352 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:40.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:40.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:40.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:40.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:40.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663060246.371724_555.1", 2022-09-13T19:12:40.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T09:10:46.371724Z" 2022-09-13T19:12:40.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:40.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:40.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:40.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:40.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663060246.371724_555.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:40.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:40.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T09:15:46.772+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T09:20:46.772+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T09:25:46.773+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T09:30:46.773+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T09:35:46.774+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T09:40:46.775+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T09:45:46.775+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T09:50:46.775+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T09:55:46.775+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2022-09-13T19:12:40.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T10:00:46.776+0000 7f9be6906600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2022-09-13T19:12:40.404 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:40.405 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.406 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:40.407 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2022-09-13T19:12:40.408 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:40.408 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:40.409 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:40.410 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:40.411 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1363, in test_bucket_reshard_index_log_trim 2022-09-13T19:12:40.412 INFO:tasks.rgw_multisite_tests: test_bucket = make_test_bucket() 2022-09-13T19:12:40.413 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1358, in make_test_bucket 2022-09-13T19:12:40.414 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2022-09-13T19:12:40.414 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:40.415 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2022-09-13T19:12:40.416 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 184, in zone_meta_checkpoint 2022-09-13T19:12:40.417 INFO:tasks.rgw_multisite_tests: period, realm_epoch, num_shards, sync_status = meta_sync_status(zone) 2022-09-13T19:12:40.420 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 141, in meta_sync_status 2022-09-13T19:12:40.422 INFO:tasks.rgw_multisite_tests: assert(retcode == 2) # ENOENT 2022-09-13T19:12:40.422 INFO:tasks.rgw_multisite_tests:AssertionError: 2022-09-13T19:12:40.423 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:40.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=yxvvuh-63 2022-09-13T19:12:40.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-63/ 2022-09-13T19:12:40.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-63/ 2022-09-13T19:12:40.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-63/ 2022-09-13T19:12:40.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:40.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.430 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.430 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.431 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.431 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 10:00:46 GMT 2022-09-13T19:12:40.432 INFO:tasks.rgw_multisite_tests:/yxvvuh-63/ 2022-09-13T19:12:40.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.432 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:tut0dZSKDsA4fcmEqBSHplNZybQ= 2022-09-13T19:12:40.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 10:00:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:tut0dZSKDsA4fcmEqBSHplNZybQ=', 'Content-Length': '0'} 2022-09-13T19:12:40.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e9ff849664b91929-00632054ce-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 10:00:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-63/ 2022-09-13T19:12:40.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-63/ 2022-09-13T19:12:40.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:40.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-63/ 2022-09-13T19:12:40.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.443 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:40.444 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.445 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.446 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 10:00:46 GMT 2022-09-13T19:12:40.447 INFO:tasks.rgw_multisite_tests:/yxvvuh-63/ 2022-09-13T19:12:40.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.449 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:oRZBJNrKgFogjVioiIcukiSXsNg= 2022-09-13T19:12:40.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 10:00:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oRZBJNrKgFogjVioiIcukiSXsNg=', 'Content-Length': '0'} 2022-09-13T19:12:40.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094687016d78e8f6b-00632054ce-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 10:00:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-63/a 2022-09-13T19:12:40.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-63/a 2022-09-13T19:12:40.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-63/a 2022-09-13T19:12:40.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T19:12:40.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.461 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.462 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2022-09-13T19:12:40.463 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:40.463 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 10:00:46 GMT 2022-09-13T19:12:40.464 INFO:tasks.rgw_multisite_tests:/yxvvuh-63/a 2022-09-13T19:12:40.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.466 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:qvqtOVyzKUmk/eboboxYh7EGQGM= 2022-09-13T19:12:40.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 10:00:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qvqtOVyzKUmk/eboboxYh7EGQGM='} 2022-09-13T19:12:40.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbb9b57feb925875-00632054ce-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 10:00:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-63/ 2022-09-13T19:12:40.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-63/ 2022-09-13T19:12:40.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:40.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-63/ 2022-09-13T19:12:40.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.478 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:40.479 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.480 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.481 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 10:00:46 GMT 2022-09-13T19:12:40.481 INFO:tasks.rgw_multisite_tests:/yxvvuh-63/ 2022-09-13T19:12:40.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.483 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:oRZBJNrKgFogjVioiIcukiSXsNg= 2022-09-13T19:12:40.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 10:00:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oRZBJNrKgFogjVioiIcukiSXsNg=', 'Content-Length': '0'} 2022-09-13T19:12:40.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb25daf77800bf32-00632054ce-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 10:00:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-63/b 2022-09-13T19:12:40.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-63/b 2022-09-13T19:12:40.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-63/b 2022-09-13T19:12:40.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T19:12:40.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.495 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.496 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2022-09-13T19:12:40.497 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:40.498 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 10:00:46 GMT 2022-09-13T19:12:40.500 INFO:tasks.rgw_multisite_tests:/yxvvuh-63/b 2022-09-13T19:12:40.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.501 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:BG3Uzf14Tk2x6lOcIaGriVz03H0= 2022-09-13T19:12:40.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 10:00:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BG3Uzf14Tk2x6lOcIaGriVz03H0='} 2022-09-13T19:12:40.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006da5117129a5965f-00632054ce-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 10:00:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-63/ 2022-09-13T19:12:40.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-63/ 2022-09-13T19:12:40.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:40.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-63/ 2022-09-13T19:12:40.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.507 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:40.508 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.509 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.509 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 10:00:46 GMT 2022-09-13T19:12:40.510 INFO:tasks.rgw_multisite_tests:/yxvvuh-63/ 2022-09-13T19:12:40.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.511 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:oRZBJNrKgFogjVioiIcukiSXsNg= 2022-09-13T19:12:40.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 10:00:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oRZBJNrKgFogjVioiIcukiSXsNg=', 'Content-Length': '0'} 2022-09-13T19:12:40.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6cb192104949359-00632054ce-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 10:00:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-63/c 2022-09-13T19:12:40.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-63/c 2022-09-13T19:12:40.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-63/c 2022-09-13T19:12:40.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T19:12:40.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.517 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.517 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2022-09-13T19:12:40.518 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:40.519 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 10:00:46 GMT 2022-09-13T19:12:40.519 INFO:tasks.rgw_multisite_tests:/yxvvuh-63/c 2022-09-13T19:12:40.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.520 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:BrZ4Rwp89tvi/xJFPV7I6/14RQo= 2022-09-13T19:12:40.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 10:00:46 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:BrZ4Rwp89tvi/xJFPV7I6/14RQo='} 2022-09-13T19:12:40.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051dd400d61bcc6f8-00632054ce-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 10:00:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-63/ 2022-09-13T19:12:40.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-63/ 2022-09-13T19:12:40.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2022-09-13T19:12:40.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-63/ 2022-09-13T19:12:40.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.526 INFO:tasks.rgw_multisite_tests:HEAD 2022-09-13T19:12:40.526 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.527 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.527 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 10:00:47 GMT 2022-09-13T19:12:40.528 INFO:tasks.rgw_multisite_tests:/yxvvuh-63/ 2022-09-13T19:12:40.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.529 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:ZBXR2QXf6Cg+VjKzil/8M/D340M= 2022-09-13T19:12:40.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 10:00:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:ZBXR2QXf6Cg+VjKzil/8M/D340M=', 'Content-Length': '0'} 2022-09-13T19:12:40.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfd3513a15a93c5d-00632054cf-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 10:00:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-63/d 2022-09-13T19:12:40.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-63/d 2022-09-13T19:12:40.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-63/d 2022-09-13T19:12:40.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-13T19:12:40.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.535 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.536 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2022-09-13T19:12:40.536 INFO:tasks.rgw_multisite_tests:application/octet-stream 2022-09-13T19:12:40.537 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 10:00:47 GMT 2022-09-13T19:12:40.537 INFO:tasks.rgw_multisite_tests:/yxvvuh-63/d 2022-09-13T19:12:40.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.538 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:9Lz6Uu16XFX2yg2n1aih+cWj7Mc= 2022-09-13T19:12:40.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 13 Sep 2022 10:00:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:9Lz6Uu16XFX2yg2n1aih+cWj7Mc='} 2022-09-13T19:12:40.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b45666f057c1f5c-00632054cf-1452-test-zone1'), ('Date', 'Tue, 13 Sep 2022 10:00:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.539 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.540 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:40.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:40.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:40.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:40.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:40.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:40.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663063246.853875_641.1", 2022-09-13T19:12:40.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T10:00:46.853875Z" 2022-09-13T19:12:40.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:40.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:40.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:40.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:40.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663063246.853875_641.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:40.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:40.551 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.551 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T10:05:47.225+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T10:10:47.226+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T10:15:47.225+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T10:20:47.227+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T10:25:47.227+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T10:30:47.228+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T10:35:47.228+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T10:40:47.229+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T10:45:47.230+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2022-09-13T19:12:40.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T10:50:47.229+0000 7fdbc7be8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2022-09-13T19:12:40.559 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:40.559 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.560 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:40.560 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_incremental 2022-09-13T19:12:40.561 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:40.561 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:40.561 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:40.562 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:40.562 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1440, in test_bucket_reshard_incremental 2022-09-13T19:12:40.563 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2022-09-13T19:12:40.563 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:40.564 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2022-09-13T19:12:40.564 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 184, in zone_meta_checkpoint 2022-09-13T19:12:40.565 INFO:tasks.rgw_multisite_tests: period, realm_epoch, num_shards, sync_status = meta_sync_status(zone) 2022-09-13T19:12:40.566 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 141, in meta_sync_status 2022-09-13T19:12:40.567 INFO:tasks.rgw_multisite_tests: assert(retcode == 2) # ENOENT 2022-09-13T19:12:40.568 INFO:tasks.rgw_multisite_tests:AssertionError: 2022-09-13T19:12:40.569 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:40.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-64/ 2022-09-13T19:12:40.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-64/ 2022-09-13T19:12:40.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-64/ 2022-09-13T19:12:40.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:40.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.577 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.578 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.579 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.580 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 10:50:47 GMT 2022-09-13T19:12:40.580 INFO:tasks.rgw_multisite_tests:/yxvvuh-64/ 2022-09-13T19:12:40.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.582 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:pAj3Gqg0qHalGIb5gy7l2SEtBhM= 2022-09-13T19:12:40.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 10:50:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:pAj3Gqg0qHalGIb5gy7l2SEtBhM=', 'Content-Length': '0'} 2022-09-13T19:12:40.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000009d8ba5ce80390655-0063206087-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 10:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=yxvvuh-64 2022-09-13T19:12:40.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.589 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:40.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:40.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:40.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:40.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:40.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663066247.304693_728.1", 2022-09-13T19:12:40.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T10:50:47.304693Z" 2022-09-13T19:12:40.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:40.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:40.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663066247.304693_728.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:40.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:40.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T10:55:47.516+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T11:00:47.518+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T11:05:47.517+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T11:10:47.518+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T11:15:47.518+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T11:20:47.519+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T11:25:47.520+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T11:30:47.519+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T11:35:47.520+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2022-09-13T19:12:40.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T11:40:47.521+0000 7f999b1a0600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2022-09-13T19:12:40.622 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:40.623 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.624 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:40.625 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_full 2022-09-13T19:12:40.626 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:40.627 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:40.628 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:40.629 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:40.631 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1470, in test_bucket_reshard_full 2022-09-13T19:12:40.632 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:40.633 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 184, in zone_meta_checkpoint 2022-09-13T19:12:40.634 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 141, in meta_sync_status 2022-09-13T19:12:40.635 INFO:tasks.rgw_multisite_tests:AssertionError: 2022-09-13T19:12:40.635 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:40.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-65/ 2022-09-13T19:12:40.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-65/ 2022-09-13T19:12:40.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-65/ 2022-09-13T19:12:40.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:40.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.647 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.648 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.649 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.650 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 11:40:47 GMT 2022-09-13T19:12:40.651 INFO:tasks.rgw_multisite_tests:/yxvvuh-65/ 2022-09-13T19:12:40.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.652 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:hVXCgj6CC3K+I3jynIRFWWPmbNQ= 2022-09-13T19:12:40.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 11:40:47 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:hVXCgj6CC3K+I3jynIRFWWPmbNQ=', 'Content-Length': '0'} 2022-09-13T19:12:40.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000003691cd7ae24c11d3-0063206c3f-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 11:40:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=yxvvuh-65 2022-09-13T19:12:40.656 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:40.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:40.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:40.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:40.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663069247.571769_814.1", 2022-09-13T19:12:40.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T11:40:47.571769Z" 2022-09-13T19:12:40.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:40.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:40.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:40.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:40.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663069247.571769_814.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:40.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:40.680 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.682 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T11:45:47.969+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T11:50:47.969+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T11:55:47.969+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T12:00:47.970+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T12:05:47.971+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T12:10:47.970+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T12:15:47.971+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T12:20:47.972+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T12:25:47.972+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2022-09-13T19:12:40.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T12:30:47.973+0000 7fc9b1c27600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2022-09-13T19:12:40.704 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:40.706 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.707 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:40.708 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_creation_time 2022-09-13T19:12:40.709 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:40.711 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:40.712 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:40.713 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:40.714 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1503, in test_bucket_creation_time 2022-09-13T19:12:40.715 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:40.716 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 184, in zone_meta_checkpoint 2022-09-13T19:12:40.717 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 141, in meta_sync_status 2022-09-13T19:12:40.718 INFO:tasks.rgw_multisite_tests:AssertionError: 2022-09-13T19:12:40.719 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:40.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:40.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:40.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:40.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:40.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:40.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663069247.571769_814.1", 2022-09-13T19:12:40.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T11:40:47.571769Z" 2022-09-13T19:12:40.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:40.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:40.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:40.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:40.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663069247.571769_814.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:40.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:40.744 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.745 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T12:35:48.167+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T12:40:48.167+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T12:45:48.167+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T12:50:48.167+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T12:55:48.169+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T13:00:48.169+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T13:05:48.169+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T13:10:48.170+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T13:15:48.170+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2022-09-13T19:12:40.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T13:20:48.170+0000 7f81d6092600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2022-09-13T19:12:40.764 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:40.764 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.765 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:40.766 INFO:tasks.rgw_multisite_tests:FAIL: Create several generations of objects, then run bucket sync 2022-09-13T19:12:40.767 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:40.768 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:40.769 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:40.769 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:40.770 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1588, in test_bucket_sync_run_basic_incremental 2022-09-13T19:12:40.772 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:40.773 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 184, in zone_meta_checkpoint 2022-09-13T19:12:40.773 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 141, in meta_sync_status 2022-09-13T19:12:40.780 INFO:tasks.rgw_multisite_tests:AssertionError: 2022-09-13T19:12:40.781 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:40.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-66/ 2022-09-13T19:12:40.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-66/ 2022-09-13T19:12:40.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-66/ 2022-09-13T19:12:40.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:40.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.792 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.794 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.795 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.796 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 13:20:48 GMT 2022-09-13T19:12:40.797 INFO:tasks.rgw_multisite_tests:/yxvvuh-66/ 2022-09-13T19:12:40.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.798 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:rQtwI+SV5b/INxSvNTw32SCHKxg= 2022-09-13T19:12:40.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 13:20:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:rQtwI+SV5b/INxSvNTw32SCHKxg=', 'Content-Length': '0'} 2022-09-13T19:12:40.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002d4e53e0ad90d229-00632083b0-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 13:20:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=yxvvuh-66 2022-09-13T19:12:40.801 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.802 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:40.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:40.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:40.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:40.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:40.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:40.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663075248.236352_982.1", 2022-09-13T19:12:40.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T13:20:48.236352Z" 2022-09-13T19:12:40.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:40.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:40.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:40.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:40.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663075248.236352_982.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:40.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:40.824 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.825 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T13:25:48.445+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T13:30:48.445+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T13:35:48.446+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T13:40:48.445+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T13:45:48.446+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T13:50:48.446+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T13:55:48.447+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T14:00:48.447+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T14:05:48.448+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2022-09-13T19:12:40.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T14:10:48.449+0000 7fb8cbd93600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2022-09-13T19:12:40.838 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:40.839 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.840 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:40.841 INFO:tasks.rgw_multisite_tests:FAIL: Create several generations of objects, trash them, then run bucket sync init 2022-09-13T19:12:40.843 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:40.845 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:40.846 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:40.847 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:40.848 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1651, in test_zap_init_bucket_sync_run 2022-09-13T19:12:40.849 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:40.850 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 184, in zone_meta_checkpoint 2022-09-13T19:12:40.851 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 141, in meta_sync_status 2022-09-13T19:12:40.853 INFO:tasks.rgw_multisite_tests:AssertionError: 2022-09-13T19:12:40.854 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:40.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-67/ 2022-09-13T19:12:40.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-67/ 2022-09-13T19:12:40.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-67/ 2022-09-13T19:12:40.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:40.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.876 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.876 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.877 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.877 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 14:10:48 GMT 2022-09-13T19:12:40.878 INFO:tasks.rgw_multisite_tests:/yxvvuh-67/ 2022-09-13T19:12:40.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.879 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:qR2tlxB6Z7JKf5yhsDRfve/RbqA= 2022-09-13T19:12:40.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 14:10:48 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:qR2tlxB6Z7JKf5yhsDRfve/RbqA=', 'Content-Length': '0'} 2022-09-13T19:12:40.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000fdc965eaeb628d62-0063208f68-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 14:10:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=yxvvuh-67 2022-09-13T19:12:40.881 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.882 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:40.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:40.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:40.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:40.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:40.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663078248.520924_1066.1", 2022-09-13T19:12:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T14:10:48.520924Z" 2022-09-13T19:12:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663054100.288347_77.1", 2022-09-13T19:12:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:28:20.288347Z" 2022-09-13T19:12:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:40.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:40.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663078248.520924_1066.1', 3: '1_1663054100.288347_77.1'} 2022-09-13T19:12:40.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:40.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.895 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T14:15:48.822+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T14:20:48.823+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T14:25:48.824+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T14:30:48.824+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T14:35:48.825+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T14:40:48.826+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T14:45:48.826+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T14:50:48.826+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T14:55:48.826+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2022-09-13T19:12:40.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T15:00:48.828+0000 7fdb008da600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2022-09-13T19:12:40.913 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:40.913 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.913 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:40.914 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_full_sync_after_data_sync_init 2022-09-13T19:12:40.915 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:40.915 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:40.916 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:40.916 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:40.916 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1716, in test_bucket_full_sync_after_data_sync_init 2022-09-13T19:12:40.917 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:40.917 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 184, in zone_meta_checkpoint 2022-09-13T19:12:40.918 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 141, in meta_sync_status 2022-09-13T19:12:40.918 INFO:tasks.rgw_multisite_tests:AssertionError: 2022-09-13T19:12:40.919 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:40.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-68/ 2022-09-13T19:12:40.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-68/ 2022-09-13T19:12:40.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-68/ 2022-09-13T19:12:40.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:40.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.926 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.926 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.927 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.927 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 15:02:09 GMT 2022-09-13T19:12:40.928 INFO:tasks.rgw_multisite_tests:/yxvvuh-68/ 2022-09-13T19:12:40.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.929 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:u0tWWDjh9hMJPd85SxEiEkC2ANQ= 2022-09-13T19:12:40.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 15:02:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:u0tWWDjh9hMJPd85SxEiEkC2ANQ=', 'Content-Length': '0'} 2022-09-13T19:12:40.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000086b3e5c57e96d14d-0063209b71-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 15:02:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=yxvvuh-68 2022-09-13T19:12:40.931 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:40.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:40.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:40.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:40.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663081329.473165_1154.1", 2022-09-13T19:12:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T15:02:09.473165Z" 2022-09-13T19:12:40.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663081248.863419_600.1", 2022-09-13T19:12:40.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T15:00:48.863419Z" 2022-09-13T19:12:40.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:40.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:40.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663081329.473165_1154.1', 3: '1_1663081248.863419_600.1'} 2022-09-13T19:12:40.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:40.950 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T15:07:09.660+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T15:12:09.660+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T15:17:09.660+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T15:22:09.660+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T15:27:09.661+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T15:32:09.661+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T15:37:09.662+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T15:42:09.663+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T15:47:09.663+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2022-09-13T19:12:40.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T15:52:09.664+0000 7f704b0f1600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2022-09-13T19:12:40.958 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:40.958 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.959 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:40.959 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init 2022-09-13T19:12:40.960 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:40.960 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:40.961 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:40.961 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:40.962 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1748, in test_resharded_bucket_full_sync_after_data_sync_init 2022-09-13T19:12:40.962 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:40.963 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 184, in zone_meta_checkpoint 2022-09-13T19:12:40.963 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 141, in meta_sync_status 2022-09-13T19:12:40.964 INFO:tasks.rgw_multisite_tests:AssertionError: 2022-09-13T19:12:40.964 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:40.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-69/ 2022-09-13T19:12:40.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-69/ 2022-09-13T19:12:40.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:40.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-69/ 2022-09-13T19:12:40.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:40.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:40.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:40.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:40.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:40.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:40.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:40.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:40.970 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:40.970 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.971 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:40.971 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 15:52:09 GMT 2022-09-13T19:12:40.972 INFO:tasks.rgw_multisite_tests:/yxvvuh-69/ 2022-09-13T19:12:40.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:40.973 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:oN2jh2TiWO+XBGv9YrJtt7COK48= 2022-09-13T19:12:40.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 15:52:09 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:oN2jh2TiWO+XBGv9YrJtt7COK48=', 'Content-Length': '0'} 2022-09-13T19:12:40.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002299a2329e6960e2-006320a729-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 15:52:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:40.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=yxvvuh-69 2022-09-13T19:12:40.975 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.976 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:40.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:40.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:40.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:40.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:40.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:40.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663084329.711421_1240.1", 2022-09-13T19:12:40.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T15:52:09.711421Z" 2022-09-13T19:12:40.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:40.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:40.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663081248.863419_600.1", 2022-09-13T19:12:40.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T15:00:48.863419Z" 2022-09-13T19:12:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:40.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663084329.711421_1240.1', 3: '1_1663081248.863419_600.1'} 2022-09-13T19:12:40.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:40.988 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.989 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:40.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:40.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:40.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:40.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T15:57:10.078+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T16:02:10.079+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T16:07:10.080+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T16:12:10.080+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:40.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T16:17:10.081+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T16:22:10.081+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T16:27:10.082+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T16:32:10.082+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T16:37:10.082+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2022-09-13T19:12:41.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T16:42:10.083+0000 7fdea7c11600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2022-09-13T19:12:41.012 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:41.013 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:41.014 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:41.015 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init 2022-09-13T19:12:41.015 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:41.016 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:41.016 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:41.017 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:41.017 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1787, in test_bucket_incremental_sync_after_data_sync_init 2022-09-13T19:12:41.018 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:41.018 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 184, in zone_meta_checkpoint 2022-09-13T19:12:41.019 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 141, in meta_sync_status 2022-09-13T19:12:41.019 INFO:tasks.rgw_multisite_tests:AssertionError: 2022-09-13T19:12:41.020 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:41.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-70/ 2022-09-13T19:12:41.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-70/ 2022-09-13T19:12:41.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:41.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-70/ 2022-09-13T19:12:41.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:41.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:41.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:41.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:41.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:41.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:41.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:41.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:41.027 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:41.027 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:41.028 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:41.028 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 16:42:10 GMT 2022-09-13T19:12:41.029 INFO:tasks.rgw_multisite_tests:/yxvvuh-70/ 2022-09-13T19:12:41.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:41.030 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:1RZM0P9gEvvGqkUn+Sj+GyG0IYI= 2022-09-13T19:12:41.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 16:42:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:1RZM0P9gEvvGqkUn+Sj+GyG0IYI=', 'Content-Length': '0'} 2022-09-13T19:12:41.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000d0ff00ece732f25f-006320b2e2-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 16:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:41.032 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=yxvvuh-70 2022-09-13T19:12:41.033 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:41.034 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:41.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:41.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:41.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:41.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:41.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:41.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:41.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:41.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:41.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:41.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:41.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:41.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:41.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663087330.130711_1326.1", 2022-09-13T19:12:41.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T16:42:10.130711Z" 2022-09-13T19:12:41.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:41.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:41.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663081248.863419_600.1", 2022-09-13T19:12:41.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T15:00:48.863419Z" 2022-09-13T19:12:41.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:41.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:41.050 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663087330.130711_1326.1', 3: '1_1663081248.863419_600.1'} 2022-09-13T19:12:41.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:41.051 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:41.052 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:41.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:41.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T16:47:10.339+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T16:52:10.340+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T16:57:10.341+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T17:02:10.341+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T17:07:10.342+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T17:12:10.341+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T17:17:10.342+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T17:22:10.342+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T17:27:10.343+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2022-09-13T19:12:41.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T17:32:10.343+0000 7fc1e8c67600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2022-09-13T19:12:41.081 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:41.082 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:41.083 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:41.084 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2022-09-13T19:12:41.085 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:41.086 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:41.086 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:41.087 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:41.088 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1825, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2022-09-13T19:12:41.089 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:41.090 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 184, in zone_meta_checkpoint 2022-09-13T19:12:41.090 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 141, in meta_sync_status 2022-09-13T19:12:41.091 INFO:tasks.rgw_multisite_tests:AssertionError: 2022-09-13T19:12:41.092 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:41.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-71/ 2022-09-13T19:12:41.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-71/ 2022-09-13T19:12:41.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:41.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-71/ 2022-09-13T19:12:41.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:41.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:41.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:41.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:41.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:41.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:41.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:41.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:41.102 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:41.103 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:41.104 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:41.105 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 17:32:10 GMT 2022-09-13T19:12:41.105 INFO:tasks.rgw_multisite_tests:/yxvvuh-71/ 2022-09-13T19:12:41.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:41.107 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:nWafIa8uhVz6iOLhHgYxvod2AS8= 2022-09-13T19:12:41.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 17:32:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:nWafIa8uhVz6iOLhHgYxvod2AS8=', 'Content-Length': '0'} 2022-09-13T19:12:41.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000100a34e1dc03416e-006320be9a-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 17:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:41.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=yxvvuh-71 2022-09-13T19:12:41.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:41.113 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:41.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:41.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:41.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:41.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:41.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:41.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:41.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:41.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:41.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:41.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:41.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:41.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:41.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:41.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663090330.404695_1412.1", 2022-09-13T19:12:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T17:32:10.404695Z" 2022-09-13T19:12:41.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663081248.863419_600.1", 2022-09-13T19:12:41.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T15:00:48.863419Z" 2022-09-13T19:12:41.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:41.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:41.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663090330.404695_1412.1', 3: '1_1663081248.863419_600.1'} 2022-09-13T19:12:41.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:41.153 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:41.153 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:41.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:41.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:41.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:41.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T17:37:10.621+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T17:42:10.622+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T17:47:10.623+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T17:52:10.623+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T17:57:10.624+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T18:02:10.623+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T18:07:10.624+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T18:12:10.624+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T18:17:10.625+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2022-09-13T19:12:41.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T18:22:10.626+0000 7f8674457600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2022-09-13T19:12:41.169 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:41.171 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:41.173 INFO:tasks.rgw_multisite_tests:====================================================================== 2022-09-13T19:12:41.174 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2022-09-13T19:12:41.175 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:41.177 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2022-09-13T19:12:41.178 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2022-09-13T19:12:41.179 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2022-09-13T19:12:41.180 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 1873, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2022-09-13T19:12:41.181 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 207, in zonegroup_meta_checkpoint 2022-09-13T19:12:41.182 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 184, in zone_meta_checkpoint 2022-09-13T19:12:41.183 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/../src/test/rgw/rgw_multi/tests.py", line 141, in meta_sync_status 2022-09-13T19:12:41.184 INFO:tasks.rgw_multisite_tests:AssertionError: 2022-09-13T19:12:41.185 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2022-09-13T19:12:41.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yxvvuh-72/ 2022-09-13T19:12:41.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yxvvuh-72/ 2022-09-13T19:12:41.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2022-09-13T19:12:41.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yxvvuh-72/ 2022-09-13T19:12:41.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2022-09-13T19:12:41.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2022-09-13T19:12:41.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi036.front.sepia.ceph.com:8000 2022-09-13T19:12:41.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2022-09-13T19:12:41.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2022-09-13T19:12:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-13T19:12:41.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2022-09-13T19:12:41.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2022-09-13T19:12:41.197 INFO:tasks.rgw_multisite_tests:PUT 2022-09-13T19:12:41.197 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:41.199 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:41.200 INFO:tasks.rgw_multisite_tests:Tue, 13 Sep 2022 18:22:10 GMT 2022-09-13T19:12:41.200 INFO:tasks.rgw_multisite_tests:/yxvvuh-72/ 2022-09-13T19:12:41.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2022-09-13T19:12:41.203 INFO:tasks.rgw_multisite_tests:AWS RICCPJUVKDGV9UEGFN6Z:a0urf2N/1m5kw9uSKoSjlHj+AHQ= 2022-09-13T19:12:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Tue, 13 Sep 2022 18:22:10 GMT', 'Authorization': 'AWS RICCPJUVKDGV9UEGFN6Z:a0urf2N/1m5kw9uSKoSjlHj+AHQ=', 'Content-Length': '0'} 2022-09-13T19:12:41.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000009617ebeba0a61a12-006320ca52-1452-test-zone1'), ('Content-Length', '0'), ('Date', 'Tue, 13 Sep 2022 18:22:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-13T19:12:41.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=yxvvuh-72 2022-09-13T19:12:41.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:41.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:41.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:41.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stderr: INFO: No --period given, using current period=3255cc65-e34b-4e83-a436-e06431413191 2022-09-13T19:12:41.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: [ 2022-09-13T19:12:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663053952.976234_34.1", 2022-09-13T19:12:41.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T07:25:52.976234Z" 2022-09-13T19:12:41.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:41.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:41.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "", 2022-09-13T19:12:41.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "0.000000" 2022-09-13T19:12:41.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:41.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:41.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663093330.686299_1498.1", 2022-09-13T19:12:41.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T18:22:10.686299Z" 2022-09-13T19:12:41.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: }, 2022-09-13T19:12:41.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: { 2022-09-13T19:12:41.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "marker": "1_1663081248.863419_600.1", 2022-09-13T19:12:41.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: "last_update": "2022-09-13T15:00:48.863419Z" 2022-09-13T19:12:41.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: } 2022-09-13T19:12:41.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi036.stdout: INFO: ] 2022-09-13T19:12:41.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1663053952.976234_34.1', 1: '', 2: '1_1663093330.686299_1498.1', 3: '1_1663081248.863419_600.1'} 2022-09-13T19:12:41.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2022-09-13T19:12:41.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:41.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T19:12:41.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T19:12:41.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setuser ceph since I am not root 2022-09-13T19:12:41.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: ignoring --setgroup ceph since I am not root 2022-09-13T19:12:41.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T18:27:10.906+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T18:32:10.905+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T18:37:10.907+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T18:42:10.907+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T18:47:10.907+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T18:52:10.908+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T18:57:10.908+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T19:02:10.909+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T19:07:10.910+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2022-09-13T19:12:41.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: 2022-09-13T19:12:10.910+0000 7f9397c49600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T19:12:41.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi083.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2022-09-13T19:12:41.227 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2022-09-13T19:12:41.227 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:41.228 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2022-09-13T19:12:41.229 INFO:tasks.rgw_multisite_tests:Ran 39 tests in 42258.966s 2022-09-13T19:12:41.229 INFO:tasks.rgw_multisite_tests: 2022-09-13T19:12:41.230 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=1, errors=4, failures=20) 2022-09-13T19:12:41.241 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/rgw_multisite_tests.py", line 95, in begin AssertionError: rgw multisite test failures 2022-09-13T19:12:41.699 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=47d2610a4dd54b40a29ed8947ffd97b9 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/rgw_multisite_tests.py", line 95, in begin AssertionError: rgw multisite test failures 2022-09-13T19:12:41.702 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2022-09-13T19:12:41.716 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2022-09-13T19:12:41.787 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2022-09-13T19:12:41.821 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2022-09-13T19:12:41.822 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T19:12:41.828 INFO:tasks.rgw.c1.client.0.smithi036.stdout:2022-09-13T19:12:41.820+0000 7f392e4d5700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 109476) UID: 0 2022-09-13T19:12:41.829 INFO:tasks.rgw.c1.client.0.smithi036.stdout:2022-09-13T19:12:41.820+0000 7f3932740600 -1 shutting down 2022-09-13T19:12:42.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2022-09-13T19:12:47.928 INFO:tasks.rgw.c1.client.0:Stopped 2022-09-13T19:12:47.930 DEBUG:teuthology.orchestra.run.smithi036:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2022-09-13T19:12:47.972 DEBUG:teuthology.orchestra.run.smithi036:> sudo rm -f /etc/ceph/vault-root-token 2022-09-13T19:12:48.057 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2022-09-13T19:12:48.058 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T19:12:48.084 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2022-09-13T19:12:48.099 INFO:tasks.rgw.c1.client.1.smithi036.stdout:2022-09-13T19:12:48.095+0000 7f8f6d7bf700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 110836) UID: 0 2022-09-13T19:12:48.109 INFO:tasks.rgw.c1.client.1.smithi036.stdout:2022-09-13T19:12:48.095+0000 7f8f71a2a600 -1 shutting down 2022-09-13T19:12:53.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2022-09-13T19:12:53.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2022-09-13T19:12:54.161 INFO:tasks.rgw.c1.client.1:Stopped 2022-09-13T19:12:54.162 DEBUG:teuthology.orchestra.run.smithi036:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2022-09-13T19:12:54.186 DEBUG:teuthology.orchestra.run.smithi036:> sudo rm -f /etc/ceph/vault-root-token 2022-09-13T19:12:54.263 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2022-09-13T19:12:54.264 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T19:12:54.266 INFO:tasks.rgw.c2.client.0:Stopped 2022-09-13T19:12:54.267 DEBUG:teuthology.orchestra.run.smithi083:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2022-09-13T19:12:54.288 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f /etc/ceph/vault-root-token 2022-09-13T19:12:54.360 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2022-09-13T19:12:54.361 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T19:12:54.363 INFO:tasks.rgw.c2.client.1:Stopped 2022-09-13T19:12:54.364 DEBUG:teuthology.orchestra.run.smithi083:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2022-09-13T19:12:54.418 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f /etc/ceph/vault-root-token 2022-09-13T19:12:54.489 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2022-09-13T19:12:54.502 INFO:tasks.ceph.ceph_manager.c2:Canceling any pending splits or merges... 2022-09-13T19:12:54.504 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-09-13T19:12:59.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2022-09-13T19:14:54.563 DEBUG:teuthology.orchestra.run:got remote process result: 124 2022-09-13T19:14:54.565 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph.py", line 1902, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph_manager.py", line 2302, in stop_pg_num_changes File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph_manager.py", line 2536, in get_osd_dump_json File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph_manager.py", line 1609, in raw_cluster_cmd File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph_manager.py", line 1600, in run_cluster_cmd File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/remote.py", line 510, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json' 2022-09-13T19:14:54.598 INFO:teuthology.misc:Shutting down mds daemons... 2022-09-13T19:14:54.600 INFO:teuthology.misc:Shutting down osd daemons... 2022-09-13T19:14:54.602 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2022-09-13T19:14:54.603 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T19:14:54.609 INFO:tasks.ceph.c2.osd.0:Stopped 2022-09-13T19:14:54.612 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2022-09-13T19:14:54.613 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T19:14:54.614 INFO:tasks.ceph.c2.osd.1:Stopped 2022-09-13T19:14:54.615 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2022-09-13T19:14:54.616 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T19:14:54.618 INFO:tasks.ceph.c2.osd.2:Stopped 2022-09-13T19:14:54.619 INFO:teuthology.misc:Shutting down mgr daemons... 2022-09-13T19:14:54.621 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2022-09-13T19:14:54.622 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T19:14:54.624 INFO:tasks.ceph.c2.mgr.x:Stopped 2022-09-13T19:14:54.625 INFO:teuthology.misc:Shutting down mon daemons... 2022-09-13T19:14:54.626 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2022-09-13T19:14:54.628 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T19:14:54.630 INFO:tasks.ceph.c2.mon.a:Stopped 2022-09-13T19:14:54.632 INFO:tasks.ceph:Checking cluster log for badness... 2022-09-13T19:14:54.634 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2022-09-13T19:14:54.672 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi083.front.sepia.ceph.com 2022-09-13T19:14:54.673 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2022-09-13T19:14:54.843 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi083.front.sepia.ceph.com 2022-09-13T19:14:54.845 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2022-09-13T19:14:54.965 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi083.front.sepia.ceph.com 2022-09-13T19:14:54.966 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2022-09-13T19:14:55.064 INFO:tasks.ceph:Archiving mon data... 2022-09-13T19:14:55.075 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuvalif-2022-09-13_07:01:40-rgw:multisite-wip-yuval-fix-56572-distro-default-smithi/7030425/data/c2.mon.a.tgz 2022-09-13T19:14:55.077 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2022-09-13T19:14:55.099 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.0bTe4JH1F0 2022-09-13T19:14:55.100 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.0bTe4JH1F0 2022-09-13T19:14:55.240 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.0bTe4JH1F0 2022-09-13T19:14:55.330 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.0bTe4JH1F0 is 299KB 2022-09-13T19:14:55.429 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.0bTe4JH1F0 2022-09-13T19:14:55.453 INFO:tasks.ceph:Cleaning ceph cluster... 2022-09-13T19:14:55.455 DEBUG:teuthology.orchestra.run.smithi036:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2022-09-13T19:14:55.482 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2022-09-13T19:14:55.524 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph.py", line 1902, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph_manager.py", line 2302, in stop_pg_num_changes File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph_manager.py", line 2536, in get_osd_dump_json File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph_manager.py", line 1609, in raw_cluster_cmd File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph_manager.py", line 1600, in run_cluster_cmd File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/remote.py", line 510, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json' 2022-09-13T19:14:55.530 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2022-09-13T19:14:55.564 INFO:tasks.ceph.ceph_manager.c1:Canceling any pending splits or merges... 2022-09-13T19:14:55.565 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-09-13T19:14:56.108 INFO:teuthology.orchestra.run.smithi036.stdout: 2022-09-13T19:14:56.109 INFO:teuthology.orchestra.run.smithi036.stdout:{"epoch":56,"fsid":"5cb3badf-0576-4aaa-b80a-741bd397fa0b","created":"2022-09-13T07:23:47.366415+0000","modified":"2022-09-13T07:27:57.490666+0000","last_up_change":"2022-09-13T07:23:54.945776+0000","last_in_change":"2022-09-13T07:23:48.399710+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-13T07:23:55.185879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-09-13T07:23:56.179172+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2022-09-13T07:24:31.801981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2022-09-13T07:24:33.772523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2022-09-13T07:24:43.586741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2022-09-13T07:24:45.489095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2022-09-13T07:24:47.486070+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2022-09-13T07:24:49.509845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"test-zone1.rgw.buckets.data","create_time":"2022-09-13T07:25:03.871062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"test-zone1.rgw.log","create_time":"2022-09-13T07:25:05.699152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":11,"pool_name":"test-zone1.rgw.control","create_time":"2022-09-13T07:25:07.535826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"test-zone1.rgw.meta","create_time":"2022-09-13T07:25:09.583498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":13,"pool_name":"test-zone1.rgw.otp","create_time":"2022-09-13T07:25:36.170401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":14,"pool_name":"test-zone3.rgw.buckets.data","create_time":"2022-09-13T07:25:37.162415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":15,"pool_name":"test-zone3.rgw.log","create_time":"2022-09-13T07:25:38.781025+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"test-zone3.rgw.control","create_time":"2022-09-13T07:25:40.605395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":17,"pool_name":"test-zone3.rgw.meta","create_time":"2022-09-13T07:25:42.649326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":18,"pool_name":"test-zone1.rgw.buckets.index","create_time":"2022-09-13T07:27:52.019344+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":19,"pool_name":"test-zone3.rgw.buckets.index","create_time":"2022-09-13T07:27:54.611096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"1b15dcf5-6e6b-4e8a-8c6e-50e2d3bdce2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6808","nonce":101695},{"type":"v1","addr":"172.21.15.36:6809","nonce":101695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6810","nonce":101695},{"type":"v1","addr":"172.21.15.36:6811","nonce":101695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6814","nonce":101695},{"type":"v1","addr":"172.21.15.36:6815","nonce":101695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6812","nonce":101695},{"type":"v1","addr":"172.21.15.36:6813","nonce":101695}]},"public_addr":"172.21.15.36:6809/101695","cluster_addr":"172.21.15.36:6811/101695","heartbeat_back_addr":"172.21.15.36:6815/101695","heartbeat_front_addr":"172.21.15.36:6813/101695","state":["exists","up"]},{"osd":1,"uuid":"1d74416f-f6f4-49ea-815e-b2cd7ebfd4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6800","nonce":101694},{"type":"v1","addr":"172.21.15.36:6801","nonce":101694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6802","nonce":101694},{"type":"v1","addr":"172.21.15.36:6803","nonce":101694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6806","nonce":101694},{"type":"v1","addr":"172.21.15.36:6807","nonce":101694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6804","nonce":101694},{"type":"v1","addr":"172.21.15.36:6805","nonce":101694}]},"public_addr":"172.21.15.36:6801/101694","cluster_addr":"172.21.15.36:6803/101694","heartbeat_back_addr":"172.21.15.36:6807/101694","heartbeat_front_addr":"172.21.15.36:6805/101694","state":["exists","up"]},{"osd":2,"uuid":"b2c4cea1-1b7e-43ef-8535-eaa77e15ee3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6816","nonce":101724},{"type":"v1","addr":"172.21.15.36:6817","nonce":101724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6818","nonce":101724},{"type":"v1","addr":"172.21.15.36:6819","nonce":101724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6822","nonce":101724},{"type":"v1","addr":"172.21.15.36:6823","nonce":101724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6820","nonce":101724},{"type":"v1","addr":"172.21.15.36:6821","nonce":101724}]},"public_addr":"172.21.15.36:6817/101724","cluster_addr":"172.21.15.36:6819/101724","heartbeat_back_addr":"172.21.15.36:6823/101724","heartbeat_front_addr":"172.21.15.36:6821/101724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-13T07:23:52.778878+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-13T07:23:52.740972+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-13T07:23:52.769124+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.30","mappings":[{"from":0,"to":1}]},{"pgid":"3.3a","mappings":[{"from":0,"to":1}]},{"pgid":"4.4","mappings":[{"from":1,"to":0}]},{"pgid":"4.6","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":0}]},{"pgid":"4.1b","mappings":[{"from":1,"to":0}]},{"pgid":"4.28","mappings":[{"from":1,"to":0}]},{"pgid":"4.37","mappings":[{"from":1,"to":2}]},{"pgid":"10.c","mappings":[{"from":2,"to":0}]},{"pgid":"14.17","mappings":[{"from":1,"to":0}]},{"pgid":"14.20","mappings":[{"from":1,"to":2}]},{"pgid":"14.24","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-13T19:14:56.124 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2022-09-13T19:14:56.465 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph.py", line 1898, in task File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph.py", line 1902, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph_manager.py", line 2302, in stop_pg_num_changes File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph_manager.py", line 2536, in get_osd_dump_json File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph_manager.py", line 1609, in raw_cluster_cmd File "/home/teuthworker/src/git.ceph.com_ceph-c_840d05418c459144c92fb2a90d74a2ded29610da/qa/tasks/ceph_manager.py", line 1600, in run_cluster_cmd File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/remote.py", line 510, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json' 2022-09-13T19:14:56.468 INFO:teuthology.misc:Shutting down mds daemons... 2022-09-13T19:14:56.469 INFO:teuthology.misc:Shutting down osd daemons... 2022-09-13T19:14:56.470 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2022-09-13T19:14:56.471 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T19:14:56.554 INFO:tasks.ceph.c1.osd.0:Stopped 2022-09-13T19:14:56.555 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2022-09-13T19:14:56.556 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T19:14:56.639 INFO:tasks.ceph.c1.osd.1:Stopped 2022-09-13T19:14:56.640 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2022-09-13T19:14:56.641 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T19:14:56.715 INFO:tasks.ceph.c1.osd.2:Stopped 2022-09-13T19:14:56.717 INFO:teuthology.misc:Shutting down mgr daemons... 2022-09-13T19:14:56.719 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2022-09-13T19:14:56.720 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T19:14:56.745 INFO:tasks.ceph.c1.mgr.x:Stopped 2022-09-13T19:14:56.746 INFO:teuthology.misc:Shutting down mon daemons... 2022-09-13T19:14:56.747 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2022-09-13T19:14:56.748 INFO:teuthology.orchestra.run:waiting for 300 2022-09-13T19:14:56.826 INFO:tasks.ceph.c1.mon.a:Stopped 2022-09-13T19:14:56.827 INFO:tasks.ceph:Checking cluster log for badness... 2022-09-13T19:14:56.828 DEBUG:teuthology.orchestra.run.smithi036:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2022-09-13T19:14:56.969 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi036.front.sepia.ceph.com 2022-09-13T19:14:56.971 DEBUG:teuthology.orchestra.run.smithi036:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2022-09-13T19:14:57.230 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi036.front.sepia.ceph.com 2022-09-13T19:14:57.232 DEBUG:teuthology.orchestra.run.smithi036:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2022-09-13T19:14:57.354 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi036.front.sepia.ceph.com 2022-09-13T19:14:57.355 DEBUG:teuthology.orchestra.run.smithi036:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2022-09-13T19:14:57.452 INFO:tasks.ceph:Archiving mon data... 2022-09-13T19:14:57.454 DEBUG:teuthology.misc:Transferring archived files from smithi036:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuvalif-2022-09-13_07:01:40-rgw:multisite-wip-yuval-fix-56572-distro-default-smithi/7030425/data/c1.mon.a.tgz 2022-09-13T19:14:57.454 DEBUG:teuthology.orchestra.run.smithi036:> mktemp 2022-09-13T19:14:57.498 INFO:teuthology.orchestra.run.smithi036.stdout:/tmp/tmp.lFPWHYtibc 2022-09-13T19:14:57.499 DEBUG:teuthology.orchestra.run.smithi036:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.lFPWHYtibc 2022-09-13T19:14:57.700 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 0666 /tmp/tmp.lFPWHYtibc 2022-09-13T19:14:57.755 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-09-13T19:14:57.759 DEBUG:teuthology.task.console_log:Killing console logger for smithi036 2022-09-13T19:14:57.761 DEBUG:teuthology.task.console_log:Killing console logger for smithi083 2022-09-13T19:14:57.762 DEBUG:teuthology.exit:Finished running handlers